Narrative-UFET: Narrative Generation for Ultra-Fine Entity Typing

Ultra-fine entity typing (UFET) is the task of assigning highly specific types to entity mentions based on the context in which they appear Choi et al. (2018). Unlike coarse-grained typing, which selects from a small set of broad categories such as person or organization, UFET aims to capture contextually relevant distinctions: in the sentence “In his next note on Baidu, he wrote that the company is trading above its fair value,”, the pronoun “he” could be typed coarsely as a person, but the surrounding context supports more specific types, such as writer, editor, or analyst. Identifying these fine-grained types is valuable for a wide range of downstream tasks, including coreference resolution Onoe and Durrett (2020), entity linking Ling et al. (2015), relation extraction Koch et al. (2014), knowledge graph completion Li et al. (2024b) and multi-modal entity recognition and grounding Wang et al. (2024); Li et al. (2024a).

A central challenge for UFET is the long tail of the entity distribution. Pretrained language models (PLMs), which most current approaches build on Dai et al. (2021); Li et al. (2023); Deshmukh et al. (2025), perform well on entities that appear frequently in their pretraining corpora but degrade sharply on rare ones Deshmukh et al. (2025). This is precisely the setting where fine-grained disambiguation is most needed. We hypothesize that part of this gap stems from a reliance on sentence-level context. In realistic settings such as articles or books, the evidence needed to disambiguate fine-grained types is rarely contained in a single sentence. Instead, it must be pieced together from cues distributed across the surrounding narrative. Testing this hypothesis has been difficult, however, because all existing annotated resources for fine and ultra-fine entity typing are sentence-level Riedel et al. (2010); Gillick et al. (2014b); Choi et al. (2018); Ling and Weld (2021); Ding et al. (2021).

To address this gap we construct Narrative-UFET, an extended version of the UFET dataset Choi et al. (2018) in which each entity-sentence pair is paired with a short, automatically generated narrative built around the target entity. We choose to synthesize narratives rather than retrieve them from real corpora because synthesis lets us control specific properties of the discourse, holding everything else fixed. This control is the core methodological move of the paper, as it lets us isolate the effect of a single discourse property on typing performance. As a case study, we construct two variants of Narrative-UFET that differ in one such property: whether the entity’s type is held constant across the narrative (Maintain) or shifts across it (Change). We use this contrast to test whether type variation across discourse is a useful signal for ultra-fine typing.

We validate the quality of the generated narratives through automated metrics and human evaluation. Evaluating both masked and causal PLMs on the UFET task, we find that narrative context yields consistent improvements on long-tail types over sentence-level baselines, with the Change variant providing the stronger signal. A comparison against naturally occurring contexts shows that our synthetic narratives yield stronger gains than real text alone, indicating that controlled discourse construction can surface signal that real text leaves implicit. At the same time, substantial room for improvement remains, suggesting that progress on long-tail entity typing will require advances in both discourse-aware modeling and narrative construction beyond the single dimension studied here.

Our contributions are: (i) Narrative-UFET, a controlled narrative-level extension of UFET with two variants (Maintain and Change); (ii) evidence that narrative context improves typing on long-tail entities, with Change providing the strongest signal; and (iii) a comparison against naturally occurring contexts showing that synthetic narratives surface signal that real text leaves implicit.

Narrative Generation and Evaluation. With the rise of powerful generative models, large language models (LLMs) are increasingly used to generate high-quality short and long narratives that exhibit more nuanced plot development and stylistic variation Goldfarb-Tarrant et al. (2020); Yang et al. (2022); Harel-Canada et al. (2024). Additionally, recent work has explored the use of LLMs in assessing the quality of narratives. This is a complex task as quality is multi-faceted and at times subjective. Prior work has evaluated grammar as language fluency Naismith et al. (2023), creativity and originality as indicators of narrative novelty, and consistency in plot and character progression to assess story coherence over time Chhun et al. (2022); Tian et al. (2024). Recent research also emphasizes the importance of prompt engineering in guiding LLMs to produce more coherent and contextually appropriate narratives Tang et al. (2024). Although prior work has advanced narrative generation and evaluation, it remains unclear how well models can expand an entity-context pair into a complete narrative that is coherent and remains reliable when evaluated on the entity-typing task. To construct Narrative-UFET we conduct model and prompt testing to generate narratives. We then verify the validity of Narrative-UFET through automated and human evaluation.

UFET and Pre-trained Language Models. UFET was introduced by Choi et al. (2018), who proposed the task of predicting free-form type labels for a target entity mention given the sentence in which it appears. Subsequent work leveraged PLMs to improve entity typing performance. Many of these approaches frame UFET as a masked language modeling problem. Dai et al. (2021) use Hearst patterns with [MASK] tokens and BERT to predict entity types. Similarly, Pan et al. (2022) generate ultra-fine type predictions by appending an entity mention and a [MASK] token to the input sentence. Deshmukh et al. (2025) expand on this work by exploring how such approaches perform on infrequent or rare entities, showing that PLMs struggle with long-tail entity typing unless additional knowledge about rare entities is incorporated, and even then the explored knowledge injection strategies proved insufficient. Motivated by these findings, we use Narrative-UFET to investigate whether richer discourse context can help PLMs better predict long-tail types, evaluating both masked and causal PLMs.

In this section, we describe the construction and evaluation of Narrative-UFET. We build on the crowd-annotated portion of the UFET dataset Choi et al. (2018), which contains 5,994 entity mentions paired with their surrounding sentence context and a set of human-annotated ultra-fine types (hereafter, gold types), evenly divided into training, development, and test splits.

We evaluate whether Narrative-UFET improves entity typing performance with respect to sentence-level UFET, particularly for long-tail entities. Following Deshmukh et al. (2025), we partition the UFET test set into four bins. Details on the bin split and internet search hits are shown in Appendix D.1. We evaluate using both masked language model (MLM) and causal language model (CLM) approaches. All scoring is done following the UFET guidelines Choi et al. (2018).

Narrative context improves typing performance. We adopt the MLM and CLM setups of Deshmukh et al. (2025); details are in Appendices D.2 and D.3. For CLMs, we run each condition five times and report the mean to account for model variability. Figure 1 compares performance between the sentence-level baseline and Narrative-UFET across both setups. Narrative-UFET yields consistent improvements across all bins, with CLMs reaching higher overall performance than MLMs, and Qwen showing a larger relative gain. Across both model types, Narrative-UFET-Change substantially outperforms Narrative-UFET-Maintain, suggesting that changing the type across the narrative produces a richer typing signal.

Synthetic narratives outperform real context. To check that our synthetic narratives capture properties of real discourse, we run both MLM and CLM evaluations using the original OntoNotes 5.0 contexts Gillick et al. (2014a). Figure 1 shows that Narrative-UFET-Change outperforms OntoNotes context across both model types, while OntoNotes performance sits between Standard-UFET and Narrative-UFET-Maintain. This indicates that the synthetic narratives preserve properties of naturally occurring context while the controlled type-shift design in Change surfaces typing signal that real text leaves implicit.

Results for additional model families and sizes, and additional analyses in Appendix D.4.

We presented Narrative-UFET, a controlled narrative-level extension of UFET in which each entity mention is paired with an automatically generated narrative built around it. Our hypothesis was that synthesizing narratives lets us isolate the effect of specific discourse properties. To study this, we constructed two variants that differ in whether the entity’s type is held constant (Maintain) or shifts (Change) across the narrative. We found that narrative context yields consistent gains on long-tail entities over sentence-level baselines, with the Change variant providing the stronger typing signal, and outperforming naturally occurring contexts, suggesting that controlled construction can surface signal that real text leaves implicit.

The type-consistency contrast studied here is one of many discourse properties that controlled narrative construction makes accessible. Coreference density, interactions between co-occurring entities, the granularity of type shifts, and the distribution of typing evidence across sentences are all dimensions that Narrative-UFET could be extended along, and each may explain a different part of the gap between current models and the typing signal available in discourse. We see this as the main direction opened by our work: not just longer context for entity typing, but a systematic investigation of which discourse properties carry typing signal and how models can exploit them.

We identify six main limitations. (1) Our case study varies only one property of the narrative: whether the entity’s type is held constant or shifts. Other discourse properties (e.g., coreference density, inter-entity interactions, the distribution of typing evidence across sentences) are likely to carry typing signal as well, and we leave their systematic study to future work. (2) While we validate narrative quality through automated and human evaluation, and compare against naturally occurring contexts from OntoNotes, synthetic narratives may deviate from natural discourse distributions in ways that our evaluation does not capture. Extending Narrative-UFET with retrieved or human-authored narratives is an important direction. (3) We attribute the stronger performance of the Change variant to richer typing signal from type variation across discourse, but we do not control for confounding factors such as token count, lexical diversity, or the number of distinct types mentioned. The relative contribution of these factors remains open. (4) Our experiments cover one MLM (BERT) and two CLMs (Llama3.3-70B, Qwen3-32B) on a single base typing dataset (UFET, English, crowd-annotated). Findings may not transfer to other typing schemes, languages, or model families. (5) For computational reasons, all models in Section 3 were run in quantized form. Quantization can affect both narrative generation quality and typing predictions. We did not systematically measure these effects. (6) Our qualitative review during model selection was conducted by a single annotator without formal guidelines. While the final dataset is also validated through inter-annotator human evaluation with formal rubrics, the model-selection stage relied on lighter-weight judgment.

To the best of our knowledge, this work does not raise any ethical concerns. All information required to replicate our experiments is provided in the paper and the appendices. We use only open-source language models with publicly available and versioned weights. Importantly, we ensure separation between the models used for narrative generation, narrative evaluation, and entity typing evaluation, avoiding circular dependencies that could inflate results. Additional plots and implementation details are provided in the appendix.