Measuring Contextual Informativeness in Child-Directed Text
Maria Valentini*, Téa Wright*, Ali Marashian, Jennifer Weber, Eliana Colunga Katharina von der Wense · 2024
Read Measuring Contextual Informativeness in Child-Directed Text by Maria Valentini*, Téa Wright*, Ali Marashian, Jennifer Weber, Eliana Colunga Katharina von der Wense (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17427.txt.
Introduction
Maria Valentini and five co-authors tackle how to judge whether a children’s story truly teaches a new word, in a 2024 study. A story can use a word often yet still leave its meaning murky. The team calls the missing trait contextual informativeness: how well the text around a word reveals what it means.
They define the task in formal terms and build a data set of judged stories. Their method asks a large language model to rate how clear each story makes its target words. Ratings matched human judges far better than the strongest older baseline did. The same method also beat baselines on text written for adults. Better scores could help tools that write vocabulary stories on demand.
Catalog record
| CREATOR | Maria Valentini*, Téa Wright*, Ali Marashian, Jennifer Weber, Eliana Colunga Katharina von der Wense |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-17427
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Measuring Contextual Informativeness in Child-Directed Text Maria Valentini*, Téa Wright*, Ali Marashian, Jennifer Weber, Eliana Colunga Katharina von der Wense Abstract To address an important gap in creating children’s stories for vocabulary enrichment, we investigate the automatic evaluation of how well stories convey the semantics of target vocabulary words, a task with substantial implications for generating educational content. We motivate this task, which we call measuring contextual informativeness in children’s stories , and provide a formal task definition as well as a dataset for the task. We further propose a method for automating the task using a large language model (LLM). Our experiments show that our approach reaches a Spearman correlation of 0.4983 with human judgments of informativeness, while the strongest baseline only obtains a correlation of 0.3534. An additional analysis shows that the LLM-based approach is able to generalize to measuring contextual informativeness in adult-directed text, on which it also outperforms all baselines. † † * denotes equal contribution. 1 Introduction Recent advances in natural language processing (NLP) have put the fully automated generation of children’s stories within reach Valentini et al. 2023 . Automatically-generated stories can be used for targeted vocabulary interventions for preschoolers when centered around desirable target words. As early vocabulary size is strongly correlated with reading ability in elementary school Walker et al. 1994 and future academic success Brysbaert et al. 2016 , such scalable interventions will contribute to leveling out existing inequalities. Figure 1: An example of an LLM-generated story providing an uninformative context for the word "prickly" and an informative context for the word "receipt." Italicized words represent helpful context terms in the passage corresponding to the target word of the same color. Approximately 3,000 words are acquired each year in early childhood, primarily through incidental learning during reading Nagy and Anderson 1984 . However, just including target words in stories might not be enough for effective vocabulary enrichment: the amount of semantic information about a word in a story can vary widely. This issue is exasperated in stories generated by large language models (LLMs) when target words are often used in uninformative and misleading contexts; see Figure 1 . Automatically quantifying the amount of information about a word provided by a given story can help streamline the selection of effective stories for vocabulary learning and improve story generation models to support this purpose. With these benefits in mind, we introduce the task of measuring contextual informativeness in children’s stories and create a dataset for evaluation. We further propose the use of Gemini Gemini-Team et al. 2024 for this task with respect to a set of target words. We compare its performance with that of another proposed RoBERTa Liu et al. 2019a -based model, as well as multiple baselines. We find that, on the dataset we introduce, Gemini obtains a Spearman’s \rho value of 0.4983, while the RoBERTa-based model reaches 0.4601 and the strongest baseline only reaches 0.3534. We also show that our model generalizes to other domains, outperforming other approaches to measuring contextual informativeness in adult-directed text. To summarize, we make the following contributions: (1) we propose the task of measuring contextual informativeness in children’s stories; (2) we introduce a dataset for the task, which consists of automatically generated children’s stories that have been annotated for the amount of contextual support they provide for target words; (3) we propose a method for the task and show that it outperforms multiple baselines; and (4) we demonstrate that our method generalizes for adult-directed text. Our dataset is available at https://github.com/mariavale/contextual \_ inform . 2 Related Work In-context Vocabulary Learning As mentioned in Section 1 , research shows that the majority of new words are learned incidentally through reading in L1 learners Nagy and Anderson 1984 . As such, many modern vocabulary intervention approaches focus on in-context learning. Studies such as Webb 2008 provide evidence that more contextual clues about target words can lead to better learning outcomes. In addition, previous work stresses the importance of vocabulary intervention early in a child’s life and the correlation of early vocabulary size with future academic success Walker et al. 1994 ; Duff et al. 2015 ; Brysbaert et al. 2016 . Cloze Task The cloze task Taylor 1953 is designed to assess lexical and contextual understanding by removing words from a text, requiring participants to fill in the blanks with the missing words. Since its establishment, there has been disagreement about what the task truly measures. The evaluation of the task typically only allows one correct answer, raising concerns about how accurately it measures comprehension Rapaport 2005 . Despite this limitation, most experts agree it is indicative of understanding local vocabulary and semantic information Gellert and Elbro 2013 ; Carlisle and Rice 2004 . Many current language models pretrain on a masked language modeling objective, a form of the cloze task. Previous research has established that, for one such model, RoBERTa, prediction ability is correlated with human uncertainty Jacobs et al. 2022 . Learning Unknown Word Representations Previous work on learning representations of nonce and unknown words gives insight into how models may narrow down semantic space based on context. Nonce2Vec learns embeddings for unknown words from context and achieves high performance on a definitions dataset, but does not perform well with naturally occurring language. The authors hypothesize that adjusting risks taken during learning based on the informativeness of a context would improve results for naturally occurring language Herbelot and Baroni 2017 . Schick and Schütze 2019 utilize two approaches — (a) the surface-form representation (subword n-grams) and (b) learning an embedding from its context — increasing performance compared to using either of the two approaches alone. Evaluating Contextual Informativeness Two pieces of prior work also focus on the automatic evaluation of contextual informativeness. The first formalizes the task and introduces a crowd-sourced dataset that uses a Likert scale as a gold standard for contextual informativeness scores Kapelner et al. 2018 . The second work experiments with this dataset and proposes an attention-based model to create vector representations of both the word and its surrounding context Nam et al. 2022 , which provide the basis for predicting informativeness scores. This model achieves strong performance on adult-directed data which includes a single target word in each passage. 3 Task and Data In this section, we describe the creation of our dataset and formally introduce the task of measuring contextual informativeness in children’s stories . 3.1 Dataset Our dataset builds on Valentini et al. 2023 , which contains 180 LLM-generated children’s stories. Each story utilizes five target vocabulary words selected based on age of acquisition which the LLMs have been tasked to include. We annotate how much contextual support is provided for each target word. Our annotation schema is a modified version of the cloze task, in which annotators fill in blanks with the words they believe best complete a story. As the stories from Valentini et al. 2023 each contain five target words, all target words are replaced with blanks labeled 1 to 5 to simulate a child’s incomprehension of the unknown targets. Annotators guess the missing word for each number; there may be one or more blanks for each number. The cloze task traditionally only accepts one