Assessing factuality in Spanish: do experts and non-experts agree?

Authors

DOI:

https://doi.org/10.58859/resla.1545

Keywords:

corpus, factuality markers, polarity, news, Spanish

Abstract

Event veridicality and factuality are key to natural language understanding. Deciding whether the events mentioned in a text are viewed as happening or not is one of the key features in fake news detection. That is one of the reasons why factuality has been widely studied and enormous efforts have been made to annotate corpora and develop tools that automatically identify factuality and veridicality. Most of the studies carried out use English corpora annotated by non-experts; however, little has been said about factuality in Spanish and on how appropriate the use of non-expert annotators is.

Based on the discussion above, this study aims to a) shed some light on how non-expert Spanish speakers interpret the various factuality markers, b) determine if the age of the informants is a relevant factor when assessing the degree of factuality, and c) explore whether the experts and the non-expert’s interpretations differ.

Global Statistics ℹ️

Cumulative totals since publication
22
Views
4
Downloads
26
Total

Author Biography

Elisabet Comelles, University of Barcelona

Elisabet Comelles is a lecturer at the Department of Modern Languages and Literature and of English Studies at the University of Barcelona. She's a member of the GRIAL group (Linguistic Applications Inter-University Research Group) and collaborates with the GRELIC group (Lexicology and Corpus Linguistics Research Group).

Her research is focused on computational linguistics and natural language processing, as well as on the use of corpus linguistics and language technologies in the learning and teaching of English. The outcomes of her research have been published in leading international journals and specialized books. She has also taken part in several national and European projects.

References

Alonso, L., Castellón, I., Curell, H., Fernández-Montraveta, A., Oliver, S., Vázquez, G. (2018). Proyecto TAGFACT: Del texto al conocimiento. Factualidad y grados de certeza en español, Procesamiento del Lenguaje Natural, 61, p. 151-154. http://dx.doi.org/10.26342/2018-61-20

Barrios, L. (2022). La factualidad en las oraciones adversativas, concesivas y condicionales en español: El papel de los tiempos verbales en la anotación automática de corpus. [Doctoral Dissertation, Universitat de Lleida]. https://www.tdx.cat/handle/10803/673898#page=1

Box, G.E., Hunter, J.S., & Hunter, W.G. (2008). Estadística para Investigadores. Editorial Reverté.

Colomer, J.P., Keyu , L., & Rambow. O. (2016). Detecting level of belief in Chinese and Spanish. In Proceedings of the Workshop on Extra-Propositional Aspects of Meaning in Computational Linguistics (ExProM), 22-30. https://aclanthology.org/W16-5003.pdf

de Marneffe, M. C., Manning, C. D., & Potts, C. (2012). Did It Happen? The Pragmatic Complexity of Veridicality Assessment. Computational Linguistics, 38(2), 301–333. DOI: 10.1162/COLI_a_00097 DOI: https://doi.org/10.1162/COLI_a_00097

Fernández-Montraveta, A., Vàzquez, G., Curell, H., & Castellón, I. (2020a). The TAGFACT annotator and editor: A versatile tool. Research in Corpus Linguistics 8, 131-146. http://dx.doi.org/10.32714/ricl.08.01.08 DOI: https://doi.org/10.32714/ricl.08.01.08

Lee, K., Artzi, Y., Choi, Y., & Zettlemoyer, L. (2015). Event Detection and Factuality Assessment with Non-Expert Supervision. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 1643–1648. DOI: 10.18653/v1/D15-1189 DOI: https://doi.org/10.18653/v1/D15-1189

Martín, F. J. & Ruiz-Maya, L. (2001). Estadística II: Inferencia. Libros Aula Magna. Alfa Centauro.

Miller, B. (2020). Reading Certainty: Evidence from a Large Study on NLP and Witness Testimony. Digital Humanities 2020 Conference. July 20-24, 2020.

Monge, J.F., & Juan, A.A. (2002). Estadística no paramétrica: Prueba Chi-cuadrado. Proyecto e-Math. ET-M05. Universitat Oberta de Catalunya.

Peña, D. (1988). Estadística, Modelos y Métodos. Editorial Alianza Universidad.

Prabhakaran, V., By, T., Hirschberg, J., Rambow, O., Shaikh, S., Strzalkowski, T., Tracey, J., Arrigo, M., Basu, R., Clark, M., Dalton, A., Diab, M., Guthrie, L., Prokofieva, A., Strassel, S., Werner, G., Wiebe, J., & Wilks, Y. (2015). A New Dataset and Evaluation for Belief/Factuality. In Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics (*SEM 2015), 82-91. DOI: 10.18653/v1/S15-1009 DOI: https://doi.org/10.18653/v1/S15-1009

Rosá, A., Alonso, L., Castellón, I., Chiruzzo, L., Curell, H., Fernández-Montraveta, A., Góngora, S., Malcouri, M., Vázquez, G., & Wonsever, D. (2020b). Overview of FACT at IBERLEF 2020: Events detection and classification. In CEUR Workshop Proceedings, 2664, 197-205. https://ceur-ws.org/Vol-2664/fact_overview.pdf

Rudinger, R., White, A.S., & Van Durme, B. (2018). Neural Models of Factuality. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 731–744. DOI: 10.18653/v1/N18-1067 DOI: https://doi.org/10.18653/v1/N18-1067

Saurí, R. (2008). A Factuality Profiler for Eventualities in Text. [Doctoral Dissertation, Brandeis University] https://www.cs.brandeis.edu/~roser/pubs/sauriDiss_1.5.pdf

Saurí, R., & Pustejovsky, J. (2009). FactBank: a corpus annotated with event factuality. Language Resources and Evaluation 43, 227–268. https://doi.org/10.1007/s10579-009-9089-9 DOI: https://doi.org/10.1007/s10579-009-9089-9

Saurí, R., & Pustejovsky, J. (2012). Are You Sure That This Happened? Assessing the Factuality Degree of Events in Text. Computational Linguistics, 38(2), 261–299. DOI: 10.1162/COLI_a_00096 DOI: https://doi.org/10.1162/COLI_a_00096

Soni, S., Mitra, T., Gilbert, E., & Eisenstein, J. (2014). Modeling Factuality Judgments in Social Media Text. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 415–420. DOI: 10.3115/v1/P14-2068 DOI: https://doi.org/10.3115/v1/P14-2068

Stanovsky, G., Eckle-Kohler, J., Puzikov, Y., Dagan, I., & Gurevych, I. (2017). Integrating Deep Linguistic Features in Factuality Prediction over Unified Datasets. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 352–357. DOI: 10.18653/v1/P17-2056 DOI: https://doi.org/10.18653/v1/P17-2056

Tonelli, S., Sprugnoli, R., Speranza, M. & Minard, A. (2014). NewsReader Guidelines for Annotation at Document Level. Extension of Deliverable D3. Technical Report NWR- 2014-2. Trento.

Tracey, J., Arrigo, M., & Strassel, S. (2019). DEFT Spanish Commited Belief Annotation. Linguistic Data Consortium LDG2019T09. Web download. Philadelphia: Linguistic Data Consortium. https://catalog.ldc.upenn.edu/LDC2019T09

Tracey, J., Rambow, O., Cardie, C., Dalton, A., Trang Dang, H., Diab, M., Dorr, B., Guthrie, L., Markowska, M., Muresan, S., Prabhakaran, V., Shaikh, S.,

& Strzalkowski, T. (2022). BeSt: The Belief and Sentiment Corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, 2460–2467. https://aclanthology.org/2022.lrec-1.262 DOI: https://doi.org/10.63317/4run4u9jq8mx

Uzzaman, N., Llorens, H., Derczynski, L., Verhagen, M., Allen, J., & Pustejovsky, J. (2013). Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), 1–9.

van Son, C., van Erp, M., Fokkens, A., and Vossen, P. (2014). Hope and Fear: Interpreting Perspectives by Integrating Sentiment and Event Factuality. In Proceedings of the 9th Language Resources and Evaluation Conference, 3857-3864. DOI: https://doi.org/10.63317/2voodkchtwdx

White, A. S., Reisinger, D., Sakaguchi, K., Vieira, T., Zhang, S., Rudinger,R., Rawlins, K., & Van Durme, J. (2016). Universal decompositional semantics on universal dependencies. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 1713–1723. DOI: https://doi.org/10.18653/v1/D16-1177

Wonsever, D., Aiala, R. & Malcuori, M. (2016). Factuality Annotation and Learning in Spanish Texts. In Proceedings of the 10th Edition of the Language Resources and Evaluation Conference, 2076-2080. DOI: https://doi.org/10.63317/3zbqbkmidsff

Downloads

Published

2026-11-15

How to Cite

Comelles, E., Curell, H., Aparicio Mera, J., & Castellón Massalles, I. (2026). Assessing factuality in Spanish: do experts and non-experts agree?. Revista Española De Lingüística Aplicada Spanish Journal of Applied Linguistics, 39(2). https://doi.org/10.58859/resla.1545

Similar Articles

1 2 3 > >> 

You may also start an advanced similarity search for this article.