Attività seminariali
Integrating Large Language Models for Political Content Analysis: A Validated Pipeline for Studying Italian Elections on Facebook
Abstract dell'evento
This presentation introduces a comprehensive methodological framework for analyzing political content on social media using Large Language Models (LLMs). Our study developed and validated a three-stage LLM-integrated pipeline to analyze approximately 85,000 news articles shared on Facebook during the 2018 and 2022 Italian general elections. The methodology combines supervised and unsupervised learning approaches through three key components: a fine-tuned political content classifier (achieving an F1 score of 0.897), an embedding-based clustering system using OpenAI's text-embedding-3-large model, and an automated cluster labeling process using GPT-4. Our validation protocol addresses critical methodological challenges including model selection, clustering optimization, and the assessment of cluster coherence. For clustering validation, we implemented a human expert assessment protocol requiring >80% pair coherence in stratified samples, while cluster labels were validated to achieve >70% good fit ratings. The study particularly highlights three fundamental challenges in LLM-based research: the Swiss Army Knife Dilemma of validating general-purpose tools, the Granularity Spectrum Problem in narrative clustering, and the Expertise Paradox in human validation of LLM outputs. This research not only advances our understanding of digital political communication but also establishes a rigorous methodological framework for social scientists integrating AI tools into their research workflows. Our findings demonstrate both the potential and limitations of LLM-based methodologies in social science research, while providing practical guidelines for ensuring research validity and reliability.
Programma
Wednesday, January 15th, 2025 h 11 a.m. - 1 p.m. IN PERSON AND ONLINE
Seminar: Integrating Large Language Models for Political Content Analysis: A Validated Pipeline for Studying Italian Elections on Facebook
Welcome and Introduction: Angela Genova (University of Urbino)
Speaker: Fabio Giglietto (University of Urbino)
Students and scholars who are unable to join the event in person in Sala del Consiglio (Palazzo Battiferri - via Saffi, 42) can attend online at the following Zoom link: https://bit.ly/Seminar_Giglietto_PhDGlobalStudies
Relatori/Relatrici
Fabio Giglietto (University of Urbino)
Dettagli sull'evento
Data e luogo
Inizio: 15/01/2025
alle ore 11:00
Fine: 15/01/2025
alle ore 13:00
Palazzo Battiferri (Urbino, Via Saffi, 42) Sc - Sala Consiglio
Organizzato e promosso da:
XXXVIII - Global Studies. Economy, Society and Law
Modalità di partecipazione
Altre informazioni utili
ingresso libero
