Telegram Message Deletion Affects Study Results

Summary

The study scrutinized data from Telegram, emphasizing the impact of message deletion on the platform. It explored how message deletion compromises dataset integrity and reliability and alters the outcomes of computational analyses.

Figure 1: Computational dictionary analyses of deleted and nondeleted messages. The bars depict the relative share of messages and their respective category in the “Not deleted” and “Deleted” subsamples.

Using a series of messages from public Telegram entities, the research repeatedly sampled and analyzed the data and discovered that message deletion introduces significant biases in both data collection and analysis processes. The deletion of a considerable number of text messages from group chats skewed distributions and potentially impacted the accuracy of computational content analysis methods, highlighting the need for careful interpretation of results from such platforms.

Key Findings

• Message Deletion & Biases: Message deletion on Telegram introduces biases to the computational collection and analysis of data, affecting the consistency of datasets, the quality of social network analyses, and the results of computational content analysis methods.

• Disproportionate Prone to Deletion: The study identified that different data types are disproportionately prone to deletion on Telegram, impacting the reliability of the collected data.

• Effects of Message Ephemerality: Message ephemerality reduces dataset consistency, making it challenging to obtain complete and reliable data for analysis, thus impacting the outcomes of content analysis using computational methods such as topic modeling and dictionaries.

• Impact on Computational Methods: Message deletion has significant effects, and it can introduce biased results in computational content analysis methods, affecting the reliability of studies utilizing these methods.

Implications

• Awareness of Deletion Impact: Researchers need to be cognizant of the extent of message deletion on Telegram and its potential impact on their analysis. Understanding the limitations and biases introduced by missing data is crucial for accurate analysis.

• Caution with Analysis Methods: Computational content analysis and social network analysis methods should be approached with caution when analyzing Telegram data due to the potential biases and inaccuracies introduced by message deletion.

• Collaborative Practices: Collaborative practices of privacy-sensitive data sharing among scholars using Telegram data are essential to address the issues of missing data and to improve the reliability and replicability of studies.

• Strategic Approach to Data Collection: Researchers should strategically approach data collection and analysis on platforms like Telegram, considering the inherent challenges and biases, to ensure the validity and reliability of their findings.

• Impact on Computational Methods: Message deletion has significant effects, and it can introduce biased results in computational content analysis methods, affecting the reliability of studies utilizing these methods.

Source

Annett Heft, Kilian Buehling, Xixuan Zhang, Dominik Schindler & Miriam Milzner. (2023) Challenges of and approaches to data collection across platforms and time: Conspiracy-related digital traces as examples of political contention. Journal of Information Technology & Politics 0:0, pages 1-17.