AnthroScore: A Computational Linguistic Measure of Anthropomorphism

Myra Cheng, Kristina Gligoric, Tiziano Piccardi, Dan Jurafsky

EACL 2024 Main Conference

Anthropomorphism, or the attribution of human-like characteristics to non-human entities, has shaped conversations about the impacts and possibilities of technology. We present AnthroScore, an automatic metric of implicit anthropomorphism in language. AnthroScore is lexicon-free and can be directly used on any text.

arXiv, source code

Try it out!

Note: This is the average AnthroScore in the text across all mentions of the entities. In a longer text, high AnthroScore of a single entity may be obscured by low AnthroScore of others. To see AnthroScore for all instances at once, use the source code!

Score Interpretation

The score A implies that the entity is e^A times more likely to be implicitly framed as human than as non-human in the context of the sentence (e is the log base), e.g.:

A = 0 means that the entity is equally likely to be implicitly framed as either human or non-human
A = 1 means that the entity is e¹ ≈ 2.7 times more likely to be implicitly framed as human than as non-human

Example Sentences

S↑: Sentences with high AnthroScore (A > 1)	S↓: Sentences with low AnthroScore (A < −1)
• When a job arrives, the system must decide whether to admit it or reject it, and if admitted, in which server to schedule the job. • Meanwhile, anti-forensic attacks have been developed to fool these CNN-based forensic algorithms. • The models demonstrated qualifications in various computer-related fields, such as cloud and virtualization, business analytics, cybersecurity, network setup...	• More and more users and developers are using Issue Tracking Systems to report issues, including bugs, featurerequests, enhancement suggestions, etc. • Our approach delivers forecast improvements over a competitive benchmark and we discover evidence for strong spatial interactions. • To this end, for training the model, we convert the knowledge graph triples into reasonable and unreasonable texts.
• Large language models don’t actually think and tend to make elementary mistakes, even make things up. • The algorithms also picked up on racial biases linking Black people to weapons. • The AI system was able to defeat human players in...	• Microsoft is betting heavily on integrating OpenAI’s GPT language models into its products to compete with Google. • Deepmind has been the pioneer in making AI models that have the capability to mimic a human’s cognitive... • For workers who use machine-learning models to help them make decisions, knowing when to...

S↑: Sentences with high AnthroScore (A > 1)

S↓: Sentences with low AnthroScore (A < −1)

• When a job arrives, the system must decide whether to admit it or reject it, and if admitted, in which server to schedule the job.
• Meanwhile, anti-forensic attacks have been developed to fool these CNN-based forensic algorithms.
• The models demonstrated qualifications in various computer-related fields, such as cloud and virtualization, business analytics, cybersecurity, network setup...

• More and more users and developers are using Issue Tracking Systems to report issues, including bugs, featurerequests, enhancement suggestions, etc.
• Our approach delivers forecast improvements over a competitive benchmark and we discover evidence for strong spatial interactions.
• To this end, for training the model, we convert the knowledge graph triples into reasonable and unreasonable texts.

• Large language models don’t actually think and tend to make elementary mistakes, even make things up.
• The algorithms also picked up on racial biases linking Black people to weapons.
• The AI system was able to defeat human players in...

• Microsoft is betting heavily on integrating OpenAI’s GPT language models into its products to compete with Google.
• Deepmind has been the pioneer in making AI models that have the capability to mimic a human’s cognitive...
• For workers who use machine-learning models to help them make decisions, knowing when to...

How does AnthroScore work?

Given an entity in a text, we use RoBERTa (a masked language model) to compute the probability that the entity would be replaced by human pronouns versus non-human pronouns. The log-ratio of these probabilities can be interpreted as how likely the entity is implicitly framed as human by the surrounding context.

Our Findings

Motivated by concerns of misleading anthropomorphism, we use AnthroScore to analyze 15 years of computer science, statistics, and computational linguistics research papers and downstream news articles.

Papers related to NLP & language models have the most anthropomorphism.
Within ACL papers, temporal increases in anthropomorphism are correlated with key neural advancements.
Building upon concerns of scientific misinformation in mass media, news headlines have more anthropomorphism than the research papers they cite.

Check out the full paper for more details!

Citation:

Myra Cheng, Kristina Gligoric, Tiziano Piccardi, and Dan Jurafsky. 2024. AnthroScore: A Computational Linguistic Measure of Anthropomorphism. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, Malta. Association for Computational Linguistics.

BibTeX:


@inproceedings{cheng-etal-2024-anthroscore,
    title = "{A}nthro{S}core: A Computational Linguistic Measure of Anthropomorphism",
    author = "Cheng, Myra  and
      Gligoric, Kristina  and
      Piccardi, Tiziano and
      Jurafsky, Dan",
    booktitle = "Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics",
    month = march,
    year = "2024",
    address = "Malta",
    publisher = "Association for Computational Linguistics"
}