Social Bias in LLMs and VLMs

This curated list brings together key methods, datasets, and benchmarks related to social bias in LLMs and VLMs.

As multimodal models continue to advance, understanding, evaluating, and mitigating social bias has become an increasingly important research direction. This repository is intended as a practical index for researchers and engineers to quickly track relevant work and recent progress.

Contributions are welcome—feel free to open a Pull Request to add high-quality resources.

Table of Contents

• Social Bias in LLMs and VLMs
  • Foundations
  • Survey
  • Data & Benchmarks
  • Mitigation
  • Findings

Foundations

Sociology

Title	Introduction	Date	Code
Getting a Job: Is There a Motherhood Penalty?	Motherhood Penalty	2007-03	-
A Meta-Analytic Test of Intergroup Contact Theory	Contact Works	2006-05	-
Are Emily and Greg More Employable Than Lakisha and Jamal? A Field Experiment	The Quality Paradox	2004-09	Code
Measuring Individual Differences in Implicit Cognition: The Implicit Association Test	Latency is the Key	1998-06	-
Rethinking Racism: Toward a Structural Interpretation	Racialized Social System	1997-06	-
Opposition to Race-Targeting: Self-Interest, Stratification Ideology, or Racial Attitudes?	Self-Interest is weak	1993-08	-
Understanding Everyday Racism: An Interdisciplinary Theory	Racism is Routine	1991-07	-
Race Prejudice as a Sense of Group Position	Group-level “sense of Position”	1958-03	-
Measuring Social Distances	Degree of Intimacy	1925-03	-

NLP (Pre-LLM)

Title	Introduction	Date	Code
Language (Technology) is Power: A Critical Survey of “Bias” in NLP	-	2020-07	-
Balanced Datasets Are Not Enough: Estimating and Mitigating Gender Bias in Deep Image Representations		2019-10	Github

Survey

LLM Surveys

Title	Introduction	Date	Code
Bias and Fairness in Large Language Models: A Survey		2024-09	Github
Fairness in Large Language Models: A Taxonomic Survey		2024-07	-
An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models	-	2022-05	Github

VLM Surveys

Title	Introduction	Date	Code
Fairness in Deep Learning: A Survey on Vision and Language Research		2025-02	-
Survey of Social Bias in Vision-Language Models		2023-09	-

Data & Benchmarks

LLMs

Title	Introduction	Date	Code
Performance unfairness of large language models in cross-language fact-checking		2026-06	-
How Quantization Shapes Bias in Large Language Models		2026-01	Github
What’s Not Said Still Hurts: A Description-Based Evaluation Framework for Measuring Social Bias in LLMs		2025-09	Github
Investigating Intersectional Bias in Large Language Models using Confidence Disparities in Coreference Resolution		2025-08	Github
MALIBU Benchmark: Multi-Agent LLM Implicit Bias Uncovered		2025-04	-
FLEX: A Benchmark for Evaluating Robustness of Fairness in Large Language Models		2025-04	Github
Evaluating and Mitigating Discrimination in Language Model Decisions		2024-12	-
Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactions		2024-10	Github
Ask LLMs Directly, “What shapes your bias? ”: Measuring Social Bias in Large Language Models		2024-08	Github(BBQ)
CHBias: Bias Evaluation and Mitigation of Chinese Conversational Language Models		2023-05	Github
BBQ: A Hand-Built Bias Benchmark for Question Answering		2022-05	Github

VLMs

Title	Introduction	Date	Code
BBQ-V: Benchmarking Visual Stereotype Bias in Large Multimodal Models		2026-01	Github
Vision Language Models are Biased		2026-01	Code
VISBIAS: Measuring Explicit and Implicit Social Biases in Vision Language Models		2025-11	Github
Evaluating Fairness in Large Vision-Language Models Across Diverse Demographic Attributes and Prompts		2025-11	-
VIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Models		2025-05	Github
HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation		2025-05	Github
MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Models		2024-12	Code
ModSCAN: Measuring Stereotypical Bias in Large Vision-Language Models from Vision and Language Modalities		2024-11	Github
FMBench: Benchmarking Fairness in Multimodal Large Language Models on Medical Tasks		2024-10	-
Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Search		2021-11	Github

Mitigation

Title	Introduction	Date	Code
Observations and Remedies for Large Language Model Bias in Self-Consuming Performative Loop		2026-01	-
Open-DeBias: Toward Mitigating Open-Set Bias in Language Models		2025-11	Resource
Debiasing Reward Models by Representation Learning with Guarantees		2025-10	-
Robustly Improving LLM Fairness in Realistic Settings via Interpretability		2025-09	Github
Guiding LLM Decision-Making with Fairness Reward Models		2025-09	Github
Wisdom from Diversity: Bias Mitigation Through Hybrid Human-LLM Crowds		2025-08	-
Joint Vision-Language Social Bias Removal for CLIP		2025-06	Github
Prompt Tuning Pushes Farther, Contrastive Learning Pulls Closer: A Two-Stage Approach to Mitigate Social Biases		2023-07	-
BLIND: Bias Removal With No Demographics		2023-07	Github
Don’t Just Clean It, Proxy Clean It: Mitigating Bias by Proxy in Pre-Trained Models		2022-12	Data: WIKI Madlibs BIOS
Fairness without Demographics through Adversarially Reweighted Learning		2020-11	-

Findings

Title	Introduction	Date	Code
Biases in the Blind Spot: Detecting What LLMs Fail to Mention		2026-02	-
FairReason: Balancing Reasoning and Social Bias in MLLMs		2025-09	Github
Evaluating Sex and Age Biases in Multimodal Large Language Models for Skin Disease Identification from Dermatoscopic Images		2025-04	-
Systematic Biases in LLM Simulations of Debates		2024-12	-
Large Language Models are not Fair Evaluators		2024-07	Github
Data Feedback Loops: Model-driven Amplification of Dataset Biases		2023-07	Github
Upstream Mitigation Is Not All You Need: Testing the Bias Transfer Hypothesis in Pre-Trained Language Models		2022-05	Adapted code: HurtfulWords sent_debias biosbias unintended-ml-bias-analysis roberta-base
Challenges in Automated Debiasing for Toxic Language Detection		2021-04	Github

---

Disclaimer

This repository is intended for educational and research purposes only. The images and figures displayed in this repository are the property of their respective authors and publishers and are used here for illustrative purposes to facilitate quick browsing and understanding of the papers.

If you are a copyright holder and would like any content removed, please open an issue, and I will remove it promptly.