FairStress is a benchmark designed to measure demographic fairness in LLMs by evaluating how consistently models defend well-supported decisions against adversarial pressure. Unlike traditional benchmarks that focus only on final outcomes, this approach uncovers hidden asymmetries in how models change their decisions based on demographic identity. The research demonstrates that even when models select the same candidate across scenarios, they often exhibit inconsistent resistance to persuasion.
Key points
Outcome-based fairness metrics are insufficient because they fail to capture how easily a model's decision can be swayed by biased arguments.
Models frequently show asymmetric resistance to changing their decisions when presented with identical adversarial prefixes that favor different demographic groups.
Internal activation patterns and model explanations often reveal latent demographic bias even when final output choices appear consistent.
Defining fairness as symmetric resistance to change allows for more robust evaluation of decision-making stability in high-stakes environments.