AA-PersonalityBench: how sycophantic are AI models?

What is it?

On October 19, Artificial Analysis launched AA-PersonalityBench, a benchmark for how AI models communicate (Artificial Analysis). It does not grade whether an answer is right, but how strongly a given trait shows in it. The launch results cover 38 models.

How it measures

• 30 personality traits with 30 questions each, 900 questions in total. Models answer with no system prompt, so the results reflect default behavior.
• A panel of LLM judges compares answers pairwise. Judges do not see model names, and answer order is randomized.
• Scores are Elo ratings converted to percentile ranks from 1 to 100. The traits roll up into 8 indexes, such as Transparency, Warmth and Slop.
• It also measures 21 communication characteristics programmatically, such as response length, structure and emphasis.

What sycophancy shows

The Sycophantic trait asks whether a model excessively flatters or agrees with the user instead of answering honestly (Artificial Analysis). On the current leaderboard, which has since grown to 40 models, the November 2024 version of GPT-4o is the most sycophantic and Grok 4.7 (xhigh) the least. The gap between them is 765 Elo.

Sycophancy counts inversely in the Transparency index. Claude Sonnet 5.5 leads that index, and GPT-3.5 Turbo ranks last.

Why it matters

A flattering model feels pleasant but can lead to bad decisions: it endorses weak ideas and stays quiet about mistakes. Until now, comparisons were mostly anecdotal; now there is a metric. LLM judges do the scoring, so results may not match human impressions. If you are picking a model for a support or advisory agent, check the Transparency index too.