Clasificación

Actualizado el 19 de septiembre de 2026

Rendimiento de mediadores de IA en 62 conflictos simulados

Las tablas, los gráficos y sus controles se mantienen en inglés.

Current leaderboard · updated 2026-10-19

AI mediator scores · 0–100

Provisional · ordered within each version

Current test · 62 simulated conflicts

Each result is one complete run, scored across all 62 conflicts by Sol and by any other judge that covers all 62. One or two judges make a result provisional. Hover over a row for its judges and endings, or open it to keep them in view. Instrument validation pending.

Benchmark version

The same 62 conflicts and scoring rubric. Version 3.1 adds bounded judge retries and reliability reporting. Results are ordered within each version while the effect of the retry change is assessed.

Version 3.0

  1. 1 Claude Opus 5 · Gemma / Novita BF16 · Sol / Azure Benchmark version 3.0 simple prompt Score 71.2 Judges 2 Ended appropriately 55/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 67.5
    • Gemini · 62/62Score 74.9
    Ended appropriately: 55 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 4.6 rounds on average. 0 of 62 reached the 25-round limit; 0 ended at round 24 or later.

    What Sol found at the end

    • 50 supported agreements
    • 3 useful resolutions
    • 2 reasonable endings without resolution
    • 7 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 58 of 62 conversations. Sol agreed in 50 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 228 of 55092 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 4.6 rounds. Round 3: 17 · Round 4: 17 · Round 5: 15 · Round 6: 5 · Round 7: 4 · Round 8: 3 · Round 10: 1

    Moderator ending claims compared with Sol
    • Claimed agreement: 50 supported agreements, 3 useful resolutions, 0 reasonable endings without resolution, 5 too soon, 0 unclear.
    • Claimed useful resolution: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 0 useful resolutions, 2 reasonable endings without resolution, 2 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 67.5 Evaluator recipe: 269d705430e76bd2f9aea7863510a110f8debb70a5bde654016dbe762b553772 Endings: 50 supported agreements, 3 useful resolutions, 2 reasonable endings without resolution, 7 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 9 found, 229 missed, 179 extra; recall 3.8%.
    • Gemini · score 74.9 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 56 supported agreements, 2 useful resolutions, 4 reasonable endings without resolution, 0 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 14 found, 224 missed, 100 extra; recall 5.9%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    93.5%
    Model errors
    4 of 66 responses
    Repeat calls
    4
    Assessments accepted
    62 of 62 observed
    Error tokens
    60,364
    Error cost
    $0.85
    Recorded tokens
    967,093
    Recorded cost
    $13.64
    Request time
    Not recorded
    Gemini
    Valid first response
    85.5%
    Model errors
    16 of 78 responses
    Repeat calls
    16
    Assessments accepted
    62 of 62 observed
    Error tokens
    165,544
    Error cost
    $0.37
    Recorded tokens
    858,523
    Recorded cost
    $1.95
    Request time
    Not recorded
  2. 2 Claude Opus 5.5 · Gemma / Novita BF16 · Sol / Azure Benchmark version 3.0 xhigh · simple prompt Score 70.3 Judges 2 Ended appropriately 58/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 67.4
    • Gemini · 62/62Score 73.1
    Ended appropriately: 58 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 5.4 rounds on average. 0 of 62 reached the 25-round limit; 0 ended at round 24 or later.

    What Sol found at the end

    • 52 supported agreements
    • 1 useful resolutions
    • 5 reasonable endings without resolution
    • 4 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 57 of 62 conversations. Sol agreed in 52 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 383 of 55066 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 5.4 rounds. Round 3: 6 · Round 4: 18 · Round 5: 15 · Round 6: 9 · Round 7: 5 · Round 8: 4 · Round 9: 3 · Round 11: 1 · Round 13: 1

    Moderator ending claims compared with Sol
    • Claimed agreement: 52 supported agreements, 1 useful resolutions, 0 reasonable endings without resolution, 4 too soon, 0 unclear.
    • Claimed useful resolution: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 0 useful resolutions, 5 reasonable endings without resolution, 0 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 67.4 Evaluator recipe: 269d705430e76bd2f9aea7863510a110f8debb70a5bde654016dbe762b553772 Endings: 52 supported agreements, 1 useful resolutions, 5 reasonable endings without resolution, 4 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 12 found, 226 missed, 202 extra; recall 5.0%.
    • Gemini · score 73.1 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 54 supported agreements, 3 useful resolutions, 5 reasonable endings without resolution, 0 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 11 found, 227 missed, 112 extra; recall 4.6%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    95.2%
    Model errors
    5 of 67 responses
    Repeat calls
    5
    Assessments accepted
    62 of 62 observed
    Error tokens
    80,241
    Error cost
    $1.14
    Recorded tokens
    1,014,949
    Recorded cost
    $14.19
    Request time
    Not recorded
    Gemini
    Valid first response
    85.5%
    Model errors
    12 of 74 responses
    Repeat calls
    12
    Assessments accepted
    62 of 62 observed
    Error tokens
    131,825
    Error cost
    $0.27
    Recorded tokens
    856,729
    Recorded cost
    $1.91
    Request time
    Not recorded
  3. 3 Claude Fable 5.1 · Gemma / Novita BF16 · Sol / Azure Benchmark version 3.0 xhigh · simple prompt Score 69.9 Judges 2 Ended appropriately 51/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 67.3
    • Gemini · 62/62Score 72.4
    Ended appropriately: 51 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 4.3 rounds on average. 0 of 62 reached the 25-round limit; 0 ended at round 24 or later.

    What Sol found at the end

    • 46 supported agreements
    • 2 useful resolutions
    • 3 reasonable endings without resolution
    • 11 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 55 of 62 conversations. Sol agreed in 46 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 265 of 43787 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 4.3 rounds. Round 3: 16 · Round 4: 25 · Round 5: 12 · Round 6: 7 · Round 7: 2

    Moderator ending claims compared with Sol
    • Claimed agreement: 46 supported agreements, 1 useful resolutions, 0 reasonable endings without resolution, 8 too soon, 0 unclear.
    • Claimed useful resolution: 0 supported agreements, 1 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 0 useful resolutions, 3 reasonable endings without resolution, 3 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 67.3 Evaluator recipe: 269d705430e76bd2f9aea7863510a110f8debb70a5bde654016dbe762b553772 Endings: 46 supported agreements, 2 useful resolutions, 3 reasonable endings without resolution, 11 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 11 found, 227 missed, 194 extra; recall 4.6%.
    • Gemini · score 72.4 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 53 supported agreements, 2 useful resolutions, 5 reasonable endings without resolution, 2 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 12 found, 226 missed, 92 extra; recall 5.0%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    95.2%
    Model errors
    6 of 68 responses
    Repeat calls
    6
    Assessments accepted
    62 of 62 observed
    Error tokens
    89,797
    Error cost
    $1.31
    Recorded tokens
    969,299
    Recorded cost
    $13.86
    Request time
    Not recorded
    Gemini
    Valid first response
    88.7%
    Model errors
    9 of 71 responses
    Repeat calls
    9
    Assessments accepted
    62 of 62 observed
    Error tokens
    104,661
    Error cost
    $0.26
    Recorded tokens
    766,846
    Recorded cost
    $1.79
    Request time
    Not recorded
  4. 4 Kimi K3 · Gemma / Novita BF16 · Sol / Azure Benchmark version 3.0 OpenRouter · simple prompt Score 68.7 Judges 2 Ended appropriately 48/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 64.8
    • Gemini · 62/62Score 72.7
    Ended appropriately: 48 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 5.0 rounds on average. 0 of 62 reached the 25-round limit; 0 ended at round 24 or later.

    What Sol found at the end

    • 45 supported agreements
    • 1 useful resolutions
    • 2 reasonable endings without resolution
    • 14 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 56 of 62 conversations. Sol agreed in 44 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 250 of 46364 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 5.0 rounds. Round 3: 16 · Round 4: 9 · Round 5: 18 · Round 6: 7 · Round 7: 7 · Round 8: 2 · Round 9: 3

    Moderator ending claims compared with Sol
    • Claimed agreement: 44 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 12 too soon, 0 unclear.
    • Claimed useful resolution: 1 supported agreements, 1 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 0 useful resolutions, 2 reasonable endings without resolution, 2 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 64.8 Evaluator recipe: 269d705430e76bd2f9aea7863510a110f8debb70a5bde654016dbe762b553772 Endings: 45 supported agreements, 1 useful resolutions, 2 reasonable endings without resolution, 14 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 7 found, 231 missed, 200 extra; recall 2.9%.
    • Gemini · score 72.7 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 53 supported agreements, 4 useful resolutions, 4 reasonable endings without resolution, 1 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 7 found, 231 missed, 111 extra; recall 2.9%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    93.5%
    Model errors
    7 of 69 responses
    Repeat calls
    7
    Assessments accepted
    62 of 62 observed
    Error tokens
    123,250
    Error cost
    $1.77
    Recorded tokens
    1,025,201
    Recorded cost
    $14.44
    Request time
    Not recorded
    Gemini
    Valid first response
    90.3%
    Model errors
    7 of 69 responses
    Repeat calls
    7
    Assessments accepted
    62 of 62 observed
    Error tokens
    77,836
    Error cost
    $0.17
    Recorded tokens
    792,022
    Recorded cost
    $1.82
    Request time
    Not recorded
  5. 5 GLM 5.3 · Gemma / Novita BF16 · Sol / Azure Benchmark version 3.0 DeepInfra, FP4 · simple prompt Score 67.6 Judges 2 Ended appropriately 36/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 62.7
    • Gemini · 62/62Score 72.5
    Ended appropriately: 36 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 5.4 rounds on average. 0 of 62 reached the 25-round limit; 0 ended at round 24 or later.

    What Sol found at the end

    • 30 supported agreements
    • 6 useful resolutions
    • 0 reasonable endings without resolution
    • 26 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 60 of 62 conversations. Sol agreed in 30 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 210 of 37893 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 5.4 rounds. Round 3: 11 · Round 4: 13 · Round 5: 16 · Round 6: 5 · Round 7: 6 · Round 8: 7 · Round 9: 1 · Round 10: 1 · Round 11: 1 · Round 13: 1

    Moderator ending claims compared with Sol
    • Claimed agreement: 30 supported agreements, 6 useful resolutions, 0 reasonable endings without resolution, 24 too soon, 0 unclear.
    • Claimed useful resolution: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 1 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 1 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 62.7 Evaluator recipe: 269d705430e76bd2f9aea7863510a110f8debb70a5bde654016dbe762b553772 Endings: 30 supported agreements, 6 useful resolutions, 0 reasonable endings without resolution, 26 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 15 found, 223 missed, 186 extra; recall 6.3%.
    • Gemini · score 72.5 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 55 supported agreements, 5 useful resolutions, 0 reasonable endings without resolution, 2 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 12 found, 226 missed, 101 extra; recall 5.0%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    91.9%
    Model errors
    6 of 68 responses
    Repeat calls
    6
    Assessments accepted
    62 of 62 observed
    Error tokens
    89,439
    Error cost
    $1.31
    Recorded tokens
    983,467
    Recorded cost
    $13.97
    Request time
    Not recorded
    Gemini
    Valid first response
    85.5%
    Model errors
    10 of 72 responses
    Repeat calls
    10
    Assessments accepted
    62 of 62 observed
    Error tokens
    102,574
    Error cost
    $0.24
    Recorded tokens
    809,202
    Recorded cost
    $1.90
    Request time
    Not recorded
  6. 6 GPT-5.6 Sol · Gemma / Novita BF16 · Sol / Azure Benchmark version 3.0 Azure · high effort · simple prompt Score 66.9 Judges 2 Ended appropriately 58/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 66.1 Same provider lab as mediator
    • Gemini · 62/62Score 67.6
    Ended appropriately: 58 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 4.5 rounds on average. 0 of 62 reached the 25-round limit; 0 ended at round 24 or later.

    What Sol found at the end

    • 55 supported agreements
    • 0 useful resolutions
    • 3 reasonable endings without resolution
    • 4 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 56 of 62 conversations. Sol agreed in 54 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 50 of 27105 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 4.5 rounds. Round 3: 18 · Round 4: 17 · Round 5: 14 · Round 6: 7 · Round 7: 5 · Round 9: 1

    Moderator ending claims compared with Sol
    • Claimed agreement: 54 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 2 too soon, 0 unclear.
    • Claimed useful resolution: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    • Claimed no resolution: 1 supported agreements, 0 useful resolutions, 3 reasonable endings without resolution, 2 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 66.1 · same provider lab as mediator Evaluator recipe: 269d705430e76bd2f9aea7863510a110f8debb70a5bde654016dbe762b553772 Endings: 55 supported agreements, 0 useful resolutions, 3 reasonable endings without resolution, 4 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 11 found, 227 missed, 194 extra; recall 4.6%.
    • Gemini · score 67.6 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 50 supported agreements, 5 useful resolutions, 5 reasonable endings without resolution, 2 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 8 found, 230 missed, 101 extra; recall 3.4%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    98.4%
    Model errors
    1 of 63 responses
    Repeat calls
    1
    Assessments accepted
    62 of 62 observed
    Error tokens
    17,099
    Error cost
    $0.24
    Recorded tokens
    865,903
    Recorded cost
    $12.37
    Request time
    Not recorded
    Gemini
    Valid first response
    91.9%
    Model errors
    6 of 68 responses
    Repeat calls
    6
    Assessments accepted
    62 of 62 observed
    Error tokens
    58,146
    Error cost
    $0.14
    Recorded tokens
    701,506
    Recorded cost
    $1.64
    Request time
    Not recorded
  7. 7 Muse Spark 1.3 Benchmark version 3.0 Meta · simple prompt Score 66.8 Judges 2 Ended appropriately 59/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 63.3
    • Gemini · 62/62Score 70.3
    Ended appropriately: 59 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 8.4 rounds on average. 1 of 62 reached the 25-round limit; 1 ended at round 24 or later.

    What Sol found at the end

    • 55 supported agreements
    • 2 useful resolutions
    • 2 reasonable endings without resolution
    • 3 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 57 of 62 conversations. Sol agreed in 54 of those cases.

    The moderator chose when to stop in 61 conversations; 1 reached the limit instead.

    Wording overlap check: 153 of 34802 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 8.4 rounds. Round 4: 6 · Round 5: 8 · Round 6: 9 · Round 7: 10 · Round 8: 6 · Round 9: 6 · Round 10: 3 · Round 11: 3 · Round 12: 2 · Round 13: 1 · Round 14: 3 · Round 15: 3 · Round 19: 1 · Round 25: 1

    Moderator ending claims compared with Sol
    • Claimed agreement: 54 supported agreements, 1 useful resolutions, 0 reasonable endings without resolution, 2 too soon, 0 unclear.
    • Claimed useful resolution: 0 supported agreements, 1 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 0 useful resolutions, 2 reasonable endings without resolution, 1 too soon, 0 unclear.
    • No ending claim: 1 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 63.3 Evaluator recipe: 320b75fc6ed0209f9b39e2cbb8276e552b4b6bc182ca181f0ad2763dfdbe6ddb Endings: 55 supported agreements, 2 useful resolutions, 2 reasonable endings without resolution, 3 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 10 found, 228 missed, 190 extra; recall 4.2%.
    • Gemini · score 70.3 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 54 supported agreements, 5 useful resolutions, 3 reasonable endings without resolution, 0 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 10 found, 228 missed, 105 extra; recall 4.2%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    96.8%
    Model errors
    2 of 64 responses
    Repeat calls
    2
    Assessments accepted
    62 of 62 observed
    Error tokens
    37,797
    Error cost
    $0.48
    Recorded tokens
    975,997
    Recorded cost
    $13.39
    Request time
    Not recorded
    Gemini
    Valid first response
    85.5%
    Model errors
    13 of 75 responses
    Repeat calls
    13
    Assessments accepted
    62 of 62 observed
    Error tokens
    157,547
    Error cost
    $0.34
    Recorded tokens
    917,321
    Recorded cost
    $2.06
    Request time
    Not recorded
  8. 8 GLM 5.3 Flash Benchmark version 3.0 OpenRouter · simple prompt Score 66.0 Judges 2 Ended appropriately 42/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 61.6
    • Gemini · 62/62Score 70.4
    Ended appropriately: 42 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 5.5 rounds on average. 0 of 62 reached the 25-round limit; 0 ended at round 24 or later.

    What Sol found at the end

    • 31 supported agreements
    • 5 useful resolutions
    • 6 reasonable endings without resolution
    • 20 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 52 of 62 conversations. Sol agreed in 30 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 273 of 46250 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 5.5 rounds. Round 3: 10 · Round 4: 13 · Round 5: 16 · Round 6: 12 · Round 7: 3 · Round 9: 3 · Round 10: 3 · Round 12: 1 · Round 14: 1

    Moderator ending claims compared with Sol
    • Claimed agreement: 30 supported agreements, 4 useful resolutions, 0 reasonable endings without resolution, 18 too soon, 0 unclear.
    • Claimed useful resolution: 1 supported agreements, 1 useful resolutions, 0 reasonable endings without resolution, 1 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 0 useful resolutions, 6 reasonable endings without resolution, 1 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 61.6 Evaluator recipe: 320b75fc6ed0209f9b39e2cbb8276e552b4b6bc182ca181f0ad2763dfdbe6ddb Endings: 31 supported agreements, 5 useful resolutions, 6 reasonable endings without resolution, 20 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 11 found, 227 missed, 195 extra; recall 4.6%.
    • Gemini · score 70.4 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 49 supported agreements, 5 useful resolutions, 6 reasonable endings without resolution, 2 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 6 found, 232 missed, 115 extra; recall 2.5%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    79.0%
    Model errors
    22 of 84 responses
    Repeat calls
    22
    Assessments accepted
    62 of 62 observed
    Error tokens
    325,877
    Error cost
    $4.49
    Recorded tokens
    1,201,025
    Recorded cost
    $16.59
    Request time
    Not recorded
    Gemini
    Valid first response
    88.7%
    Model errors
    9 of 71 responses
    Repeat calls
    9
    Assessments accepted
    62 of 62 observed
    Error tokens
    105,522
    Error cost
    $0.24
    Recorded tokens
    828,436
    Recorded cost
    $1.92
    Request time
    Not recorded
  9. 9 GPT-5.6 Terra · Gemma / Novita BF16 · Sol / Azure Benchmark version 3.0 Azure · high effort · simple prompt Score 64.9 Judges 2 Ended appropriately 56/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 62.9 Same provider lab as mediator
    • Gemini · 62/62Score 66.8
    Ended appropriately: 56 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 5.5 rounds on average. 0 of 62 reached the 25-round limit; 0 ended at round 24 or later.

    What Sol found at the end

    • 43 supported agreements
    • 6 useful resolutions
    • 7 reasonable endings without resolution
    • 6 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 48 of 62 conversations. Sol agreed in 42 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 131 of 34651 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 5.5 rounds. Round 3: 11 · Round 4: 14 · Round 5: 18 · Round 6: 7 · Round 7: 5 · Round 8: 1 · Round 9: 2 · Round 10: 1 · Round 11: 1 · Round 15: 1 · Round 20: 1

    Moderator ending claims compared with Sol
    • Claimed agreement: 42 supported agreements, 1 useful resolutions, 0 reasonable endings without resolution, 5 too soon, 0 unclear.
    • Claimed useful resolution: 1 supported agreements, 5 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 0 useful resolutions, 7 reasonable endings without resolution, 1 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 62.9 · same provider lab as mediator Evaluator recipe: 269d705430e76bd2f9aea7863510a110f8debb70a5bde654016dbe762b553772 Endings: 43 supported agreements, 6 useful resolutions, 7 reasonable endings without resolution, 6 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 12 found, 226 missed, 199 extra; recall 5.0%.
    • Gemini · score 66.8 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 46 supported agreements, 5 useful resolutions, 10 reasonable endings without resolution, 1 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 9 found, 229 missed, 107 extra; recall 3.8%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    93.5%
    Model errors
    11 of 73 responses
    Repeat calls
    11
    Assessments accepted
    62 of 62 observed
    Error tokens
    167,388
    Error cost
    $2.45
    Recorded tokens
    1,018,329
    Recorded cost
    $14.26
    Request time
    Not recorded
    Gemini
    Valid first response
    90.3%
    Model errors
    10 of 72 responses
    Repeat calls
    10
    Assessments accepted
    62 of 62 observed
    Error tokens
    102,581
    Error cost
    $0.23
    Recorded tokens
    800,238
    Recorded cost
    $1.86
    Request time
    Not recorded
  10. 10 Claude Sonnet 5 · Gemma / Novita BF16 · Sol / Azure Benchmark version 3.0 xhigh · simple prompt Score 62.1 Judges 2 Ended appropriately 47/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 60.5
    • Gemini · 62/62Score 63.7
    Ended appropriately: 47 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 5.0 rounds on average. 0 of 62 reached the 25-round limit; 0 ended at round 24 or later.

    What Sol found at the end

    • 33 supported agreements
    • 5 useful resolutions
    • 9 reasonable endings without resolution
    • 15 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 48 of 62 conversations. Sol agreed in 33 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 202 of 49858 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 5.0 rounds. Round 3: 16 · Round 4: 17 · Round 5: 13 · Round 6: 6 · Round 7: 1 · Round 8: 4 · Round 9: 3 · Round 11: 1 · Round 13: 1

    Moderator ending claims compared with Sol
    • Claimed agreement: 33 supported agreements, 4 useful resolutions, 0 reasonable endings without resolution, 11 too soon, 0 unclear.
    • Claimed useful resolution: 0 supported agreements, 1 useful resolutions, 0 reasonable endings without resolution, 2 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 0 useful resolutions, 9 reasonable endings without resolution, 2 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 60.5 Evaluator recipe: 269d705430e76bd2f9aea7863510a110f8debb70a5bde654016dbe762b553772 Endings: 33 supported agreements, 5 useful resolutions, 9 reasonable endings without resolution, 15 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 10 found, 228 missed, 183 extra; recall 4.2%.
    • Gemini · score 63.7 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 43 supported agreements, 6 useful resolutions, 12 reasonable endings without resolution, 1 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 10 found, 228 missed, 107 extra; recall 4.2%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    90.3%
    Model errors
    13 of 74 responses
    Repeat calls
    12
    Assessments accepted
    61 of 62 observed
    Error tokens
    192,649
    Error cost
    $2.78
    Recorded tokens
    1,052,879
    Recorded cost
    $14.58
    Request time
    Not recorded
    Gemini
    Valid first response
    83.9%
    Model errors
    14 of 76 responses
    Repeat calls
    14
    Assessments accepted
    62 of 62 observed
    Error tokens
    127,798
    Error cost
    $0.28
    Recorded tokens
    881,996
    Recorded cost
    $2.06
    Request time
    Not recorded
  11. 11 Qwen3.8 Max · Gemma / Novita BF16 · Sol / Azure Benchmark version 3.0 0902, OpenRouter / Alibaba · simple prompt Score 61.6 Judges 2 Ended appropriately 49/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 59.9
    • Gemini · 62/62Score 63.3
    Ended appropriately: 49 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 4.7 rounds on average. 0 of 62 reached the 25-round limit; 0 ended at round 24 or later.

    What Sol found at the end

    • 41 supported agreements
    • 1 useful resolutions
    • 7 reasonable endings without resolution
    • 13 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 51 of 62 conversations. Sol agreed in 39 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 133 of 31146 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 4.7 rounds. Round 3: 12 · Round 4: 24 · Round 5: 12 · Round 6: 5 · Round 7: 3 · Round 8: 4 · Round 9: 1 · Round 10: 1

    Moderator ending claims compared with Sol
    • Claimed agreement: 39 supported agreements, 1 useful resolutions, 0 reasonable endings without resolution, 11 too soon, 0 unclear.
    • Claimed useful resolution: 1 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    • Claimed no resolution: 1 supported agreements, 0 useful resolutions, 7 reasonable endings without resolution, 2 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 59.9 Evaluator recipe: 269d705430e76bd2f9aea7863510a110f8debb70a5bde654016dbe762b553772 Endings: 41 supported agreements, 1 useful resolutions, 7 reasonable endings without resolution, 13 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 10 found, 228 missed, 182 extra; recall 4.2%.
    • Gemini · score 63.3 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 51 supported agreements, 1 useful resolutions, 9 reasonable endings without resolution, 1 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 12 found, 226 missed, 103 extra; recall 5.0%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    95.2%
    Model errors
    3 of 65 responses
    Repeat calls
    3
    Assessments accepted
    62 of 62 observed
    Error tokens
    43,153
    Error cost
    $0.60
    Recorded tokens
    873,084
    Recorded cost
    $12.24
    Request time
    Not recorded
    Gemini
    Valid first response
    90.3%
    Model errors
    11 of 73 responses
    Repeat calls
    11
    Assessments accepted
    62 of 62 observed
    Error tokens
    126,238
    Error cost
    $0.30
    Recorded tokens
    829,608
    Recorded cost
    $2.00
    Request time
    Not recorded
  12. 12 Claude Haiku 5.5 · Gemma / Novita BF16 · Sol / Azure Benchmark version 3.0 high · simple prompt Score 60.1 Judges 1 Ended appropriately 51/62

    Provisional · 1 run · scored by Sol

    • Sol · 62/62Score 60.1
    Ended appropriately: 51 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 6.4 rounds on average. 0 of 62 reached the 25-round limit; 0 ended at round 24 or later.

    What Sol found at the end

    • 40 supported agreements
    • 3 useful resolutions
    • 8 reasonable endings without resolution
    • 11 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 46 of 62 conversations. Sol agreed in 39 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 480 of 51139 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 6.4 rounds. Round 3: 8 · Round 4: 9 · Round 5: 14 · Round 6: 5 · Round 7: 7 · Round 8: 7 · Round 9: 4 · Round 10: 2 · Round 11: 2 · Round 12: 2 · Round 16: 2

    Moderator ending claims compared with Sol
    • Claimed agreement: 39 supported agreements, 1 useful resolutions, 0 reasonable endings without resolution, 6 too soon, 0 unclear.
    • Claimed useful resolution: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    • Claimed no resolution: 1 supported agreements, 2 useful resolutions, 8 reasonable endings without resolution, 5 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 60.1 Evaluator recipe: 269d705430e76bd2f9aea7863510a110f8debb70a5bde654016dbe762b553772 Endings: 40 supported agreements, 3 useful resolutions, 8 reasonable endings without resolution, 11 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 12 found, 226 missed, 187 extra; recall 5.0%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    90.3%
    Model errors
    7 of 69 responses
    Repeat calls
    7
    Assessments accepted
    62 of 62 observed
    Error tokens
    98,614
    Error cost
    $1.32
    Recorded tokens
    1,002,833
    Recorded cost
    $13.61
    Request time
    Not recorded
    Gemini
    Valid first response
    91.9%
    Model errors
    8 of 67 responses
    Repeat calls
    5
    Assessments accepted
    59 of 62 observed
    Error tokens
    96,362
    Error cost
    $0.22
    Recorded tokens
    841,471
    Recorded cost
    $1.95
    Request time
    Not recorded
  13. 13 Qwen3.8 Flash Benchmark version 3.0 OpenRouter · simple prompt Score 60.0 Judges 2 Ended appropriately 45/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 58.2
    • Gemini · 62/62Score 61.7
    Ended appropriately: 45 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 4.8 rounds on average. 0 of 62 reached the 25-round limit; 0 ended at round 24 or later.

    What Sol found at the end

    • 40 supported agreements
    • 1 useful resolutions
    • 4 reasonable endings without resolution
    • 17 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 42 of 62 conversations. Sol agreed in 37 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 130 of 32624 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 4.8 rounds. Round 3: 17 · Round 4: 13 · Round 5: 16 · Round 6: 7 · Round 7: 3 · Round 8: 4 · Round 9: 2

    Moderator ending claims compared with Sol
    • Claimed agreement: 37 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 5 too soon, 0 unclear.
    • Claimed useful resolution: 3 supported agreements, 0 useful resolutions, 1 reasonable endings without resolution, 0 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 1 useful resolutions, 3 reasonable endings without resolution, 12 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 58.2 Evaluator recipe: 320b75fc6ed0209f9b39e2cbb8276e552b4b6bc182ca181f0ad2763dfdbe6ddb Endings: 40 supported agreements, 1 useful resolutions, 4 reasonable endings without resolution, 17 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 12 found, 226 missed, 192 extra; recall 5.0%.
    • Gemini · score 61.7 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 43 supported agreements, 3 useful resolutions, 11 reasonable endings without resolution, 5 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 11 found, 227 missed, 104 extra; recall 4.6%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    95.2%
    Model errors
    4 of 66 responses
    Repeat calls
    4
    Assessments accepted
    62 of 62 observed
    Error tokens
    51,068
    Error cost
    $0.73
    Recorded tokens
    881,132
    Recorded cost
    $12.32
    Request time
    Not recorded
    Gemini
    Valid first response
    88.7%
    Model errors
    9 of 71 responses
    Repeat calls
    9
    Assessments accepted
    62 of 62 observed
    Error tokens
    99,903
    Error cost
    $0.23
    Recorded tokens
    749,648
    Recorded cost
    $1.72
    Request time
    Not recorded
  14. 14 GPT-5.6 Luna · Gemma / Novita BF16 · Sol / Azure Benchmark version 3.0 Azure · high effort · simple prompt Score 59.4 Judges 2 Ended appropriately 48/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 59.4 Same provider lab as mediator
    • Gemini · 62/62Score 59.3
    Ended appropriately: 48 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 4.6 rounds on average. 0 of 62 reached the 25-round limit; 0 ended at round 24 or later.

    What Sol found at the end

    • 39 supported agreements
    • 2 useful resolutions
    • 7 reasonable endings without resolution
    • 14 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 43 of 62 conversations. Sol agreed in 39 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 157 of 30182 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 4.6 rounds. Round 3: 24 · Round 4: 18 · Round 5: 8 · Round 6: 4 · Round 7: 3 · Round 8: 1 · Round 10: 1 · Round 12: 2 · Round 15: 1

    Moderator ending claims compared with Sol
    • Claimed agreement: 39 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 4 too soon, 0 unclear.
    • Claimed useful resolution: 0 supported agreements, 1 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 1 useful resolutions, 7 reasonable endings without resolution, 10 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 59.4 · same provider lab as mediator Evaluator recipe: 269d705430e76bd2f9aea7863510a110f8debb70a5bde654016dbe762b553772 Endings: 39 supported agreements, 2 useful resolutions, 7 reasonable endings without resolution, 14 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 12 found, 226 missed, 172 extra; recall 5.0%.
    • Gemini · score 59.3 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 41 supported agreements, 3 useful resolutions, 11 reasonable endings without resolution, 7 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 9 found, 229 missed, 103 extra; recall 3.8%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    98.4%
    Model errors
    1 of 63 responses
    Repeat calls
    1
    Assessments accepted
    62 of 62 observed
    Error tokens
    16,749
    Error cost
    $0.24
    Recorded tokens
    840,149
    Recorded cost
    $11.82
    Request time
    Not recorded
    Gemini
    Valid first response
    90.3%
    Model errors
    9 of 71 responses
    Repeat calls
    9
    Assessments accepted
    62 of 62 observed
    Error tokens
    75,633
    Error cost
    $0.17
    Recorded tokens
    808,887
    Recorded cost
    $2.00
    Request time
    Not recorded
  15. 15 MiniMax M3 · Gemma / Novita BF16 · Sol / Azure Benchmark version 3.0 Together · simple prompt Score 56.2 Judges 2 Ended appropriately 25/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 54.2
    • Gemini · 62/62Score 58.2
    Ended appropriately: 25 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 5.5 rounds on average. 0 of 62 reached the 25-round limit; 0 ended at round 24 or later.

    What Sol found at the end

    • 16 supported agreements
    • 4 useful resolutions
    • 5 reasonable endings without resolution
    • 37 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 40 of 62 conversations. Sol agreed in 15 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 275 of 43639 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 5.5 rounds. Round 3: 10 · Round 4: 12 · Round 5: 17 · Round 6: 8 · Round 7: 5 · Round 8: 5 · Round 9: 2 · Round 10: 1 · Round 12: 1 · Round 18: 1

    Moderator ending claims compared with Sol
    • Claimed agreement: 15 supported agreements, 3 useful resolutions, 0 reasonable endings without resolution, 22 too soon, 0 unclear.
    • Claimed useful resolution: 1 supported agreements, 1 useful resolutions, 0 reasonable endings without resolution, 3 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 0 useful resolutions, 5 reasonable endings without resolution, 12 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 54.2 Evaluator recipe: 269d705430e76bd2f9aea7863510a110f8debb70a5bde654016dbe762b553772 Endings: 16 supported agreements, 4 useful resolutions, 5 reasonable endings without resolution, 37 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 11 found, 227 missed, 179 extra; recall 4.6%.
    • Gemini · score 58.2 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 27 supported agreements, 11 useful resolutions, 9 reasonable endings without resolution, 15 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 11 found, 227 missed, 114 extra; recall 4.6%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    98.4%
    Model errors
    2 of 64 responses
    Repeat calls
    2
    Assessments accepted
    62 of 62 observed
    Error tokens
    21,654
    Error cost
    $0.28
    Recorded tokens
    878,221
    Recorded cost
    $12.05
    Request time
    Not recorded
    Gemini
    Valid first response
    91.9%
    Model errors
    6 of 68 responses
    Repeat calls
    6
    Assessments accepted
    62 of 62 observed
    Error tokens
    75,692
    Error cost
    $0.18
    Recorded tokens
    851,050
    Recorded cost
    $2.07
    Request time
    Not recorded
  16. 16 Gemini 4 Argon · Gemma / Novita BF16 · Sol / Azure Benchmark version 3.0 OpenRouter / Google Vertex, high effort; admission candidate · simple prompt Score 56.0 Judges 2 Ended appropriately 30/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 53.9
    • Gemini · 62/62Score 58.1 Same provider lab as mediator
    Ended appropriately: 30 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 5.2 rounds on average. 0 of 62 reached the 25-round limit; 0 ended at round 24 or later.

    What Sol found at the end

    • 26 supported agreements
    • 0 useful resolutions
    • 4 reasonable endings without resolution
    • 32 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 37 of 62 conversations. Sol agreed in 26 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 157 of 27447 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 5.2 rounds. Round 3: 10 · Round 4: 15 · Round 5: 12 · Round 6: 12 · Round 7: 7 · Round 8: 5 · Round 9: 1

    Moderator ending claims compared with Sol
    • Claimed agreement: 26 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 11 too soon, 0 unclear.
    • Claimed useful resolution: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 1 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 0 useful resolutions, 4 reasonable endings without resolution, 20 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 53.9 Evaluator recipe: 269d705430e76bd2f9aea7863510a110f8debb70a5bde654016dbe762b553772 Endings: 26 supported agreements, 0 useful resolutions, 4 reasonable endings without resolution, 32 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 10 found, 228 missed, 179 extra; recall 4.2%.
    • Gemini · score 58.1 · same provider lab as mediator Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 37 supported agreements, 1 useful resolutions, 11 reasonable endings without resolution, 13 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 13 found, 225 missed, 101 extra; recall 5.5%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    98.4%
    Model errors
    1 of 63 responses
    Repeat calls
    1
    Assessments accepted
    62 of 62 observed
    Error tokens
    13,217
    Error cost
    $0.18
    Recorded tokens
    791,686
    Recorded cost
    $10.94
    Request time
    Not recorded
    Gemini
    Valid first response
    85.5%
    Model errors
    11 of 73 responses
    Repeat calls
    11
    Assessments accepted
    62 of 62 observed
    Error tokens
    116,873
    Error cost
    $0.27
    Recorded tokens
    718,855
    Recorded cost
    $1.62
    Request time
    Not recorded
  17. 17 GPT-5.4 Mini Benchmark version 3.0 simple prompt Score 55.9 Judges 2 Ended appropriately 35/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 55.0 Same provider lab as mediator
    • Gemini · 62/62Score 56.7
    Ended appropriately: 35 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 4.7 rounds on average. 0 of 62 reached the 25-round limit; 1 ended at round 24 or later.

    What Sol found at the end

    • 23 supported agreements
    • 2 useful resolutions
    • 10 reasonable endings without resolution
    • 27 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 36 of 62 conversations. Sol agreed in 20 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 265 of 34052 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 4.7 rounds. Round 3: 24 · Round 4: 18 · Round 5: 9 · Round 6: 6 · Round 9: 1 · Round 10: 1 · Round 11: 1 · Round 14: 1 · Round 24: 1

    Moderator ending claims compared with Sol
    • Claimed agreement: 20 supported agreements, 0 useful resolutions, 1 reasonable endings without resolution, 15 too soon, 0 unclear.
    • Claimed useful resolution: 3 supported agreements, 1 useful resolutions, 0 reasonable endings without resolution, 1 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 1 useful resolutions, 9 reasonable endings without resolution, 11 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 55.0 · same provider lab as mediator Evaluator recipe: 320b75fc6ed0209f9b39e2cbb8276e552b4b6bc182ca181f0ad2763dfdbe6ddb Endings: 23 supported agreements, 2 useful resolutions, 10 reasonable endings without resolution, 27 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 4 found, 234 missed, 181 extra; recall 1.7%.
    • Gemini · score 56.7 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 33 supported agreements, 3 useful resolutions, 11 reasonable endings without resolution, 15 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 11 found, 227 missed, 102 extra; recall 4.6%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    93.5%
    Model errors
    4 of 66 responses
    Repeat calls
    4
    Assessments accepted
    62 of 62 observed
    Error tokens
    59,470
    Error cost
    $0.85
    Recorded tokens
    877,735
    Recorded cost
    $12.20
    Request time
    Not recorded
    Gemini
    Valid first response
    93.5%
    Model errors
    5 of 67 responses
    Repeat calls
    5
    Assessments accepted
    62 of 62 observed
    Error tokens
    43,597
    Error cost
    $0.11
    Recorded tokens
    777,170
    Recorded cost
    $1.92
    Request time
    Not recorded
  18. 18 DeepSeek V4.1 Flash · Gemma / Novita BF16 · Sol / Azure Benchmark version 3.0 DeepInfra, FP8 · simple prompt Score 54.8 Judges 2 Ended appropriately 35/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 54.3
    • Gemini · 62/62Score 55.4
    Ended appropriately: 35 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 4.9 rounds on average. 0 of 62 reached the 25-round limit; 0 ended at round 24 or later.

    What Sol found at the end

    • 17 supported agreements
    • 3 useful resolutions
    • 15 reasonable endings without resolution
    • 27 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 30 of 62 conversations. Sol agreed in 14 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 505 of 45818 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 4.9 rounds. Round 3: 16 · Round 4: 15 · Round 5: 12 · Round 6: 8 · Round 7: 6 · Round 8: 2 · Round 9: 2 · Round 13: 1

    Moderator ending claims compared with Sol
    • Claimed agreement: 14 supported agreements, 1 useful resolutions, 0 reasonable endings without resolution, 15 too soon, 0 unclear.
    • Claimed useful resolution: 2 supported agreements, 1 useful resolutions, 0 reasonable endings without resolution, 1 too soon, 0 unclear.
    • Claimed no resolution: 1 supported agreements, 1 useful resolutions, 15 reasonable endings without resolution, 11 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 54.3 Evaluator recipe: 269d705430e76bd2f9aea7863510a110f8debb70a5bde654016dbe762b553772 Endings: 17 supported agreements, 3 useful resolutions, 15 reasonable endings without resolution, 27 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 15 found, 223 missed, 184 extra; recall 6.3%.
    • Gemini · score 55.4 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 25 supported agreements, 2 useful resolutions, 28 reasonable endings without resolution, 7 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 9 found, 229 missed, 108 extra; recall 3.8%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    95.2%
    Model errors
    3 of 65 responses
    Repeat calls
    3
    Assessments accepted
    62 of 62 observed
    Error tokens
    48,976
    Error cost
    $0.68
    Recorded tokens
    874,366
    Recorded cost
    $11.85
    Request time
    Not recorded
    Gemini
    Valid first response
    95.2%
    Model errors
    3 of 65 responses
    Repeat calls
    3
    Assessments accepted
    62 of 62 observed
    Error tokens
    41,134
    Error cost
    $0.09
    Recorded tokens
    793,069
    Recorded cost
    $1.90
    Request time
    Not recorded
  19. 19 Grok 4.6 Benchmark version 3.0 OpenRouter · simple prompt Score 54.6 Judges 2 Ended appropriately 40/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 56.1
    • Gemini · 62/62Score 53.2
    Ended appropriately: 40 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 4.4 rounds on average. 0 of 62 reached the 25-round limit; 0 ended at round 24 or later.

    What Sol found at the end

    • 32 supported agreements
    • 0 useful resolutions
    • 8 reasonable endings without resolution
    • 22 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 35 of 62 conversations. Sol agreed in 31 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 223 of 18725 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 4.4 rounds. Round 3: 22 · Round 4: 15 · Round 5: 12 · Round 6: 7 · Round 7: 2 · Round 8: 4

    Moderator ending claims compared with Sol
    • Claimed agreement: 31 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 4 too soon, 0 unclear.
    • Claimed useful resolution: 1 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 0 useful resolutions, 8 reasonable endings without resolution, 18 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 56.1 Evaluator recipe: 320b75fc6ed0209f9b39e2cbb8276e552b4b6bc182ca181f0ad2763dfdbe6ddb Endings: 32 supported agreements, 0 useful resolutions, 8 reasonable endings without resolution, 22 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 8 found, 230 missed, 195 extra; recall 3.4%.
    • Gemini · score 53.2 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 36 supported agreements, 0 useful resolutions, 12 reasonable endings without resolution, 14 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 8 found, 230 missed, 106 extra; recall 3.4%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    96.8%
    Model errors
    2 of 64 responses
    Repeat calls
    2
    Assessments accepted
    62 of 62 observed
    Error tokens
    29,755
    Error cost
    $0.45
    Recorded tokens
    801,202
    Recorded cost
    $11.29
    Request time
    Not recorded
    Gemini
    Valid first response
    90.3%
    Model errors
    6 of 68 responses
    Repeat calls
    6
    Assessments accepted
    62 of 62 observed
    Error tokens
    46,502
    Error cost
    $0.11
    Recorded tokens
    650,948
    Recorded cost
    $1.51
    Request time
    Not recorded
  20. 20 Gemini 3.1 Pro · Gemma / Novita BF16 · Sol / Azure Benchmark version 3.0 OpenRouter · simple prompt Score 52.3 Judges 3 Ended appropriately 41/62

    Scored by 3 judges · 1 run · Sol, Gemini, Astra (Azure)

    • Sol · 62/62Score 53.0
    • Gemini · 62/62Score 55.1 Same provider lab as mediator
    • Astra (Azure) · 62/62Score 48.9
    Ended appropriately: 41 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 6.9 rounds on average. 0 of 62 reached the 25-round limit; 0 ended at round 24 or later.

    What Sol found at the end

    • 24 supported agreements
    • 3 useful resolutions
    • 14 reasonable endings without resolution
    • 21 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 36 of 62 conversations. Sol agreed in 23 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 335 of 34262 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 6.9 rounds. Round 3: 8 · Round 4: 6 · Round 5: 9 · Round 6: 4 · Round 7: 12 · Round 8: 10 · Round 9: 6 · Round 10: 3 · Round 11: 1 · Round 12: 1 · Round 14: 1 · Round 23: 1

    Moderator ending claims compared with Sol
    • Claimed agreement: 23 supported agreements, 3 useful resolutions, 0 reasonable endings without resolution, 10 too soon, 0 unclear.
    • Claimed useful resolution: 1 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 1 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 0 useful resolutions, 14 reasonable endings without resolution, 10 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 53.0 Evaluator recipe: 269d705430e76bd2f9aea7863510a110f8debb70a5bde654016dbe762b553772 Endings: 24 supported agreements, 3 useful resolutions, 14 reasonable endings without resolution, 21 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 13 found, 225 missed, 196 extra; recall 5.5%.
    • Gemini · score 55.1 · same provider lab as mediator Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 25 supported agreements, 7 useful resolutions, 21 reasonable endings without resolution, 9 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 18 found, 220 missed, 99 extra; recall 7.6%.
    • Astra (Azure) · score 48.9 Evaluator recipe: c9141e9fd50645c9ae95f59bd2177b746e23f6e389787cd7c5c8e8b477dce89e Endings: 23 supported agreements, 0 useful resolutions, 23 reasonable endings without resolution, 16 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 14 found, 224 missed, 166 extra; recall 5.9%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    87.1%
    Model errors
    9 of 71 responses
    Repeat calls
    9
    Assessments accepted
    62 of 62 observed
    Error tokens
    120,699
    Error cost
    $1.60
    Recorded tokens
    971,543
    Recorded cost
    $13.09
    Request time
    Not recorded
    Astra (Azure)
    Valid first response
    96.8%
    Model errors
    5 of 67 responses
    Repeat calls
    5
    Assessments accepted
    62 of 62 observed
    Error tokens
    64,117
    Error cost
    $2.25
    Recorded tokens
    837,320
    Recorded cost
    $27.22
    Request time
    Not recorded
    Gemini
    Valid first response
    100.0%
    Model errors
    0 of 62 responses
    Repeat calls
    0
    Assessments accepted
    62 of 62 observed
    Error tokens
    0
    Error cost
    $0.00
    Recorded tokens
    750,625
    Recorded cost
    $1.76
    Request time
    Not recorded
  21. 21 GPT-6 Astra · Gemma / Novita BF16 · Sol / Azure Benchmark version 3.0 Azure · high effort · simple prompt Score 51.3 Judges 2 Ended appropriately 34/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 52.5 Same provider lab as mediator
    • Gemini · 62/62Score 50.1
    Ended appropriately: 34 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 7.0 rounds on average. 0 of 62 reached the 25-round limit; 0 ended at round 24 or later.

    What Sol found at the end

    • 23 supported agreements
    • 2 useful resolutions
    • 9 reasonable endings without resolution
    • 28 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 23 of 62 conversations. Sol agreed in 21 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 66 of 27145 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 7.0 rounds. Round 3: 7 · Round 4: 13 · Round 5: 10 · Round 6: 2 · Round 7: 8 · Round 8: 7 · Round 10: 4 · Round 11: 3 · Round 12: 1 · Round 13: 3 · Round 14: 2 · Round 15: 1 · Round 18: 1

    Moderator ending claims compared with Sol
    • Claimed agreement: 21 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 2 too soon, 0 unclear.
    • Claimed useful resolution: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    • Claimed no resolution: 2 supported agreements, 2 useful resolutions, 9 reasonable endings without resolution, 26 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 52.5 · same provider lab as mediator Evaluator recipe: 269d705430e76bd2f9aea7863510a110f8debb70a5bde654016dbe762b553772 Endings: 23 supported agreements, 2 useful resolutions, 9 reasonable endings without resolution, 28 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 13 found, 225 missed, 186 extra; recall 5.5%.
    • Gemini · score 50.1 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 23 supported agreements, 3 useful resolutions, 18 reasonable endings without resolution, 18 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 7 found, 231 missed, 105 extra; recall 2.9%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    91.9%
    Model errors
    8 of 70 responses
    Repeat calls
    8
    Assessments accepted
    62 of 62 observed
    Error tokens
    108,509
    Error cost
    $1.53
    Recorded tokens
    935,393
    Recorded cost
    $12.79
    Request time
    Not recorded
    Gemini
    Valid first response
    88.7%
    Model errors
    8 of 70 responses
    Repeat calls
    8
    Assessments accepted
    62 of 62 observed
    Error tokens
    63,681
    Error cost
    $0.13
    Recorded tokens
    717,653
    Recorded cost
    $1.58
    Request time
    Not recorded
  22. 22 Inkling · Gemma / Novita BF16 · Sol / Azure Benchmark version 3.0 Thinking Machines / Together, FP4 · simple prompt Score 50.8 Judges 2 Ended appropriately 30/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 51.0
    • Gemini · 62/62Score 50.6
    Ended appropriately: 30 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 4.2 rounds on average. 0 of 62 reached the 25-round limit; 0 ended at round 24 or later.

    What Sol found at the end

    • 20 supported agreements
    • 1 useful resolutions
    • 9 reasonable endings without resolution
    • 32 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 30 of 62 conversations. Sol agreed in 19 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 181 of 20328 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 4.2 rounds. Round 3: 27 · Round 4: 16 · Round 5: 9 · Round 6: 5 · Round 7: 3 · Round 9: 1 · Round 10: 1

    Moderator ending claims compared with Sol
    • Claimed agreement: 19 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 11 too soon, 0 unclear.
    • Claimed useful resolution: 1 supported agreements, 1 useful resolutions, 0 reasonable endings without resolution, 1 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 0 useful resolutions, 9 reasonable endings without resolution, 20 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 51.0 Evaluator recipe: 269d705430e76bd2f9aea7863510a110f8debb70a5bde654016dbe762b553772 Endings: 20 supported agreements, 1 useful resolutions, 9 reasonable endings without resolution, 32 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 13 found, 225 missed, 169 extra; recall 5.5%.
    • Gemini · score 50.6 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 25 supported agreements, 1 useful resolutions, 17 reasonable endings without resolution, 19 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 8 found, 230 missed, 107 extra; recall 3.4%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    95.2%
    Model errors
    5 of 67 responses
    Repeat calls
    5
    Assessments accepted
    62 of 62 observed
    Error tokens
    57,617
    Error cost
    $0.77
    Recorded tokens
    798,453
    Recorded cost
    $11.01
    Request time
    Not recorded
    Gemini
    Valid first response
    95.2%
    Model errors
    3 of 65 responses
    Repeat calls
    3
    Assessments accepted
    62 of 62 observed
    Error tokens
    30,022
    Error cost
    $0.07
    Recorded tokens
    703,009
    Recorded cost
    $1.74
    Request time
    Not recorded
  23. 23 gpt-oss-120b · Gemma / Novita BF16 · Sol / Azure Benchmark version 3.0 Cerebras · simple prompt Score 47.3 Judges 2 Ended appropriately 24/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 49.5 Same provider lab as mediator
    • Gemini · 62/62Score 45.0
    Ended appropriately: 24 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 6.9 rounds on average. 2 of 62 reached the 25-round limit; 2 ended at round 24 or later.

    What Sol found at the end

    • 18 supported agreements
    • 2 useful resolutions
    • 4 reasonable endings without resolution
    • 38 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 18 of 62 conversations. Sol agreed in 11 of those cases.

    The moderator chose when to stop in 60 conversations; 2 reached the limit instead.

    Wording overlap check: 248 of 45883 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 6.9 rounds. Round 3: 13 · Round 4: 11 · Round 5: 11 · Round 6: 5 · Round 7: 4 · Round 8: 5 · Round 9: 1 · Round 10: 1 · Round 11: 3 · Round 13: 2 · Round 14: 1 · Round 15: 1 · Round 16: 1 · Round 18: 1 · Round 25: 2

    Moderator ending claims compared with Sol
    • Claimed agreement: 11 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 7 too soon, 0 unclear.
    • Claimed useful resolution: 6 supported agreements, 2 useful resolutions, 0 reasonable endings without resolution, 5 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 0 useful resolutions, 4 reasonable endings without resolution, 25 too soon, 0 unclear.
    • No ending claim: 1 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 1 too soon, 0 unclear.
    Judge evidence
    • Sol · score 49.5 · same provider lab as mediator Evaluator recipe: 269d705430e76bd2f9aea7863510a110f8debb70a5bde654016dbe762b553772 Endings: 18 supported agreements, 2 useful resolutions, 4 reasonable endings without resolution, 38 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 16 found, 222 missed, 194 extra; recall 6.7%.
    • Gemini · score 45.0 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 16 supported agreements, 3 useful resolutions, 13 reasonable endings without resolution, 30 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 18 found, 220 missed, 101 extra; recall 7.6%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    98.4%
    Model errors
    1 of 63 responses
    Repeat calls
    1
    Assessments accepted
    62 of 62 observed
    Error tokens
    14,365
    Error cost
    $0.21
    Recorded tokens
    896,558
    Recorded cost
    $11.99
    Request time
    Not recorded
    Gemini
    Valid first response
    93.5%
    Model errors
    7 of 69 responses
    Repeat calls
    7
    Assessments accepted
    62 of 62 observed
    Error tokens
    71,336
    Error cost
    $0.15
    Recorded tokens
    883,130
    Recorded cost
    $2.06
    Request time
    Not recorded
  24. 24 Nemotron 3 Ultra Benchmark version 3.0 DeepInfra · simple prompt Score 46.8 Judges 2 Ended appropriately 21/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 47.3
    • Gemini · 62/62Score 46.3
    Ended appropriately: 21 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 6.1 rounds on average. 1 of 62 reached the 25-round limit; 2 ended at round 24 or later.

    What Sol found at the end

    • 15 supported agreements
    • 1 useful resolutions
    • 5 reasonable endings without resolution
    • 41 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 24 of 62 conversations. Sol agreed in 11 of those cases.

    The moderator chose when to stop in 61 conversations; 1 reached the limit instead.

    Wording overlap check: 180 of 31239 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 6.1 rounds. Round 3: 17 · Round 4: 13 · Round 5: 6 · Round 6: 9 · Round 7: 4 · Round 9: 4 · Round 10: 4 · Round 11: 1 · Round 12: 1 · Round 16: 1 · Round 24: 1 · Round 25: 1

    Moderator ending claims compared with Sol
    • Claimed agreement: 11 supported agreements, 1 useful resolutions, 0 reasonable endings without resolution, 12 too soon, 0 unclear.
    • Claimed useful resolution: 3 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 1 too soon, 0 unclear.
    • Claimed no resolution: 1 supported agreements, 0 useful resolutions, 5 reasonable endings without resolution, 27 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 1 too soon, 0 unclear.
    Judge evidence
    • Sol · score 47.3 Evaluator recipe: 320b75fc6ed0209f9b39e2cbb8276e552b4b6bc182ca181f0ad2763dfdbe6ddb Endings: 15 supported agreements, 1 useful resolutions, 5 reasonable endings without resolution, 41 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 13 found, 225 missed, 177 extra; recall 5.5%.
    • Gemini · score 46.3 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 18 supported agreements, 5 useful resolutions, 14 reasonable endings without resolution, 25 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 12 found, 226 missed, 111 extra; recall 5.0%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    98.4%
    Model errors
    1 of 63 responses
    Repeat calls
    1
    Assessments accepted
    62 of 62 observed
    Error tokens
    14,324
    Error cost
    $0.21
    Recorded tokens
    830,451
    Recorded cost
    $11.17
    Request time
    Not recorded
    Gemini
    Valid first response
    93.5%
    Model errors
    4 of 66 responses
    Repeat calls
    4
    Assessments accepted
    62 of 62 observed
    Error tokens
    60,863
    Error cost
    $0.15
    Recorded tokens
    792,147
    Recorded cost
    $1.87
    Request time
    Not recorded
  25. 25 DeepSeek V4 Pro Benchmark version 3.0 DeepInfra · simple prompt Score 46.5 Judges 2 Ended appropriately 9/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 49.1
    • Gemini · 62/62Score 44.0
    Ended appropriately: 9 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 3.7 rounds on average. 0 of 62 reached the 25-round limit; 0 ended at round 24 or later.

    What Sol found at the end

    • 6 supported agreements
    • 0 useful resolutions
    • 3 reasonable endings without resolution
    • 53 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 25 of 62 conversations. Sol agreed in 5 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 259 of 24204 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 3.7 rounds. Round 3: 33 · Round 4: 18 · Round 5: 8 · Round 6: 2 · Round 7: 1

    Moderator ending claims compared with Sol
    • Claimed agreement: 5 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 20 too soon, 0 unclear.
    • Claimed useful resolution: 1 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 4 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 0 useful resolutions, 3 reasonable endings without resolution, 29 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 49.1 Evaluator recipe: 320b75fc6ed0209f9b39e2cbb8276e552b4b6bc182ca181f0ad2763dfdbe6ddb Endings: 6 supported agreements, 0 useful resolutions, 3 reasonable endings without resolution, 53 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 8 found, 230 missed, 175 extra; recall 3.4%.
    • Gemini · score 44.0 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 9 supported agreements, 2 useful resolutions, 5 reasonable endings without resolution, 46 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 8 found, 230 missed, 108 extra; recall 3.4%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    96.8%
    Model errors
    4 of 66 responses
    Repeat calls
    4
    Assessments accepted
    62 of 62 observed
    Error tokens
    59,838
    Error cost
    $0.88
    Recorded tokens
    762,894
    Recorded cost
    $10.41
    Request time
    Not recorded
    Gemini
    Valid first response
    98.4%
    Model errors
    1 of 63 responses
    Repeat calls
    1
    Assessments accepted
    62 of 62 observed
    Error tokens
    9,947
    Error cost
    $0.02
    Recorded tokens
    617,845
    Recorded cost
    $1.46
    Request time
    Not recorded
  26. 26 DeepSeek V4 Pro · Gemma / Novita BF16 · Sol / Azure Benchmark version 3.0 DeepInfra · simple prompt Score 43.9 Judges 2 Ended appropriately 9/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 45.6
    • Gemini · 62/62Score 42.3
    Ended appropriately: 9 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 3.9 rounds on average. 0 of 62 reached the 25-round limit; 0 ended at round 24 or later.

    What Sol found at the end

    • 6 supported agreements
    • 0 useful resolutions
    • 3 reasonable endings without resolution
    • 53 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 17 of 62 conversations. Sol agreed in 5 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 331 of 23900 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 3.9 rounds. Round 3: 33 · Round 4: 15 · Round 5: 9 · Round 6: 1 · Round 7: 1 · Round 8: 2 · Round 9: 1

    Moderator ending claims compared with Sol
    • Claimed agreement: 5 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 12 too soon, 0 unclear.
    • Claimed useful resolution: 1 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 4 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 0 useful resolutions, 3 reasonable endings without resolution, 37 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 45.6 Evaluator recipe: 269d705430e76bd2f9aea7863510a110f8debb70a5bde654016dbe762b553772 Endings: 6 supported agreements, 0 useful resolutions, 3 reasonable endings without resolution, 53 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 9 found, 229 missed, 176 extra; recall 3.8%.
    • Gemini · score 42.3 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 10 supported agreements, 3 useful resolutions, 13 reasonable endings without resolution, 36 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 9 found, 229 missed, 105 extra; recall 3.8%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    98.4%
    Model errors
    1 of 63 responses
    Repeat calls
    1
    Assessments accepted
    62 of 62 observed
    Error tokens
    10,044
    Error cost
    $0.13
    Recorded tokens
    733,880
    Recorded cost
    $10.03
    Request time
    Not recorded
    Gemini
    Valid first response
    95.2%
    Model errors
    3 of 65 responses
    Repeat calls
    3
    Assessments accepted
    62 of 62 observed
    Error tokens
    37,315
    Error cost
    $0.10
    Recorded tokens
    629,616
    Recorded cost
    $1.46
    Request time
    Not recorded
  27. 27 Gemma 4 31B Benchmark version 3.0 Cerebras · simple prompt Score 42.4 Judges 4 Ended appropriately 28/62

    Scored by 4 judges · 1 run · Sol, Fable, Gemini, Astra (Azure)

    • Sol · 62/62Score 43.9
    • Fable · 62/62Score 43.8
    • Gemini · 62/62Score 40.2 Same provider lab as mediator
    • Astra (Azure) · 62/62Score 41.6
    Ended appropriately: 28 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 17.4 rounds on average. 19 of 62 reached the 25-round limit; 22 ended at round 24 or later.

    What Sol found at the end

    • 8 supported agreements
    • 7 useful resolutions
    • 13 reasonable endings without resolution
    • 34 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 20 of 62 conversations. Sol agreed in 6 of those cases.

    The moderator chose when to stop in 44 conversations; 18 reached the limit instead.

    Wording overlap check: 144 of 14521 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 17.4 rounds. Round 3: 1 · Round 4: 1 · Round 5: 1 · Round 8: 1 · Round 9: 4 · Round 10: 4 · Round 12: 4 · Round 13: 6 · Round 14: 4 · Round 15: 1 · Round 16: 4 · Round 17: 2 · Round 19: 2 · Round 20: 2 · Round 21: 3 · Round 24: 3 · Round 25: 19

    Moderator ending claims compared with Sol
    • Claimed agreement: 6 supported agreements, 5 useful resolutions, 0 reasonable endings without resolution, 9 too soon, 0 unclear.
    • Claimed useful resolution: 1 supported agreements, 2 useful resolutions, 0 reasonable endings without resolution, 3 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 0 useful resolutions, 8 reasonable endings without resolution, 10 too soon, 0 unclear.
    • No ending claim: 1 supported agreements, 0 useful resolutions, 5 reasonable endings without resolution, 12 too soon, 0 unclear.
    Judge evidence
    • Sol · score 43.9 Evaluator recipe: 320b75fc6ed0209f9b39e2cbb8276e552b4b6bc182ca181f0ad2763dfdbe6ddb Endings: 8 supported agreements, 7 useful resolutions, 13 reasonable endings without resolution, 34 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 20 found, 218 missed, 196 extra; recall 8.4%.
    • Fable · score 43.8 Evaluator recipe: 9c74f304f277356199dc92d43e9674674534f4121c9a9bdda1e407f78b092577 Endings: 14 supported agreements, 6 useful resolutions, 3 reasonable endings without resolution, 39 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 25 found, 213 missed, 260 extra; recall 10.5%.
    • Gemini · score 40.2 · same provider lab as mediator Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 12 supported agreements, 3 useful resolutions, 13 reasonable endings without resolution, 34 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 17 found, 221 missed, 107 extra; recall 7.1%.
    • Astra (Azure) · score 41.6 Evaluator recipe: c9141e9fd50645c9ae95f59bd2177b746e23f6e389787cd7c5c8e8b477dce89e Endings: 2 supported agreements, 2 useful resolutions, 22 reasonable endings without resolution, 36 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 15 found, 223 missed, 194 extra; recall 6.3%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Astra (Azure)
    Valid first response
    93.5%
    Model errors
    7 of 69 responses
    Repeat calls
    7
    Assessments accepted
    62 of 62 observed
    Error tokens
    111,229
    Error cost
    $3.53
    Recorded tokens
    938,674
    Recorded cost
    $27.96
    Request time
    Not recorded
    Fable
    Valid first response
    91.9%
    Model errors
    14 of 75 responses
    Repeat calls
    13
    Assessments accepted
    61 of 62 observed
    Error tokens
    437,652
    Error cost
    $15.28
    Recorded tokens
    1,990,424
    Recorded cost
    $66.49
    Request time
    Not recorded
    Sol
    Valid first response
    95.2%
    Model errors
    5 of 67 responses
    Repeat calls
    5
    Assessments accepted
    62 of 62 observed
    Error tokens
    68,125
    Error cost
    $0.82
    Recorded tokens
    962,403
    Recorded cost
    $11.61
    Request time
    Not recorded
    Gemini
    Valid first response
    93.5%
    Model errors
    4 of 66 responses
    Repeat calls
    4
    Assessments accepted
    62 of 62 observed
    Error tokens
    41,394
    Error cost
    $0.07
    Recorded tokens
    820,833
    Recorded cost
    $1.60
    Request time
    Not recorded
  28. 28 Gemini 3.8 Flash · Gemma / Novita BF16 · Sol / Azure Benchmark version 3.0 OpenRouter · simple prompt Score 42.0 Judges 3 Ended appropriately 12/62

    Scored by 3 judges · 1 run · Sol, Gemini, Astra (Azure)

    • Sol · 62/62Score 44.3
    • Gemini · 62/62Score 39.1 Same provider lab as mediator
    • Astra (Azure) · 62/62Score 42.6
    Ended appropriately: 12 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 4.2 rounds on average. 0 of 62 reached the 25-round limit; 0 ended at round 24 or later.

    What Sol found at the end

    • 9 supported agreements
    • 0 useful resolutions
    • 3 reasonable endings without resolution
    • 50 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 13 of 62 conversations. Sol agreed in 7 of those cases.

    The moderator chose when to stop in 62 conversations; 0 reached the limit instead.

    Wording overlap check: 45 of 10938 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 4.2 rounds. Round 3: 28 · Round 4: 18 · Round 5: 7 · Round 6: 2 · Round 7: 3 · Round 8: 1 · Round 9: 1 · Round 10: 1 · Round 11: 1

    Moderator ending claims compared with Sol
    • Claimed agreement: 7 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 6 too soon, 0 unclear.
    • Claimed useful resolution: 2 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 1 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 0 useful resolutions, 3 reasonable endings without resolution, 43 too soon, 0 unclear.
    • No ending claim: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    Judge evidence
    • Sol · score 44.3 Evaluator recipe: 269d705430e76bd2f9aea7863510a110f8debb70a5bde654016dbe762b553772 Endings: 9 supported agreements, 0 useful resolutions, 3 reasonable endings without resolution, 50 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 10 found, 228 missed, 184 extra; recall 4.2%.
    • Gemini · score 39.1 · same provider lab as mediator Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 12 supported agreements, 1 useful resolutions, 14 reasonable endings without resolution, 35 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 11 found, 227 missed, 99 extra; recall 4.6%.
    • Astra (Azure) · score 42.6 Evaluator recipe: c9141e9fd50645c9ae95f59bd2177b746e23f6e389787cd7c5c8e8b477dce89e Endings: 6 supported agreements, 0 useful resolutions, 7 reasonable endings without resolution, 49 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 9 found, 229 missed, 159 extra; recall 3.8%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    96.8%
    Model errors
    4 of 64 responses
    Repeat calls
    2
    Assessments accepted
    60 of 62 observed
    Error tokens
    39,141
    Error cost
    $0.54
    Recorded tokens
    684,309
    Recorded cost
    $9.34
    Request time
    Not recorded
    Astra (Azure)
    Valid first response
    100.0%
    Model errors
    0 of 62 responses
    Repeat calls
    0
    Assessments accepted
    62 of 62 observed
    Error tokens
    0
    Error cost
    $0.00
    Recorded tokens
    574,054
    Recorded cost
    $18.16
    Request time
    Not recorded
    Gemini
    Valid first response
    93.5%
    Model errors
    5 of 67 responses
    Repeat calls
    5
    Assessments accepted
    62 of 62 observed
    Error tokens
    37,777
    Error cost
    $0.08
    Recorded tokens
    601,847
    Recorded cost
    $1.41
    Request time
    Not recorded
  29. 29 Claude Haiku 4.5 · Gemma / Novita BF16 · Sol / Azure Benchmark version 3.0 simple prompt Score 31.1 Judges 2 Ended appropriately 5/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 35.0
    • Gemini · 62/62Score 27.2
    Ended appropriately: 5 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 25.0 rounds on average. 62 of 62 reached the 25-round limit; 62 ended at round 24 or later.

    What Sol found at the end

    • 1 supported agreements
    • 0 useful resolutions
    • 4 reasonable endings without resolution
    • 57 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 0 of 62 conversations. Sol agreed in 0 of those cases.

    The moderator chose when to stop in 0 conversations; 62 reached the limit instead.

    Wording overlap check: 0 of 0 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 25.0 rounds. Round 25: 62

    Moderator ending claims compared with Sol
    • Claimed agreement: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    • Claimed useful resolution: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    • No ending claim: 1 supported agreements, 0 useful resolutions, 4 reasonable endings without resolution, 57 too soon, 0 unclear.
    Judge evidence
    • Sol · score 35.0 Evaluator recipe: 269d705430e76bd2f9aea7863510a110f8debb70a5bde654016dbe762b553772 Endings: 1 supported agreements, 0 useful resolutions, 4 reasonable endings without resolution, 57 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 17 found, 221 missed, 200 extra; recall 7.1%.
    • Gemini · score 27.2 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 3 supported agreements, 0 useful resolutions, 4 reasonable endings without resolution, 55 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 17 found, 221 missed, 107 extra; recall 7.1%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    98.4%
    Model errors
    3 of 65 responses
    Repeat calls
    3
    Assessments accepted
    62 of 62 observed
    Error tokens
    41,138
    Error cost
    $0.46
    Recorded tokens
    914,718
    Recorded cost
    $9.99
    Request time
    Not recorded
    Gemini
    Valid first response
    95.2%
    Model errors
    3 of 65 responses
    Repeat calls
    3
    Assessments accepted
    62 of 62 observed
    Error tokens
    11,409
    Error cost
    $0.02
    Recorded tokens
    726,518
    Recorded cost
    $1.11
    Request time
    Not recorded
  30. 30 Mistral Large 3 · Gemma / Novita BF16 · Sol / Azure Benchmark version 3.0 OpenRouter / Mistral · simple prompt Score 31.0 Judges 2 Ended appropriately 8/62

    Provisional · 1 run · scored by Sol and Gemini

    • Sol · 62/62Score 35.8
    • Gemini · 62/62Score 26.2
    Ended appropriately: 8 of 62 · 0 unclear

    Sol found that the conversation ended after a supported agreement, a useful resolution, or a reasonable decision to stop without resolution. “Unclear” means Sol could not determine whether the ending fit one of those categories. This says whether the ending made sense, not whether every conversation succeeded.

    More evidence about this run

    Conversations lasted 25.0 rounds on average. 62 of 62 reached the 25-round limit; 62 ended at round 24 or later.

    What Sol found at the end

    • 2 supported agreements
    • 0 useful resolutions
    • 6 reasonable endings without resolution
    • 54 endings that came too soon
    • 0 unclear endings

    The moderator said it had reached agreement in 0 of 62 conversations. Sol agreed in 0 of those cases.

    The moderator chose when to stop in 0 conversations; 62 reached the limit instead.

    Wording overlap check: 0 of 0 phrase comparisons matched across 62 conversations. Method: prior_party_lowercase_word_4gram/v1.

    Full counts and judge recipes
    Conversation length

    Run 1 averaged 25.0 rounds. Round 25: 62

    Moderator ending claims compared with Sol
    • Claimed agreement: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    • Claimed useful resolution: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    • Claimed no resolution: 0 supported agreements, 0 useful resolutions, 0 reasonable endings without resolution, 0 too soon, 0 unclear.
    • No ending claim: 2 supported agreements, 0 useful resolutions, 6 reasonable endings without resolution, 54 too soon, 0 unclear.
    Judge evidence
    • Sol · score 35.8 Evaluator recipe: 269d705430e76bd2f9aea7863510a110f8debb70a5bde654016dbe762b553772 Endings: 2 supported agreements, 0 useful resolutions, 6 reasonable endings without resolution, 54 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 14 found, 224 missed, 203 extra; recall 5.9%.
    • Gemini · score 26.2 Evaluator recipe: 7eafa7f9912f103b2e007f373276bd39816a81e7cbf9e1197b143a72cca13559 Endings: 0 supported agreements, 1 useful resolutions, 6 reasonable endings without resolution, 55 too soon, 0 unclear. Hidden-fact checks: 62 with diagnostics; 62 with scores. Hidden-fact results: 19 found, 219 missed, 106 extra; recall 8.0%.
    Judge reliability

    Invalid judge answers and their recorded cost, separate from the mediator score. Provider failures are not counted as model errors. Totals cover recorded responses; missing usage and timing stay unknown.

    Sol
    Valid first response
    96.8%
    Model errors
    2 of 64 responses
    Repeat calls
    2
    Assessments accepted
    62 of 62 observed
    Error tokens
    26,436
    Error cost
    $0.28
    Recorded tokens
    876,063
    Recorded cost
    $9.30
    Request time
    Not recorded
    Gemini
    Valid first response
    98.4%
    Model errors
    1 of 63 responses
    Repeat calls
    1
    Assessments accepted
    62 of 62 observed
    Error tokens
    11,930
    Error cost
    $0.02
    Recorded tokens
    734,147
    Recorded cost
    $1.14
    Request time
    Not recorded

Version 3.1

  1. Mistral Large 4 · Gemma / Novita BF16 Benchmark version 3.1 OpenRouter / Mistral ZDR, high effort · simple prompt Incomplete · Sol scored 0/62
    • Sol · 0/62
Reference scores 2

What the same kind of conversation reaches with no mediator at all. Read it as context for the results above: it is never ranked, and no difference is computed.

  • No mediator · Gemma parties July 2026 historical reference 15

    Earlier result · kept from the original publication

  • No mediator · Stheno parties July 2026 reference 17

    Earlier result

Earlier results · not compared 8

These results used earlier test or scoring rules. They stay public as separate records.

  • GLM 5.3 Flash August 2026 method 43.6

    Earlier result

  • GLM-5.2 July 2026 method 52

    Earlier result

  • gpt-oss-120b July 2026 method 49

    Earlier result

  • Kimi K2.5 July 2026 method 45

    Earlier result

  • Qwen3-235B July 2026 method · Stheno parties 46

    Earlier result

  • Qwen3-235B · Gemma parties July 2026 participant check 39.5

    Earlier result

  • Reflective control July 2026 reference 40.5

    Earlier result

  • Sonnet 5 July 2026 method 51

    Earlier result

Each score is the equal-weight average of every judge that scored all 62 simulated conflicts in one complete run; the judges counted are drawn as colored rows in the bar and named in the row's evidence. One or two judges make a result provisional. Rows are ordered by score within each version; a small gap between rows is not evidence that one system is better. Reference scores are never ranked; the no-mediator run was scored under earlier rules, so read the gap as context rather than a measured difference. Earlier results used different rules and are shown only as separate records. Read the full scoring method and limitations.

Estos resultados describen simulaciones de modelos, no el rendimiento en disputas reales entre personas. La página de investigación explica el método completo y sus limitaciones.

Cómo funciona

  • La misma prueba: todos los sistemas afrontan los mismos 62 conflictos redactados para la evaluación. Los modelos de lenguaje interpretan a ambas partes.
  • Final sin guía: el mediador decide cuándo hablar, esperar o terminar dentro de un intervalo fijo de entre 3 y 25 rondas que no se le comunica.
  • Puntuación: después, Sol puntúa cada conversación terminada de 0 a 100. Otros evaluadores pueden puntuar esas mismas conversaciones; la puntuación de la fila es la media, con igual peso, de todos los evaluadores que hayan puntuado las 62.
  • Final adecuado: Sol también indica si tenía sentido terminar. Esto no significa que se hayan resuelto todos los conflictos.
  • Provisional: una ejecución puntuada por uno o dos evaluadores. Las filas se ordenan por puntuación dentro de cada versión; una pequeña diferencia no demuestra que un sistema sea mejor.
  • Otros evaluadores: se muestran al desplegar la fila, cada uno con su puntuación. Con tres o más evaluadores, el resultado deja de ser provisional.

Hay una muestra de los datos en Hugging Face: mediationbench-sample .

Los resultados anteriores, obtenidos con otras reglas, están en la página de investigación . No se comparan con la clasificación actual.

Para evaluar un sistema con MediationBench a través de hai.ai , escribe a hello@hai.io . Este sitio publica resultados y no acepta directamente solicitudes de ejecución de evaluaciones.

Preguntas frecuentes

¿Qué es MediationBench?
MediationBench compara mediadores de IA en los mismos 62 conflictos simulados. Cada sistema trabaja con las mismas partes, interpretadas por modelos, y con las mismas reglas. Sol puntúa las conversaciones terminadas y se pueden añadir otros modelos evaluadores.
¿Hasta qué punto son fiables estos resultados?
Un resultado provisional procede de una ejecución completa puntuada por uno o dos modelos evaluadores. Aporta indicios iniciales útiles. Las filas se ordenan por puntuación dentro de cada versión, pero una pequeña diferencia no demuestra que un sistema sea mejor que otro.
¿Demuestra MediationBench que la mediación con IA funciona con personas?
No. Todas las disputas de este estudio son sintéticas y cada parte está interpretada por un modelo de lenguaje. Los resultados evalúan el comportamiento de los modelos en condiciones controladas, no su seguridad ni su eficacia con personas.