TCSExam: What GPT-6 Astra and Kimi K3.1 Pro Can Actually Prove (Nov 2026)

On this page

1. What is TCSExam?
2. How is this different from a math competition benchmark?
3. Who leads the leaderboard?
4. What do the per-track scores show?
5. What new results did the models produce?
6. How much does a hint change the result?
7. What happened on the open conjectures?
8. What does TCSExam-Lite add?
9. What does the benchmark actually measure?
10. What is still missing?
11. Frequently asked

What is TCSExam?

Most math benchmarks end with a final answer. TCSExam ends with an artifact. The benchmark covers four kinds of theoretical computer science work, and each kind has its own submission format and checker.

The Full set has 100 tasks. Each attempt gets up to 12 hours of wall-clock time. Models can call tools, write programs and read feedback from the environment. The checker is Lean or an executable task checker, not a second model judging whether the answer looks right.

Tasks go through their own pipeline. An authoring agent writes the task, a separate review agent audits it, tests and pilot runs follow, and an expert approves it. Mismatched statements, scorer holes and anomalous results are sent back for repair.

The paper is arXiv 2611.07962v1, submitted on November 5, 2026. Its authors include Yifan Luo, Zhengying Liu, Yang Yuan, Zhilin Yang and Andrew Chi-Chih Yao. Zhilin Yang is the founder and CEO of Moonshot AI. Andrew Yao is a Turing Award winner at Tsinghua. The project publishes a public repository with task materials and tools.

Scroll horizontally to see the full figure.

How is this different from a math competition benchmark?

Competition benchmarks usually have a short answer and a ground truth. A grader can compare strings or values. Research outputs do not reduce to that shape.

TCSExam pushes the judgment into the artifact. A proof compiles against Lean or it does not. A solver runs on the instance set or it does not. A construction satisfies its certificate or it does not. That removes a large source of grading noise, and it makes failures more informative: the model does not lose points for writing style or for choosing an unexpected route.

The 12-hour budget and the open-conjecture track push the benchmark closer to research practice. The model can write code, test hypotheses, and revise. It can also retrieve published results, which is why the open-conjecture section needs a careful reading.

Who leads the leaderboard?

GPT-6 Astra ranks first on the Full set with an overall index of 71.9. Kimi K3.1 Pro ranks second at 57.4. Kimi K3, DeepSeek V4.1 Flash 1017, GLM-5.3 and Qwen3.8 Max cluster between 46.8 and 44.8. Grok 4.6 and DeepSeek V4.1 Flash 0910 finish at 40.5 and 40.4.

What do the per-track scores show?

The aggregate ranking hides the useful part. On bare theorem proving, Astra solves 14 of 20 tasks, 70.0%. Kimi K3.1 Pro solves 23 of 60 attempts, 38.3%, across nine distinct theorems. The other six agents do not exceed 11.7%.

Scroll horizontally to see the full figure.

The ordering is not uniform. Kimi K3.1 Pro solves the PCP alphabet-soundness task in all three attempts; Astra does not solve it. DeepSeek V4.1 Flash 0910 completes no bare theorem proof yet reaches 86.6% of the reference on constructions, and it is one of four agents that reach 630 edges on the Erdős graph task. Proof, program search and construction do not rise and fall together.

For a model developer, that spread is a diagnostic. For a user, it means different research steps may need different models.

What new results did the models produce?

Several outputs go beyond the reference values shipped with the tasks.

Kimi K3 builds a graph for the Erdős girth-5 extremal problem with 124 vertices and 630 edges. The graph has no triangles and no quadrilaterals, and it exceeds the 629-edge reference. Qwen3.8 Max, Grok 4.6 and DeepSeek V4.1 Flash 0910 also reach 630 edges, and Astra matches the parameter.

Kimi K3 also constructs a binary [[25, 7, 6]] stabilizer code, improving the reference minimum distance from 5 to 6 for 25 physical qubits and 7 logical qubits. Astra reaches the same parameter.

Astra proves A(22,10) ≤ 82, tightening the published upper bound of 84 to 82.

On the open-conjecture side, Astra proves every special case of the Alon-Jaeger-Tarsi conjecture over F5 for n ≤ 9. The n = 9 case goes beyond the classic n ≤ 7 result and a recent preprint's n ≤ 8. The proof passed independent Lean verification using standard axioms. It is a verified special case, not a resolution of the full conjecture.

One caveat applies to all of these. Exceeding a reference is not the same as solving the underlying problem, and it does not prove global optimality. What it does show is a checkable local improvement. That is a different claim from a plausible answer.

How much does a hint change the result?

The paper runs the same 20 theorems under three context levels. Bare gives the statement and definitions. Ladder adds intermediate lemmas. Guided adds a proof idea. Each setting gets three attempts.

Kimi K3's success count more than doubles under Guided, and the number of theorems it solves at least once rises from 3 to 7. DeepSeek V4.1 Flash 0910 goes from no complete proof to 11.

That changes how to read a failure. A model may have the ability but lack a decomposition or a route. Human-written lemmas and proof ideas still change search efficiency by a large factor.

More guidance is not always better. Some tasks score lower under Guided than under Ladder. A route that is too specific can constrain the search it was meant to help.

What happened on the open conjectures?

Only two agents score above zero on the open-conjecture track: DeepSeek V4.1 Flash 1017 at 5.6% and Kimi K3.1 Pro at 1.1%. Both retrieve and adapt counterexamples and Lean proofs that OpenAI had already published, with internet access. That is not an independent solution from scratch.

The other 28 conjectures have no accepted complete solution.

Partial progress still appears. Astra's F5 result covers every special case up to n = 9. A special case can be a starting point for later work, but it is not the full conjecture. The benchmark's own framing is useful here: a nonzero score can mean a new proof, a retrieved proof, or a special case.

What does TCSExam-Lite add?

The paper also releases a 20-task Lite set, five tasks per track. It cuts the number of attempts by 80% at the same per-attempt budget, which makes the benchmark cheaper to run and easier to repeat.

Opus 5.5 leads Lite with an overall index of 72.3, ahead of GPT-6 Astra at 67.0 and Kimi K3.1 Pro at 53.5. On the Lite theorem track, Opus 5.5 solves 4 of 5 tasks, 80.0%. The rankings of the eight shared agents on Full and Lite are strongly aligned, with a Spearman correlation of 0.93.

The Lite result is a useful check on the Full result. It suggests the ranking is not an artifact of the 100-task sample, at least for the models tested. It does not remove the need for independent replication.

What does the benchmark actually measure?

TCSExam is best read as a map of where current models can leave a checkable artifact. With a clear problem, a generous budget and a verifier that returns real feedback, models can complete parts of proofs, optimize solvers and improve constructions. They still do not choose research directions, judge which results matter, or close major open problems on their own.

The rejection of LLM-as-judge is the part worth copying. A proof either compiles or it does not. A solver either runs or it does not. A construction either satisfies its certificate or it does not. That removes a large source of grading noise.

For teams building research agents, three practical reads follow. Match the model to the task type instead of assuming one model for every step. Give the agent a verifier and enough budget to iterate. Treat guidance as a tunable input, because the right amount varies by task.

What is still missing?

• 100 tasks is a small sample of theoretical computer science.
• The results are self-reported and the benchmark is new. No independent replication is available yet.
• Lean verifies the proof, not the importance of the theorem. A correct proof of a minor lemma scores like a correct proof of an important one.
• Retrieval blurs the line between solving and adapting. The open-conjecture track separates the two more carefully than a headline score, but it does not eliminate the ambiguity.
• Cost is absent from the public summary. A 12-hour budget per attempt is generous, and the dollar cost of running the Full set is not part of the ranking.