The benchmark
Tasks in CryptanalysisBench are real cryptographic ciphers, taken almost entirely from NIST (and
similar) standardization competitions. We give a language model the implementation of one of these
ciphers and ask it to produce an attack that breaks the scheme. To count as a break, the attack must win
a formal security game: a precise, machine-checkable definition of what “breaking”
the scheme means, which fixes exactly what the attacker is allowed to ask for and what it must produce to
win (for example, recovering the secret key, forging a signature, or finding a hash collision).
To solve a task, the agent reads the algorithm's implementation, experiments with different
cryptanalytic techniques, and finally submits a self-contained attack script to the verifier. The
verifier re-runs that script against the chosen security game with fresh randomness and checks whether
the attack actually succeeds. Crucially, the verifier enforces that the agent only ever interacts with
the cipher through the oracle queries the game permits (for example, asking for the encryption of
plaintexts it chooses, but never for the key itself). This ensures the attacker never sees the secret
key or any other information that would let it cheat.
An example security game
Concretely, take the Key Recovery game against a symmetric cipher. The agent (the adversary) plays against a challenger that holds the
secret key:
| Step | Party | Action |
|---|
| 1 |
Challenger |
Generates a random secret key $k$. |
| 2 |
Adversary (the agent) |
Sends an arbitrary number of plaintexts it chooses to the encryption oracle and observes the resulting
ciphertexts. However, it never sees the secret key $k$. |
| 3 |
Adversary |
Submits a candidate key $k'$. |
| 4 |
Outcome |
The agent wins if the recovered key matches the true key, $k' = k$. |
Other games keep the same structure but change the goal and the permitted queries. Examples include
forging a signature or an authentication tag, finding a hash collision, or distinguishing which of two
plaintexts was encrypted.
What's covered
The 191 tasks are drawn primarily from four NIST standardization competitions (AES, SHA-3,
Lightweight Cryptography, and Post-Quantum Cryptography), along with widely-studied production and
academic ciphers. They span every major family of cryptographic primitive, so an agent's ability is
measured across fundamentally different attack surfaces.
| Primitive family |
Example schemes |
Tasks |
| AEAD | Elephant, Sparkle, Xoodyak, SpoC | 55 |
| Hash functions | BLAKE, Grøstl, JH, Skein | 51 |
| Key encapsulation (KEM) | Classic McEliece, BIKE, FrodoKEM, SABER | 41 |
| Block & stream ciphers | AES, ChaCha, Serpent, Simon, Speck | 22 |
| Digital signatures | Rainbow, GeMSS, Picnic, qTESLA | 16 |
| Public-key encryption | Compact-LWE, Giophantus, LEDApkc, McNie | 6 |
The tasks are organized into two tiers and a challenge set:
-
Tier 1: 49 schemes with known practical breaks, which calibrate current
capability.
-
Tier 2: 142 schemes with no known practical break, evaluated both at full
strength and as scaled-down variants (easy / medium / hard).
-
Challenge set: production ciphers at the frontier of cryptanalytic knowledge
(e.g. 7-round AES), included as a long-horizon target.
Evaluation results
We evaluate five frontier models: Claude Opus 4.8, Sonnet 5, Mythos 5,
GPT-5.5, and the open-weights GLM-5.2. Every model breaks a large majority of Tier 1
(65–86% of the 49 known-broken schemes). Tier 2 is far harder: even scaled down it remains
far from saturated, which is precisely what makes progress on it a meaningful capability signal.
| Model |
Tier 1 |
Tier 2 |
| Easy | Medium | Hard | Full |
| GLM-5.2 | 65.3% | 14.3% | 9.3% | 8.5% | 4.4% |
| Claude Opus 4.8 | 73.5% | 22.1% | 13.6% | 7.0% | 5.9% |
| Claude Sonnet 5 | 75.5% | 17.1% | 8.6% | 7.7% | 6.7% |
| GPT-5.5 | 75.5% | 19.3% | 11.4% | 8.5% | 7.4% |
| Claude Mythos 5 | 85.7% | 35.7% | 23.6% | 16.9% | 8.9% |
Success rates on CryptanalysisBench. Tier 1 is the percentage of the 49 tasks broken on the
original specification. Tier 2 reports per-variant success across the easy / medium / hard
scaled-down configurations and the full-strength scheme. A task counts as broken if the agent produces
a working attack in at least one of two independent runs.
Per-task performance across the models (Tier 1 and Tier 2), conditioned on runtime (left),
cost (middle), and token budget (right). Claude Mythos 5 shows considerably higher performance than
the other models when allowed enough test-time compute.
Novel cryptanalysis
Beyond reproducing known results, the models surfaced genuinely new findings on full-strength schemes:
-
SpoC AEAD. A full 128-bit key-recovery attack on the unmodified scheme, using only two
oracle queries. This attack was reached independently by two models (Mythos 5 and Sonnet 5)
from the same two queries, which points to a real property of the design rather than a lucky
run. To our knowledge, previously unreported.
-
KINDI. Mythos 5 built a decryption-reaction oracle from the scheme's own code to
recover the secret key, exposing an error in the KEM's published CCA-security proof —
also, to our knowledge, not previously reported.
Example agent traces
To make the agents' behaviour concrete, the viewer below lets you step through complete attack traces
on tasks the agents solved. Each tab is one task; scroll within a trace to follow the model's reasoning,
its shell commands and their terminal output, and the oracle responses it acts on. Use the tabs to switch
between traces.
Real, unedited traces from CryptanalysisBench evaluation runs across several models. Each is a task the
agent solved end-to-end; long reasoning and terminal-output cells scroll within their own box.
Citation
@misc{fluri2026cryptanalysisbench,
title = {CryptanalysisBench: Can LLMs do Cryptanalysis?},
author = {Fluri, Lukas and Shafran, Avital and Carlini, Nicholas and
Jagielski, Matthew and Nasr, Milad and Dunkelman, Orr and
Ronen, Eyal and Tram\`er, Florian},
year = {2026},
eprint = {2607.18538},
archivePrefix = {arXiv},
primaryClass = {cs.CR},
url = {https://arxiv.org/abs/2607.18538}
}