Crypto Expert Tests LLM Decryption, Exposing AI Toolchain Flaws

Cryptography expert Matthew Green recently launched a side project to test the practical capabilities of frontier LLMs and coding agents in cracking historical classical ciphers. This test not only revealed the current potential and limitations of AI in cryptanalysis but also unexpectedly exposed deep flaws in existing AI toolchains, triggering a profound discussion on the future paradigm shift in software development.

Testing Details and Capability Boundaries

Green found that while LLMs possess vast cryptographic knowledge, they struggle to translate it into actual decryption operations. Without external tool assistance, even top-tier coding agents get bogged down in slow trial-and-error attempts and eventually give up. The current viable solution is to brute-force the creation of specific tools for a given cipher and strictly constrain the Agent to perform multi-hypothesis verification and self-scoring. After extensive development using Claude and Codex, he concluded that AI programming tools excel at cracking substitution ciphers, homophonic ciphers, and some polyalphabetic ciphers, making them some of the best tools available today, though this approach remains difficult to generalize overall.

Toolchain Flaws and Agent Preferences

In practice, Green observed that current AI coding agents act blindly when handling specific tasks, often haphazardly scraping low-quality open-source projects for attempts, making success highly random. He criticized the local MCP server solutions favored by current designers, calling them untested (lacking dogfooding) and overly cumbersome. Due to supply chain risks and usability hurdles, AI Agents are wary of running local servers with various capabilities and instead prefer directly downloading and running command-line (CLI) tools.

Software Development Paradigm Shift

Based on these pain points, Green highlighted a noteworthy new trend: software developers' primary consideration is shifting from "human comfort" to "AI coding agent comfort." For example, adding Agent-specific instructions in a README can trigger security warnings that deter the Agent. He believes the future goal is to build a high-quality tool ecosystem that allows agents to autonomously discover and invoke tools. This view was echoed by other professionals; @Financial_Ad_7297 and @TheZachMueller both noted that as AI proliferates, software interfaces will increasingly skew towards Agent usage, emphasizing APIs, structured data, and permission controls, with humans assuming more supervisory roles.

2026-07-19 ~ 2026-07-20 · 13 related posts