Stealing Part of a Production Language Model
Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A. Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Itay Yona, Eric Wallace, David Rolnick, Florian Tramèr
cs.CR
2024-03-11
First precise attack on a production transformer: logit-bias queries reconstruct the full logit vector and SVD recovers the last layer; OpenAI Ada and Babbage cost $4-$12 to extract.
Closed production models like GPT-4 and Gemini keep their weights and architecture secret but expose the model behind an API. High-fidelity model-stealing attacks have only worked on small ReLU networks; on production transformers they fail because models take token inputs, use non-ReLU activations, add attention, residuals, and layer norm, are orders of magnitude larger, and expose only low-precision outputs. The paper asks how much an adversary can precisely learn about a production model from queries alone. The answer is the last layer.
A transformer's final step is f(p)=softmax(W·g(p)): g compresses the input into an h-dimensional hidden vector and W, the embedding projection matrix (unembedding), lifts it back to vocabulary size l, with h much smaller than l. Both attacks rest on this structure.
Stealing the hidden dimension h. The logit vector for any prompt is l-dimensional but lives in an h-dimensional subspace, because W lifts from h dimensions. Query many prompts, stack the logit vectors into a matrix Q=W·H, run SVD, and the count of significant singular values is h, found via the largest multiplicative gap between consecutive singular values.
Stealing the full layer W. The SVD gives U·Sigma = W·G for some h-by-h matrix G, so W is recovered up to an affine transformation, then aligned by least squares. A hard limit applies: with residual connections, exact recovery of W is impossible, only recovery up to a transformation. A tighter orthogonal recovery (about h^2/2 missing parameters) is information-theoretically possible but has no efficient algorithm.
Real APIs do not expose logits directly; they give top-K token log-probabilities plus an adjustable logit bias. The attack uses logit bias to push a target token into the top K, sweeps the bias, and stitches together the full logit vector, then runs the SVD attack. With top-5 that yields four logits per query; even a top-1 API with bias restricted to plus or minus one leaks information via the difference between biased and unbiased queries.
On open models (GPT-2, Pythia, LLaMA), hidden-dimension error is 0 or 1 in five of six cases, and weight RMS is 3e-5 to 8e-5, which is 100-500x better than a random-init baseline of 2e-2. On OpenAI models: Ada's full projection matrix for about $4 (RMS 5e-4) and Babbage's for about $12 (RMS 7e-4); dimension extraction costs about $1 and $2 respectively. This confirms for the first time that Ada and Babbage have hidden dimensions of 1024 and 2048. A limited attack on gpt-3.5-turbo-instruct recovers the dimension for about $200 and would cost under $2,000 for the full layer.
This is the first time any precise parameter information has been extracted from a production transformer. The alarming part is the cost: a few dollars to a few thousand dollars for the full last layer, within reach of a script kiddie. Knowing the last layer alone does not clone the model, but it leaks the hidden dimension and could seed downstream attacks, such as watching a fine-tuning API for changes in that layer. The practical lesson for builders is that innocuous API features like logprobs and logit bias are an attack surface that leaks weights.
The attack recovers only the last layer, and only up to a transformation when residual connections are present, not bit-exact; there is no efficient algorithm for the tighter orthogonal recovery. It depends on the logit-bias feature, which providers are phasing out. The authors concede there is no immediate downstream harm from knowing this one layer, and the real hazard level is an open question. Defenses exist (splitting the last layer with a nonlinearity, post-hoc widening of W, logit noise, rate-limiting logit bias), but each either hurts functionality or is defeated by Sybil accounts. Disclosure was responsible: affected parties were notified, OpenAI approved the extraction, data was deleted, and OpenAI and Google have since changed their APIs.