TEE confidential LLM inference
Confidential LLM Inference: Performance and Cost Across CPU and GPU TEEs
Reported benchmarks
| platform | model | overhead percent | overhead metric | gpus | note | latency overhead percent | workload |
|---|---|---|---|---|---|---|---|
| H100 cGPU (GPU TEE) | Llama2 7B | 4–8 | throughput | 1 | "throughput penalties of 4-8% that diminish as batch and input sizes grow" (Abstract); reproduced as the "Single resource" row of the summary table. Insight 10: "GPU TEEs achieve less than 10% overheads, which decreases with larger batch and input sizes." | — | — |
| Intel TDX (VM TEE) | Llama2 7B/13B/70B | 5–10 | throughput | — | Summary table: ~5-10% single-resource. Latency overhead is 2x the throughput overhead and is never quoted downstream. | 20 | — |
| Intel SGX (process TEE) | — | 4–5 | throughput | — | — | — | — |
| Intel TDX (VM TEE) | — | 6–7 | — | — | Insight 12 -- entire RAG pipeline inside TDX lands at the same overhead as plain inference. | — | RAG pipeline (BM25 / reranked BM25 / SBERT, Elasticsearch, BEIR) |
Approach
Hardware TEE (NVIDIA H100 confidential computing; Intel TDX and SGX CPU TEEs)
Notes
The number that frames the alternatives argument, now read from the source and standing up -- with conditions. 4-8% is real, primary, and stated in the abstract. It is also: ONE H100, ONE model that fits on it (Llama2 7B), measured against a non-confidential CLOUD VM rather than bare metal, with GPU memory LEFT UNENCRYPTED, and excluding network protection that the paper says is required and costs up to 90%. Multi-GPU serving falls off a cliff (3GB/s vs 40GB/s interconnect). So the honest form of the claim is narrower than "TEEs cost 4-8%": it is "a single-GPU, 7B-class confidential deployment with unencrypted HBM costs 4-8%". That is still three orders of magnitude cheaper than ZK, so the strategic conclusion survives intact -- if the verifier will trust NVIDIA's attestation, ZK is a very expensive way to buy similar confidence. But the headline figure is not a general constant, it is the best corner of a cost surface, and the paper's own authors expect it to rise as the hardware gets more secure.
Our reading
What is new
The first systematic measurement of what it costs to run a whole LLM inference pipeline inside a trusted execution environment, on both sides of the hardware, and end to end rather than in fragments. Prior TEE-for-ML work offloaded only pieces of the model; this runs full Llama2 (7B, 13B, 70B) entirely inside Intel TDX and SGX, accelerated by AMX, and then runs the same workload on an NVIDIA H100 with confidential computing enabled, so the two can be put side by side.
The paper is a benchmarking study and does not pretend otherwise. Its output is twelve numbered "insights" and a comparison table grading each TEE on hardware, software, overhead and cost. The useful ones for us:
- GPU TEEs cost single-digit percentages, and the overhead shrinks as batch size and input length grow, because the fixed costs (kernel launches, PCIe bounce-buffer transfers) amortize against rising arithmetic intensity.
- CPU TEEs are not obviously worse, and are sometimes better, on small models with small batches, an H100 sits unsaturated, and a TDX CPU can be more cost-effective and strictly more secure.
- An entire RAG pipeline, Elasticsearch and all, runs inside TDX at essentially the same overhead as bare inference.
The last one is the most under-appreciated. The confidentiality boundary can enclose the retrieval database, not just the model, and that is where the sensitive documents actually live.
What it actually proves
That a single-GPU, 7B-class confidential deployment with unencrypted GPU memory costs a few percent of throughput, measured against a non-confidential cloud VM.
Every clause there is load-bearing, and the repo has been quoting the headline without them.
- It is a confidentiality result, not a verifiability one. The paper never mentions zero-knowledge proofs, zkML, or SNARKs, the strings appear zero times. It measures the cost of hiding the model and the prompt from the cloud operator. The TEE-versus-ZK comparison this repo builds on top of it is ours. The overhead figure transfers cleanly (it is simply the cost of running the model in the enclave, whatever you want the enclave for), and hardware attestation does buy a form of integrity. But the paper draws no comparison to proof systems and must not be cited as though it did.
- The baseline is not bare metal, and the authors say so plainly: "As our machine is rented, we do not have access to bare metal and present the results for raw and Confidential GPUs (cGPU)." The GPU comparison is confidential-Azure-VM against non-confidential-Azure-VM. For a deployment decision that is arguably the right baseline. It is not "overhead versus the hardware's ceiling."
- The CPU latency overhead is double the throughput overhead, and only the throughput number ever gets quoted. The abstract gives both.
What to distrust
Nothing about the measurement, this is a careful, honest paper that grades its own subject harshly and publishes the numbers that undercut it. What deserves suspicion is how the figure is used, including by us. Two conditions gut the general reading.
The 4–8% is the "Single resource" row of the summary table. The moment a model does not fit on one H100, which is every model anyone would actually want to protect, you need tensor or pipeline parallelism, and the paper reports what that costs:
As the cGPU instances do not support RDMA and GPUdirect, all data is transferred through the CPU, capping throughput at 3GB/s (considerably lower than the non-confidential 40GB/s).
That is better than a 13× interconnect penalty, hitting exactly the communication pattern multi-GPU serving depends on. The same table grades the H100 cGPU "D (NVLINK unprotected)" on scale-up, the worst grade it assigns anything.
And the paper notes that a network protection scheme such as IPsec "is required on top of both CPUs and GPUs, which also introduces an overhead of up to 90%." That is not in the 4–8% either.
H100s do not encrypt their HBM memory [31], compared to CPUs that do.
The summary table grades the H100 cGPU "B (HBM unencrypted)" on memory, weaker than the Intel CPU TEEs it is being benchmarked against, which score C. So the cheap TEE is also the leakier one, and the comparison flatters it twice.
The authors then say what closing the gap will cost:
While B100s address these issues, we expect that they will add a non-negligible overhead to H100s' results, since we identified memory encryption as a significant cost in CPUs.
The people who produced the number expect the number to get worse as the hardware gets more secure.
Does the strategic claim survive? Yes, and this is worth stating clearly, because the honest reading is still bad news for zkML. Even if you triple the overhead to absorb the memory encryption the H100 is currently skipping, and even if you take the multi-GPU penalty seriously, TEE-protected inference remains three orders of magnitude cheaper than proving the same forward pass. The argument in alternatives does not depend on the difference between 4% and 40%; it depends on the difference between tens of percent and a thousand times, and that gap is not in doubt.
What changes is the rhetoric. "TEEs cost 4–8%" is not a constant of nature, it is the best corner of a cost surface, measured on the smallest interesting model, on one GPU, with a security hole open. The defensible sentence is the conditional one, and the SoK should use it.
The one thing the paper cannot tell you is the thing the whole argument actually turns on: whether the verifier is willing to trust NVIDIA's attestation chain. That is not a performance question, and no benchmark will settle it. ZK's entire claim to relevance is that some verifiers, an adversarial counterparty, a regulator, a public chain, structurally cannot make that assumption. This paper prices the alternative; it does not price the trust.
Citation neighbourhood
Builds on
- none recorded
Cited by, in this corpus
- none recorded
Edges are a proxy: paper A's text mentions B anywhere (body or bibliography). See the full graph.
Discussed in
Other recorded fields
trust_assumption: Trust the hardware vendor and the attestation chain. Vulnerable to side channels; no public verifiability of the computation itself. mentions_zk: false baseline_note: 'NOT bare metal, and the paper says so: "As our machine is rented, we do not have access to bare metal and present the results for raw and Confidential GPUs (cGPU)." The GPU baseline is a NON-CONFIDENTIAL AZURE VM. That is arguably the right comparison for a deployment decision, but it is not "overhead versus raw hardware".' scale_up_note: 'THE CAVEAT THAT REFRAMES THE HEADLINE, and it is the paper''s own. The 4-8% is a SINGLE-GPU number on a model that fits on one H100. Multi-GPU confidential inference is not 4-8%: "As the cGPU instances do not support RDMA and GPUdirect, all data is transferred through the CPU, capping throughput at 3GB/s (considerably lower than the non-confidential 40GB/s)." That is a >13x interconnect penalty, and it hits exactly the tensor- and pipeline-parallel patterns any production-scale serving stack uses. The summary table grades the H100 cGPU "D (NVLINK unprotected)" on scale-up.' security_gap_note: 'AND THE 4-8% BUYS AN INCOMPLETE TEE. "H100s do not encrypt their HBM memory [31], compared to CPUs that do." The summary table grades the H100 cGPU "B (HBM unencrypted)" on memory -- WEAKER than the CPU TEEs it is being compared against. The authors expect closing that gap to cost more: "While B100s address these issues, we expect that they will add a non-negligible overhead to H100s'' results, since we identified memory encryption as a significant cost in CPUs." Separately, network protection (IPsec) is "required on top of both CPUs and GPUs, which also introduces an overhead of up to 90%".' external_context: 'NOT FROM THIS PAPER. The strings "Phala" and "OpenRouter" occur ZERO times in the PDF. MOVED OUT OF `notes` 2026-07-13: the claim that Phala''s GPU TEEs are offered for inference on OpenRouter was sitting at the end of an otherwise entirely primary-sourced notes block, which is precisely how a vendor claim gets laundered into a citation -- and it duly ended up asserted in content/alternatives/tees.md as though this paper said it. The claim may well be TRUE; it is simply not this paper''s. Attribute it to phala.network / openrouter.ai directly, or drop it.'