Chip Huyen has spent years analyzing ways to reduce the cost of AI inference without depending on new hardware. Her guidance, first presented at the P99 conference in 2025, stays vital as the industry moves toward agentic systems that require even greater computational resources. The fundamental issue persists: training a model represents a one-off expense, while inference costs recur indefinitely. For cutting-edge models, the ratio of training to inference expenses can fall anywhere between 1:10 and 1:100, and reasoning models—with their heavy token consumption—further tilt this balance. Without careful optimization, these ongoing costs prevent providers from recovering their initial investment, which is why so many AI companies still face profitability challenges.
Huyen centers her work on three core latency measurements that directly affect user experience and operational expenses:
- Time to first token (TTFT): The interval before a user receives any response.
- Time per output token (TPOT): The average delay between successive tokens.
- End-to-end latency: TTFT plus TPOT multiplied by the number of output tokens minus one.
Reasoning models introduce additional complexity. Huyen observed that the first token generated may not be the first one displayed—the model might process internally before producing visible output. Some vendors track time to publish, which captures when users actually see the initial token. The most critical metric depends on the specific application. For instance, a chat interface prioritizes TTFT, while a coding assistant focuses on TPOT.
Beyond latency, Huyen stresses goodput—the share of requests meeting quality standards. Throughput alone (requests handled per minute) is insufficient. If an application targets 200 ms TTFT and 100 ms TPOT but only 30% of requests achieve those targets, the remaining computations are wasted resources.
Huyen rejects hardware-based solutions as impractical for most organizations, particularly since scaling with additional machines (replica parallelism) becomes prohibitively expensive and complicated. Instead, she outlines two primary approaches: model optimizations and service optimizations.
Model optimizations: Adjusting the architecture
These methods alter the model itself to cut costs and improve speed. The two most widely used are quantization and distillation.
Quantization reduces the precision of weights and activations, from 32-bit (4 bytes) to 8-bit (1 byte). The quality impact is minimal, but the efficiency gains are substantial: fewer bits mean faster calculations and lower memory usage. Huyen noted that most organizations no longer operate models at full precision. “If you do additions bit by bit and each weight is 32 bits, you have to do it 32 times. If it’s 8 bits, you only have to do it eight times,” she said. The tradeoff is a small quality hit.
Distillation involves using a large model to generate training data for a smaller model. For example, Huyen mentioned using a truly large model and collecting a large set of prompts to train a smaller model on its responses. Proceed with caution, though. Huyen warned, “A lot of model providers have the condition that they do not allow their models to be used to train competitive models. So even though it’s a very common technique, you need to check licensing.”
Service optimizations: Improving request handling
These techniques leave the model’s weights unchanged but refine how requests are processed. The most effective include:
- Batching: Combining requests for joint processing. Static batching waits for a complete batch before starting, improving efficiency but delaying early responses. Dynamic batching triggers at set intervals (e.g., every 15 ms), balancing speed and efficiency. Continuous batching handles the case where requests finish at wildly different times, which is common with LLMs. One request asks for the capital of Vietnam; another kicks off deep research. With static or dynamic batching, the finished request’s slot sits idle until the slowest one completes, and new requests queue up behind it. Continuous batching returns each request as it finishes and fills the spot with another request, improving compute resource utilization and latency.
- Separating prefill and decode phases: Prefill (processing input tokens) can run in parallel, while decode (generating output) remains sequential. Most modern services divide these stages to optimize either TTFT (allocating more resources to prefill) or TPOT (focusing on decode).
- Parallel processing strategies: Replica parallelism duplicates the model across machines, tensor parallelism splits large matrices across devices, and pipeline parallelism divides the model by layer, allowing requests to flow through stages like an assembly line.
One year after her initial presentation, Huyen’s recommendations remain highly relevant. The rise of agentic systems, with their multi-step workflows and tool integrations, has only increased the urgency of these optimizations. Reasoning models, which consume tokens at high rates, make inference costs even more pressing. Meanwhile, metrics like TTFT, TPOT, and goodput have become essential benchmarks, though their tradeoffs now demand more careful evaluation.
Huyen’s closing advice from 2025 still applies: not every inference provider delivers equal value. Many claim cost and latency reductions but compromise on model performance. When assessing a service, she advises verifying benchmarks to confirm that quality isn’t being sacrificed. “So when evaluating an inference service, it’s important to look not just at cost and latency, but also at model quality,” she said. “Does this model, provided on this service, also perform similarly on standard benchmarks?”
The financial principles Huyen outlined last year continue to shape the industry today. Most users cannot implement every possible optimization, but understanding the tradeoffs, whether through quantization, caching, or batching, helps control expenses. For those running local large language models, techniques like quantization are accessible. For others, the decisions reduce to balancing speed, accuracy, and budget constraints.
Huyen’s return to P99 CONF on October 21, 22 will likely revisit these discussions, especially as agentic systems drive inference costs to unprecedented levels. Her 2025 insights provide a clear framework for addressing these ongoing challenges.
