What DeepInfer Means for the LLM Inference Ecosystem (Oct 2026)

On this page

1. What did DeepSeek actually open-source?
2. What production numbers does DeepSeek report?
3. Why the savings are structural, not kernel tuning
4. What does it mean for NVIDIA and Huawei?
5. What does it mean for Cambricon, Hygon, Moore Threads, and the rest?
6. How does DeepInfer shift chip design priorities?
7. Does DeepInfer replace vLLM or SGLang?
8. What should teams take away?
9. Frequently asked questions
10. Sources

What did DeepSeek actually open-source?

DeepInfer grew out of work based on vLLM. The repository credits vLLM as its main starting point and includes a C++ engine for scheduling and input preparation. What shipped on October 20 is deliberately narrow: one model, two hardware targets, and offline inference only.

A fair description is that DeepInfer is the single engine instance inside DeepSeek's production system. The pieces that make large-scale online serving hard stayed home, including global KV cache management, prefill and decode disaggregation, cross-cluster load balancing, and fault handling. Cloud teams get a worked reference on day one, which is exactly what they wanted.

What production numbers does DeepSeek report?

DeepSeek’s deployment Q&A reports the following figures for its production system. These describe the internal deployment, not a throughput promise for the offline repository:

• DeepSeek reports 155 trillion tokens per day for one cluster and US$38 in revenue per GPU-hour at valley pricing. These are separate reported metrics; throughput alone does not determine revenue.
• Decode runs expert parallelism at EP64 to EP128. Prefill runs at EP2 to EP8.
• Paged block KV cache is stored in 3FS with 10+2 erasure coding: 3 days for agent traffic and 7 days for chatbot traffic. SWA state is kept separately in host memory.

Why the savings are structural, not kernel tuning

Several savings depend on the model architecture. DeepSeek’s cross-layer KV reuse and decoder SWA bounded replay reduce stored state and repeated computation. In the deployment Q&A, bounded replay is described as nearly halving total prefill computation.

That behavior cannot be bolted onto an existing checkpoint. The model has to be designed for it before training, and the engine's job is to collect the full savings. Kernel optimization still matters, but these architectural choices cannot be reproduced by kernel tuning alone. This does not establish a cost advantage over every competing deployment.

What does it mean for NVIDIA and Huawei?

The hardware abstraction has two paths, and only two. NVIDIA goes through CUDA and NCCL. Ascend goes through CANN and HCCL. Both are first-class citizens in the same engine.

• Platform-specific TileLang kernels alongside shared model and engine logic.
• Platform-specific matrix-multiplication and expert-communication implementations.

Supporting NVIDIA and Ascend in one repository gives both platforms a common reference for the model and engine. It does not prove feature or performance parity: the README, for example, currently lists batch-invariant inference and deterministic sampling as NVIDIA-only features.

The editorial implication is that an upstream Ascend path can reduce duplicated integration work and make model support easier to assess. Ascend 950 is the documented NPU target; broader adoption and the economics of upgrading from older hardware still depend on workload-specific tests.

What does it mean for Cambricon, Hygon, Moore Threads, and the rest?

The picture is less friendly for everyone else. Cambricon, Hygon, Kunlunxin, Moore Threads, MetaX, Biren, Enflame, and Iluvatar Core all sit outside the two supported paths. A native DeepInfer port would require platform engineering across the following areas. Vendors could also adapt these ideas to existing engines rather than abandon them:

• A TileLang backend and the required platform kernels, or equivalent implementations.
• FP8-by-FP4 GEMM, plus a mega MoE kernel that fuses expert dispatch, compute, and combine into one launch.
• Sparse attention and top-k selection operators.
• An expert-parallelism communication library that lets devices initiate transfers, plus the cross-card table lookups that Engram, the hashed n-gram memory component, requires.
• The whole platform layer underneath.

For every vendor, TileLang backend maturity becomes the gate. A mature backend makes this class of engine much cheaper to port. Without one, the bill is enormous.

How does DeepInfer shift chip design priorities?

The engine also reads as a wish list for future inference silicon. Three shifts stand out in the source analysis.

• Precision support has to match the actual tensors. The deployment Q&A lists FP4 for routed-expert weights and the main and indexer block KV caches; SWA state uses MXFP8. UE8M0 appears in the indexer and routed-expert quantization scales. Hardware evaluation should cover these formats and any conversion overhead.
• The memory pressure inverts. A smaller KV footprint can reduce long-context capacity pressure. But decode at EP64 and above, together with the mega MoE kernel, demands large high-speed interconnect domains and device-initiated communication. Interconnect can become a more important bottleneck; the balance still depends on the workload and deployment.
• Host bandwidth deserves attention when Engram tables are placed in CPU memory. Engram fetches table entries by n-gram hash, and DeepEP documents CPU/GPU storage with RDMA fetches. Host placement can therefore make transfer bandwidth and latency part of the performance budget.

Does DeepInfer replace vLLM or SGLang?

It is not a drop-in replacement for a complete online serving framework. The public release supports one model and offline inference; shared prefix-cache storage and the surrounding online service need additional infrastructure. Its value is as a reference: how far an engine can shrink when one team designs the model and the engine together, and which state is worth storing versus recomputing. How the general frameworks absorb these design choices is the thing to watch through the rest of 2026. Inference engineers do not need to update their resumes yet.

What should teams take away?

For model teams, this is evidence that architectural decisions made during training can materially affect inference cost. DeepSeek treats KV size, prefill compute, and speculative decoding acceptance rate as training objectives, then ships an engine to collect the savings. The practical lesson is to evaluate architecture and serving together.

For chip teams, early model collaboration, kernel tooling, low-precision formats, and communication support all matter. A published reference makes the integration work more visible, without establishing which vendor will win deployments.

For anyone buying inference, the reference offers a concrete basis for evaluating a provider. Compare measured latency, accepted output throughput, quality, and the full service cost; open source alone does not guarantee lower customer prices.

Frequently asked questions

What is DeepInfer?

DeepInfer is the inference engine DeepSeek open-sourced on October 20, 2026. The README credits vLLM as its main starting point, and the source includes a C++ engine. It ships as an offline inference engine for one model.

Which model and which hardware does DeepInfer support?

One model: DeepSeek-V4.1-Flash. Two hardware targets: NVIDIA SM100 (Blackwell) through CUDA and NCCL, and Huawei Ascend 950 through CANN and HCCL.

Does DeepInfer include online serving?

No. Online serving, PD disaggregation, cross-cluster load balancing, fault handling for online traffic, and the KV cache storage system are not in the release.

How many tokens does DeepSeek serve per day?

DeepSeek reports 155 trillion tokens per day for one cluster and US$38 in revenue per GPU-hour at valley pricing. The second figure cannot be derived from token throughput alone.

Will DeepInfer replace vLLM or SGLang?

It is not a drop-in online serving replacement. The release supports one model and offline inference; a complete service still needs cache storage, disaggregated deployment, and load balancing.

Sources

Checked October 23, 2026. Diagrams are editorial summaries, not benchmark results.

1. DeepSeek · DeepInfer README — Offline release scope, supported model and hardware, missing service infrastructure, and vLLM origins.