Updated 2026-10-23
How to build DeepInfer and run offline inference
These commands follow DeepSeek's build instructions for Linux. We have checked the documentation but have not run them on the listed accelerators. Save your source revision and use the README from that checkout so you can trace errors to the right version.
What should you prepare before cloning the source?
Check your hardware against the support list before downloading weights or renting a cluster. You need Python 3.12, the required compiler and the accelerator runtime installed before following the commands below.
Leave disk space for the source, dependencies, weights and build files. Write down the device layout, context length and parallel settings you will use for the first run.
How do you prepare a reproducible source checkout?
Clone the official repository with its submodules. This command downloads the main branch as it stands when you run it. Save the commit ID printed by git rev-parse HEAD with your logs, and follow the README in that checkout.
Create a clean Python environment and activate it. Keep that environment active for the build and the inference run.
git clone --recursive https://github.com/deepseek-ai/DeepInfer.git
cd DeepInfer
git rev-parse HEAD
python3.12 -m venv .venv
source .venv/bin/activate
python --versionHow do you select the backend and build DeepInfer?
Set DEEPINFER_ACCELERATOR in the shell you use for both commands. Choose nvidia or ascend to match your machine; you only need one backend.
If the build fails, save the first compiler or dependency error. Check it against the prerequisites in your checkout before retrying. After a successful build, you still need to load the weights and run the model on the device.
# NVIDIA backend; use ascend instead on supported Ascend hardware
export DEEPINFER_ACCELERATOR=nvidia
./build.shWhere does the model come from, and how do you run it?
The example uses deepseek-ai/DeepSeek-V4.1-Flash-1017 by default. It can download the weights into the Hugging Face cache or read a local checkpoint. Run --help and choose parallel settings that fit your hardware before starting.
To use weights you already have, set DEEPINFER_HF_MODEL_DIR to the directory containing the complete checkpoint. This changes the default model path. An explicit --model_name argument takes precedence.
python examples/offline_inference.py --help
# After choosing a layout that fits the supported hardware:
python examples/offline_inference.pyHow should you check the first result?
Save stdout, stderr and the exit status with your configuration and source revision. Check that the intended model loaded, produced useful output and exited cleanly. A finished download or build is only an earlier step.
Next, try the inputs your application needs. Check output quality and any image, tool-call or structured-output features before increasing concurrency. The examples in your checkout describe further validation options.
For throughput tests, record accepted output tokens and elapsed time together. Keep context length, request count, sampling and quality checks the same across runs. These are suggested checks; this article contains no performance measurements from our own hardware.
What changes when you turn the experiment into an API?
Once offline inference works, plan how requests will reach the model. Recipe handles protocol conversion, but its example servers use mock inference. You need a real backend, HTTP serving, tool execution and the remaining cache, routing and load-balancing components.
If you just need to call a hosted DeepSeek model, use our API and SDK guides. You can follow that route without deploying DeepInfer.
FAQ
Does this guide install an OpenAI-compatible API server?
No. It builds DeepInfer and runs an offline example. An API also needs a serving layer and the other infrastructure described above.
Can I run these commands directly in Windows PowerShell?
These commands are for Linux with supported accelerators. The official instructions do not describe a Windows installation.
Does setting DEEPINFER_HF_MODEL_DIR change an explicit --model_name?
No. The official examples say the variable changes the default model path and does not override an explicit model_name argument.
Where should I look after a successful build but a failed first run?
Save the first runtime error. Check the selected accelerator, Python environment, checkpoint directory, available memory and parallel settings before changing the configuration.
Save the working configuration with its source revision and output. Use that run as your baseline for load tests or an online service.