Check current compatibility: vLLM, PyTorch, Python and CUDA requirements change between versions. Always check the official vLLM documentation for your chosen release and model.
1. Validate your NVIDIA installation
Before installing any inference framework, confirm the host can communicate with the GPU:
nvidia-smi
uname -r
python3 --version
command -v nvcc && nvcc --versionThe CUDA API version printed by nvidia-smi reflects driver capability. It does not prove a CUDA compiler is installed, and prebuilt wheels often do not require one.
2. Isolate Python dependencies
Use a Python interpreter supported by your selected vLLM release. A virtual environment keeps experiments separate from system Python:
python3 -m venv vllm-env
source vllm-env/bin/activate
python -m pip install --upgrade pipFollow the official installation instructions and start with supported binary packages. A source build may demand substantially more host RAM and compilation time.
3. Understand the memory budget
VRAM consumption is more than the size of the weights. Include the KV cache, temporary tensors and runtime overhead. Four-bit weights alone do not promise that a large model will fit into a particular GPU.
Inspect available command-line flags for your installed release before following examples found online:
vllm serve --help
vllm collect-envA local server is generally launched with vllm serve MODEL_ID, with a compatible repository identifier substituted for the placeholder. Respect any model licence or gated-access requirement.
4. Test the local endpoint
After startup completes, query the OpenAI-compatible model-list endpoint:
curl http://127.0.0.1:8000/v1/modelsKeep the service on localhost unless proper authentication and network access controls are in place. Do not expose an unauthenticated inference API to the public internet.
5. Diagnose memory and runtime failures
nvidia-smi -l 1
free -h
journalctl -k -b --no-pager | tail -n 100If CUDA reports out-of-memory, reduce context length and concurrency first; check quantisation support and any competing GPU processes. Host RAM is a separate budget — just-in-time compilation and model loading can stress it heavily.
- GPU not visible: investigate the driver and kernel first.
- Unsupported kernel: verify vLLM release, GPU architecture and quantisation format.
- Import errors: suspect incompatible packages in the Python environment.
- Connection refused: check whether the server finished startup and is listening on the expected port.
Privacy: diagnostics may include usernames, file paths, hostnames or environment variables. Inspect and sanitize logs before sharing.
Want to understand the underlying technology? Read how large language models work.