Inference Server
- 6 installs
- Updated March 19, 2026
- huggingface/prime-rl
inference-server skill documents Start and test the prime-rl inference server.
About
inference-server skill documents Start and test the prime-rl inference server. Use when asked to run inference, start vLLM, test a model, or launch the inference server.. name: inference-server description: Start and test the prime-rl inference server. Use when asked to run inference, start vLLM, test a model, or launch the inference server.
- Start and test the prime-rl inference server.
- Platform-specific setup patterns for inference-server.
- Evidence-backed steps from upstream SKILL.md.
- When-to-use criteria for inference-server versus alternatives.
Inference Server by the numbers
- 6 all-time installs (skills.sh)
- Ranked #1,588 of 2,064 Data Science & ML skills by installs in the Skillselion catalog
- Data as of Aug 2, 2026 (Skillselion catalog sync)
inference-server capabilities & compatibility
- Capabilities
- inference server quick start · inference server when to use guidance · inference server integration patterns
- Use cases
- research
What inference-server says it does
uv run inference @ path/to/config.toml
uv run inference --model.name Qwen/Qwen3-0.6B --model.max_model_len 2048 --model.enforce_eager
npx skills add https://github.com/huggingface/prime-rl --skill inference-serverAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 6 |
|---|---|
| Last updated | March 19, 2026 |
| Repository | huggingface/prime-rl ↗ |
How do I use inference-server correctly?
Start and test the prime-rl inference server. Use when asked to run inference, start vLLM, test a model, or launch the inference server.
Who is it for?
Teams implementing inference-server workflows from the catalog.
Skip if: Skip when requirements clearly match a different specialized stack.
When should I use this skill?
User asks about inference-server, start and test the prime-rl inference server. use when asked to run inference, start vllm,.
What you get
Working inference-server setup with validated configuration and next steps.
Files
Inference Server
Starting the server
Always use the inference entry point — never vllm serve or python -m vllm.entrypoints.openai.api_server directly. The entry point runs setup_vllm_env() which configures environment variables (LoRA, multiprocessing) before vLLM is imported.
# With a TOML config
uv run inference @ path/to/config.toml
# With CLI overrides
uv run inference --model.name Qwen/Qwen3-0.6B --model.max_model_len 2048 --model.enforce_eager
# Combined
uv run inference @ path/to/config.toml --server.port 8001 --gpu-memory-utilization 0.5SLURM scheduling
The inference entrypoint supports optional SLURM scheduling, following the same patterns as SFT and RL.
Single-node SLURM
# inference_slurm.toml
output_dir = "/shared/outputs/my-inference"
[model]
name = "Qwen/Qwen3-8B"
[parallel]
tp = 8
[slurm]
job_name = "my-inference"
partition = "cluster"uv run inference @ inference_slurm.tomlMulti-node SLURM (independent vLLM replicas)
Each node runs an independent vLLM instance. No cross-node parallelism — TP and DP must fit within a single node's GPUs.
# inference_multinode.toml
output_dir = "/shared/outputs/my-inference"
[model]
name = "PrimeIntellect/INTELLECT-3-RL-600"
[parallel]
tp = 8
dp = 1
[deployment]
type = "multi_node"
num_nodes = 4
gpus_per_node = 8
[slurm]
job_name = "my-inference"
partition = "cluster"Dry run
Add dry_run = true to generate the sbatch script without submitting:
uv run inference @ config.toml --dry-run trueCustom endpoints
The server extends vLLM with:
/v1/chat/completions/tokens— accepts token IDs as prompt input (used by multi-turn RL rollouts)/update_weights— hot-reload model weights from the trainer/load_lora_adapter— load LoRA adapters at runtime/init_broadcaster— initialize weight broadcast for distributed training
Testing the server
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-0.6B",
"messages": [{"role": "user", "content": "Hi"}],
"max_tokens": 50
}'Key files
src/prime_rl/entrypoints/inference.py— entrypoint with local/SLURM routingsrc/prime_rl/inference/server.py— vLLM env setupsrc/prime_rl/configs/inference.py—InferenceConfigand all sub-configssrc/prime_rl/inference/vllm/server.py— FastAPI routes and vLLM monkey-patchessrc/prime_rl/templates/inference.sbatch.j2— SLURM template (handles both single and multi-node)configs/debug/infer.toml— minimal debug config
Related skills
Microsoft FoundryDeploy, evaluate, and continuously improve Microsoft Foundry agents from a single agent interface.490k1.3k
Ai Research ReproductionOrchestrate trustworthy, auditable reproduction of deep learning repositories directly from their READMEs.176k511Run TrainSafely execute selected deep learning training commands with standardized evidence capture.176k511Explore RunSafely run isolated exploratory experiments with clear recording and conservative selection before committing changes.176k511Paper Context ResolverFetch precise reproduction-critical details like dataset splits, preprocessing steps, or evaluation protocols from the original academic paper when the repo README leav141k511Repo Intake And PlanScan unfamiliar AI research repositories and receive a minimal, trustworthy reproduction target before investing significant time.140k511FAQ
What does inference-server do?
inference-server skill documents Start and test the prime-rl inference server.
When should I use inference-server?
User asks about inference-server, start and test the prime-rl inference server. use when asked to run inference, start vllm,.
Is this skill safe to install?
Review the Security Audits panel on this page before installing in production.