Skip to content

You are viewing the latest developer preview docs. Click here to view docs for the latest stable release.

vLLM

suffix

Initializing search

Home
User Guide
Developer Guide
Benchmarking
API Reference
CLI Reference
Community

vLLM

Home
User Guide
User Guide
- Getting Started
  Getting Started
  - Quickstart
  - Installation
    
    Installation
    
    GPU
    
    CPU
    
    TPU
  - Examples
    
    Examples
    
    Applications
    Applications
    
    API Server
    
    Chatbot
    
    Rag
    
    Basic
    Basic
    
    Offline Inference
    
    Online Serving
    
    Deployment
    Deployment
    
    Async LLM Streaming
    
    Helm Charts
    
    LLM Engine Example
    
    Sagemaker-Entrypoint
    
    Disaggregated
    Disaggregated
    
    Disaggregated Encoder
    
    Disaggregated Serving
    
    Ec Both Encoder
    
    Disaggregated Prefill V1
    
    Flexkv Connector
    
    KV Load Failure Recovery Test
    
    LMCache Examples
    
    Mooncake Connector
    
    Mooncake Store
    
    Features
    Features
    
    Automatic Prefix Caching
    
    Context Extension
    
    Data Parallel
    
    Kv Events
    
    Logging Configuration
    
    Custom Logits Processors
    
    LoRA
    
    Offline Inference with the OpenAI Batch file format
    
    Profiling
    
    Prompt Embed
    
    Reset Kv
    
    Sharded State
    
    Speculative Decoding
    
    Structured reads on DiffusionGemma
    
    Structured Outputs
    
    Tensorize vLLM Model
    
    Torchrun
    
    Generate
    Generate
    
    Batched Chat Completions Online
    
    Multimodal
    
    Qwen 1M Offline
    
    Trace Replay Offline
    
    Observability
    Observability
    
    Monitoring Dashboards
    
    Metrics
    
    Setup OpenTelemetry POC
    
    Prometheus and Grafana
    
    Pooling
    Pooling
    
    Classify
    
    Embed
    
    Plugin
    
    Reward
    
    Score
    
    Token Classify
    
    Token Embed
    
    Ray Serving
    Ray Serving
    
    Batch LLM Inference
    
    Elastic Ep
    
    Multi-Node-Serving
    
    Ray Serve Deepseek
    
    Run Cluster
    
    Reasoning
    Reasoning
    
    OpenAI Chat Completion Tool Calls With Reasoning
    
    OpenAI Chat Completion With Reasoning
    
    OpenAI Chat Completion With Reasoning Streaming
    
    OpenAI Responses Client
    
    RL
    RL
    
    Batch Invariance
    
    Pause Resume
    
    Rdt vLLM Serve
    
    Rdt Weight Source
    
    RLHF Async New APIs
    
    RLHF Http IPC
    
    RLHF Http IPC No vLLM
    
    RLHF Http NCCL
    
    RLHF Http NCCL No vLLM
    
    RLHF IPC Fsdp Ep
    
    RLHF M2N
    
    RLHF NCCL Fsdp Ep
    
    RLHF NCCL Multi Source
    
    RLHF Sharded Rdt Small Ep
    
    RLHF Sparse NCCL
    
    Routed Experts E2E
    
    Skip Loading Weights In Engine Init
    
    Scale Out
    Scale Out
    
    Init
    
    Example Mm Serve
    
    Token Generation Client
    
    Speech To Text
    Speech To Text
    
    OpenAI
    
    Realtime
    
    Tool Calling
    Tool Calling
    
    Chat With Tools Offline
    
    OpenAI Chat Completion Client With Tools
    
    OpenAI Chat Completion Client With Tools Required
    
    OpenAI Chat Completion Client With Tools Xlam
    
    OpenAI Chat Completion Client With Tools Xlam Streaming
    
    OpenAI Responses Client With Mcp Tools
    
    OpenAI Responses Client With Tools
- General
  General
- Inference and Serving
  Inference and Serving
  - Offline Inference
  - Online Serving
    
    Online Serving
    
    Derenderer APIs
    
    Generative Scoring
    
    OpenAI-Compatible Server
    
    Renderer APIs
    
    Speech to Text APIs
    
    Tokens In <> Tokens Out API
    
    Trace Replay
  - Context Parallel Deployment
  - Data Parallel Deployment
  - Troubleshooting distributed deployments
  - Expert Parallel Deployment
  - Parallelism and Scaling
  - Integrations
    Integrations
    
    Claude Code
    
    Codex
    
    LangChain
    
    LlamaIndex
- Deployment
  Deployment
  - Using Docker
  - Using Kubernetes
  - Using Nginx
  - Frameworks
    Frameworks
    
    Anyscale
    
    AnythingLLM
    
    AutoGen
    
    BentoML
    
    Cerebrium
    
    Chatbox
    
    Crusoe
    
    Dify
    
    dstack
    
    Haystack
    
    Helm
    
    Hugging Face Inference Endpoints
    
    LiteLLM
    
    Lobe Chat
    
    LWS
    
    Modal
    
    Nebius Serverless AI
    
    Open WebUI
    
    Retrieval-Augmented Generation
    
    RunPod
    
    SkyPilot
    
    Streamlit
    
    NVIDIA Triton
  - Integrations
    Integrations
    
    AIBrix
    
    NVIDIA Dynamo
    
    k8s-aibom
    
    KAITO
    
    KServe
    
    Kthena
    
    KubeAI
    
    KubeRay
    
    Llama Stack
    
    llm-d
    
    llmaz
    
    Production stack
- Training
  Training
- Configuration
  Configuration
- Models
  Models
  - Supported Models
  - Generative Models
  - Pooling Models
    
    Pooling Models
    
    Classification Usages
    
    Embedding Usages
    
    Reward Usages
    
    Scoring Usages
    
    Specific Model Examples
    
    Token Classification Usages
    
    Token Embedding Usages
  - Structured Decisions Models
  - Extensions
    Extensions
    
    Loading model weights with fastsafetensors
    
    Loading Model Weights with InstantTensor
    
    Loading models with Run:ai Model Streamer
    
    Loading models with CoreWeave's Tensorizer
  - Hardware Supported Models
    Hardware Supported Models
    
    CPU - Intel® Xeon®
    
    XPU - Intel® GPUs
    
    TPU
- Features
  Features
Developer Guide
Developer Guide
Benchmarking
Benchmarking
API Reference
API Reference
- vllm
  vllm
  - collect_env
  - connections
  - env_override
  - envs
  - exceptions
  - forward_context
  - logger
  - logits_process
  - logprobs
  - model_inspection
  - outputs
  - pooling_params
  - sampling_params
  - scalar_type
  - sequence
  - tasks
  - version
  - assets
  - benchmarks
  - compilation
  - config
  - cute_utils
  - device_allocator
  - distributed
  - engine
  - entrypoints
  - inputs
  - ir
  - kernels
  - logging_utils
  - lora
  - model_executor
  - models
  - multimodal
  - parser
  - platforms
  - plugins
  - profiler
  - ray
  - reasoning
  - renderers
  - snapshot
  - tilelang_utils
  - tokenizers
  - tool_parsers
  - tracing
  - transformers_utils
  - triton_utils
  - usage
  - utils
  - v1
    
    v1
    
    cudagraph_dispatcher
    
    kv_cache_interface
    
    kv_cache_layout
    
    kv_cache_spec_registry
    
    outputs
    
    request
    
    serial_utils
    
    utils
    
    attention
    
    core
    
    engine
    
    executor
    
    fault_tolerance
    
    hisparse
    
    kv_hints
    
    kv_offload
    
    metrics
    
    pool
    
    sample
    
    simple_kv_offload
    
    spec_decode
    
    structured_output
    
    watermarking
    
    worker
    
    worker
    
    block_table
    
    cp_utils
    
    cpu_model_runner
    
    cpu_worker
    
    dp_utils
    
    ec_connector_model_runner_mixin
    
    encoder_cudagraph
    
    encoder_cudagraph_defs
    
    extensible_kv_cache
    
    gpu_input_batch
    
    gpu_model_runner
    
    gpu_ubatch_wrapper
    
    gpu_worker
    
    kv_connector_model_runner_mixin
    
    lora_model_runner_mixin
    
    mamba_utils
    
    mm_encoder_model_runner
    
    startup_plan
    
    ubatch_utils
    
    ubatching
    
    utils
    
    worker_base
    
    workspace
    
    xpu_model_runner
    
    xpu_worker
    
    cpu
    
    gpu
    
    gpu
    
    async_utils
    
    attn_utils
    
    block_table
    
    buffer_utils
    
    cp_utils
    
    cudagraph_utils
    
    dp_utils
    
    ec_connector
    
    eplb_utils
    
    input_batch
    
    kv_connector
    
    lora_utils
    
    model_runner
    
    pcp_manager
    
    pp_utils
    
    shutdown
    
    states
    
    structured_outputs
    
    ubatch_utils
    
    warmup
    
    metrics
    
    mm
    
    model_states
    
    pool
    
    sample
    
    spec_decode
    
    spec_decode
    
    acceptance_estimator
    
    adaptive_verification
    
    draft_logits
    
    extract_hidden_states
    
    extract_hidden_states_config
    
    rejection_sampler
    
    rejection_sampler_utils
    
    speculator
    
    speculator_config
    
    utils
    
    v1_speculator_configs
    
    dflash
    
    dflash2
    
    dspark
    
    eagle
    
    gemma4
    
    lilicorr
    
    mtp
    
    multi_module_mtp
    
    ngram
    
    standalone_ar
    
    suffix
    
    suffix
    
    config
    
    speculator
    
    target_dependent_ar
    
    xpress
    
    sentinel
CLI Reference
CLI Reference
- vllm
  vllm
  - chat
  - complete
  - download-kernels
  - preload
  - run-batch
  - serve
  - bench
    
    bench
    
    forward
    
    latency
    
    mm-processor
    
    serve
    
    startup
    
    throughput
    
    sweep
    
    sweep
    
    plot
    
    plot_pareto
    
    serve
    
    serve_workload
    
    startup
  - launch
    
    launch
    
    render
Community
Community
- Contact Us
- Meetups
- Reviewers
- Sponsors
- Governance
  Governance
- Blog
- Forum
- Slack

Home
API Reference
vllm
v1
worker
gpu
spec_decode
suffix

`vllm.v1.worker.gpu.spec_decode.suffix` ¶

Modules:

speculator –

GPU suffix decoding speculator for the V2 model runner.

November 3, 2026

Made with Material for MkDocs