Private Multimodal LLM API Deployed on DGX Spark

GenAI Protos deployed Gemma-4-26B-A4B-it as a private LLM service on NVIDIA DGX Spark to support chat, reasoning, document understanding, and vision-enabled workflows through a governed API layer.

The model was served with vLLM and FP8 runtime quantization. GPU allocation was right-sized so it could coexist with other model workloads instead of monopolizing the DGX Spark environment.

The outcome is a private, API-ready LLM capability that supports standard chat access, model discovery, gateway control, and practical operations for teams building enterprise AI applications.

Use case

Private chat and multimodal LLM API for enterprise AI workflows.

Deployment style

vLLM-based model serving on NVIDIA DGX Spark with FP8 runtime quantization.

Core model capability

Gemma-4 Mixture-of-Experts model with 26B total parameters and 4B active parameters.

Primary interface

OpenAI-compatible chat completions through the GenAI Protos gateway.

Multimodal support

Text and image input support documented in the model reference.

Operational profile

GPU memory allocation right-sized for coexistence with other private models.

Technical outcome

Deployed as part of the GenAI Protos DGX Spark model catalog and private gateway pattern.

Enterprise AI teams often need a private LLM that can support chat, reasoning, documentation, and image-aware workflows without moving sensitive prompts or assets into a public model endpoint.

The challenge was to deploy a capable LLM in a way that was fast enough for interactive use, controlled enough for shared infrastructure, and familiar enough for application teams to integrate.

• Expose the LLM through a standard chat API instead of a local-only model runtime.
• Control GPU memory usage so the model can coexist with other DGX Spark services.
• Support both text and vision-style input patterns for multimodal workflows.
• Keep the model discoverable inside the GenAI Protos private model catalog.
• Document sampling, rate limits, serving behavior, and operational constraints.

GenAI Protos packaged Gemma-4 as a private LLM service with a standard gateway interface and GPU-aware deployment settings.

• vLLM deployment for Gemma-4-26B-A4B-it with FP8 runtime quantization.
• OpenAI-compatible chat access through the shared LiteLLM gateway.
• Right-sized GPU utilization for coexistence with other local models.
• Warm-up behavior added so the first real request is not slowed by initialization work.
• Catalog registration with rate limits, token limits, and model capability notes.

The architecture separates the application interface, gateway layer, LLM serving layer, and operations model.

Application or user

Submits chat, reasoning, document, or image-aware prompts through a standard API call.

Gateway layer

Provides governed access, discovery, rate limits, and routing to the private LLM backend.

LLM serving layer

Runs Gemma-4 through vLLM with FP8 runtime quantization.

Model runtime

Uses a Mixture-of-Experts architecture with a smaller active parameter footprint per request.

Operations layer

Handles warm-up, memory sizing, lifecycle control, and coexistence with other models.

The prompt-to-response workflow shows how chat, reasoning, document, or image-aware prompts move through the private gateway, Gemma runtime, and response layer.

This section summarises the vLLM serving, performance tuning, warm-up behaviour, gateway integration, and documentation choices used to make Gemma-4 practical as a private LLM service.

Model serving

Gemma-4 was served through vLLM as a private chat-completion backend.

Performance tuning

GPU memory utilization was reduced from an over-allocated setting to a coexistence-friendly default.

Cold-start handling

Warm-up behavior was added to improve first-request readiness.

Gateway integration

The model was registered behind the shared gateway for OpenAI-compatible access.

Documentation

Serving details, cost fields, rate limits, sampling guidance, and use cases were documented for operators and developers.

The technical details below record model architecture, weights, memory profile, performance observations, context settings, rate limits, and catalog context.

The value of this build is not only running a private LLM. The stronger pattern is turning a capable chat and vision model into a shared, governed service that can support real enterprise workflows without becoming a local-only model runtime.

The implementation turned a private LLM deployment into a shared API service that can support real product workflows instead of remaining a local model run.

This pattern applies to private LLM deployments that need to move from model weights into a usable enterprise service.

• Serve the LLM behind a standard API layer.
• Right-size memory and context settings based on shared infrastructure constraints.
• Add warm-up behavior for better user-facing readiness.
• Expose the model through a gateway for access control and catalog discovery.
• Document rate limits, sampling settings, model behavior, and operational tradeoffs.