Executive Summary
GenAI Protos built a private AI video generation service to show how a heavy generative media model can move from a specialist runtime into an API-ready system that application teams can call and operators can manage.
The service supports both text-to-video and image-to-video workflows. It runs as a custom FastAPI service on NVIDIA DGX Spark, is registered in the GenAI Protos model catalog, and is exposed through gateway-managed routes for controlled access.
The practical outcome is a reusable private AI media service pattern: governed entry point, async job handling, MP4 output, model warm-up controls, idle unload behavior, and operational visibility around GPU usage and service health.
At a Glance
- Use case
- Private text-to-video and image-to-video API for generative video prototyping and application integration.
- Deployment style
- On-demand model serving on NVIDIA DGX Spark with a gateway-managed API surface.
- Core model capability
- Text prompts or input images converted into short MP4 video outputs.
- Primary interface
- Custom FastAPI service exposed through LiteLLM pass-through routes.
- Output modes
- Synchronous MP4 generation and asynchronous job submission with polling and asset download.
- Operational controls
- Health endpoint, preload, unload, on-demand loading, 20-minute idle auto-unload, and request rate limiting.
- Technical outcome
- A private generative video service pattern for models that require custom runtime behavior rather than a standard OpenAI chat interface.
The Challenge
Generative video models are more demanding than standard text or image workloads. They often need larger memory windows, longer inference times, async job handling, and clear lifecycle controls so GPU resources are not tied up unnecessarily.
The challenge was to turn a custom video-generation runtime into a usable private service without overstating its maturity or forcing it into the wrong API pattern. The model needed a stable interface for applications and a clear operating model for platform teams.
- Expose text-to-video and image-to-video generation through a controlled API layer.
- Support long-running generation without making every product flow depend on a blocking request.
- Keep the model discoverable through the GenAI Protos gateway and model catalog.
- Control GPU memory with preload, unload, and idle auto-unload behavior.
- Document request limits, endpoint shape, runtime behavior, and known operational characteristics.
What GenAI Protos Built
GenAI Protos packaged the Sulphur-2 video runtime behind a private FastAPI service and registered two model entries for text-to-video and image-to-video generation. Because the runtime uses custom video endpoints, the gateway exposes it through authenticated pass-through routes rather than treating it as a native chat model.
- A custom FastAPI service for text-to-video and image-to-video requests.
- Two catalog entries: one for text-to-video and one for image-to-video mode.
- Gateway routes for video generation, async job submission, job polling, model listing, preload, unload, and health checks.
- Synchronous generation for direct MP4 output when a blocking request is acceptable.
- Asynchronous job flow for longer video requests, including submit, poll, and download behavior.
- On-demand model loading with an idle auto-unload window to release GPU memory when not in use.
- Rate-limited access suitable for controlled experimentation and early product integration.
Solution Architecture
The architecture separates the application interface, gateway, video service, model runtime, output handling, and operator controls. This keeps the product-facing API predictable while allowing the backend to use a custom video-generation pipeline.
- Application or developer
- Submits prompt, negative prompt, image input when needed, resolution, frame count, FPS, inference steps, guidance scale, and seed settings.
- LiteLLM gateway
- Provides unified access control and pass-through routing for the private model fleet.
- Video API service
- FastAPI service receives text-to-video, image-to-video, and async job requests.
- Model runtime
- Sulphur-2 video pipeline uses the FP8Mixed model and Gemma 3 27B text encoder for prompt embeddings.
- Output layer
- Returns MP4 directly for sync requests or exposes job status and video asset download for async jobs.
- Operations lane
- Preload, unload, health, model listing, and idle auto-unload support day-to-day operation.
Prompt-to-Output Workflow
A user or application submits a prompt, optional negative prompt, resolution, frame count, FPS, inference steps, guidance scale, and seed.
The gateway authenticates the request and forwards it to the video service through a custom route.
The service loads or warms the video pipeline when needed.
The model generates a short video sequence and returns an MP4 file for synchronous requests.
For async requests, the service returns a job ID first, then the application polls status and downloads the finished MP4 asset.
Implementation Highlights
- Model serving
- A Python FastAPI service was used to wrap the video generation runtime behind API endpoints.
- Gateway integration
- The service was registered in the GenAI Protos catalog and exposed through custom pass-through routes because the endpoint is not a native OpenAI chat completion interface.
- Async job design
- Longer video jobs can be submitted to a job endpoint, polled for status, and downloaded when complete.
- Resource control
- The model runs on demand and can unload automatically after 20 minutes of inactivity to release memory.
- Operator controls
- Preload and unload endpoints let operators warm the pipeline before planned use or free GPU memory after testing.
- Output handling
- The service returns MP4 output directly for synchronous generation and provides job-based retrieval for asynchronous generation.
- Catalog consistency
- Text-to-video and image-to-video modes were documented alongside the broader DGX Spark model fleet.
Measured Technical Details
Why This Matters
The value of this use case is not only video generation. The more important pattern is the conversion of a heavy, custom generative media runtime into a private service that application teams can use and platform teams can operate.
Results
GenAI Protos delivered a working private video generation service that can be discovered, called, warmed, unloaded, monitored, and integrated into application workflows without exposing the raw model runtime directly.
| Outcome | What changed |
|---|---|
| Private API access | The video model became an API-callable service for text-to-video and image-to-video generation. |
| Developer-ready workflow | Applications can use either direct MP4 generation or async job submission with polling and download. |
| Operational readiness | Preload, unload, health, job status, and idle auto-unload behavior were documented for operators. |
| Gateway consistency | The service follows the same governed gateway and catalog pattern used across the GenAI Protos private model fleet. |
| Resource-aware serving | The model can be warmed when needed and released after inactivity instead of occupying GPU memory indefinitely. |
Reusable Pattern
This implementation creates a repeatable pattern for private generative media systems where the model is useful but the raw repository is not enough for product integration. The same structure can support video, image, 3D, speech, voice, and segmentation services with different runtime requirements.
- Model wrapper: package the model behind a stable service layer.
- Gateway route: expose the service through a governed access layer.
- Job design: use async workflows where generation time exceeds normal web request expectations.
- Lifecycle design: preload for planned use, unload after idle periods, and monitor service health.
- Documentation: record endpoint shapes, request limits, lifecycle controls, and operational behavior.
Build a Private Generative Media Prototype You Can Operate
GenAI Protos helps teams turn promising video, image, audio, and multimodal AI models into private, API-ready systems with model serving, gateway integration, async workflows, health checks, and product integration around the model.
Get custom solutions