For many teams at low volumes, calling a hosted API costs less than self-hosting once hardware and operational overhead are included. For teams at high volumes, self-hosted inference can cost significantly less per request. The crossover point depends on your token volumes, concurrency, model size, hardware choice, and operational costs. The part most teams get wrong is not doing the math - with their actual numbers - before making the architecture decision.
Self-hosted LLM deployment is not inherently better or worse than calling a hosted API. It is better for specific organizations at specific usage volumes with specific data security requirements. This post compares three commonly used tools that serve different points in the LLM deployment spectrum: vLLM (high-throughput multi-GPU serving engine), a single-server developer runtime, and OpenRouter (hosted multi-provider API router). These are not equivalent categories; each addresses a different deployment pattern. The comparison includes illustrative cost numbers, hardware requirements, and the honest failure modes of each.
The TCO Calculation Framework
Before choosing an architecture, calculate the full cost on both sides. The two components are:
There is no universal request-count threshold at which self-hosting becomes cheaper. The crossover depends on the specific model, hardware configuration, GPU utilisation rate, idle-capacity costs, and per-token API pricing at the time. Use the TCO formula above with your actual numbers. The illustrative example below shows the calculation approach for one specific configuration; results will differ materially for other models, hardware, and utilisation levels.
Explore our private deployment service: Custom Private AI & Edge Solutions helps teams compare hosted, dedicated and self-managed inference against security, latency and operating requirements.
Option 1: Ollama
What it is:
Ollama is an open-source runtime that simplifies running LLMs on a single server or workstation. It handles model downloading, quantization, and serving through a simple CLI and REST API that is OpenAI-compatible (models served by it can be called with the OpenAI client by changing the base URL).
Who it is for:
Individual developers, small teams, and organizations that need a self-hosted LLM for development, testing, or low-volume internal tooling. The tool is designed for single-server, low-concurrency use cases rather than high-throughput production workloads.
Hardware requirements:
The runtime runs on a machine with a modern consumer GPU. A single RTX 4090 (24GB VRAM) runs Llama 3.1 8B in full precision or Llama 3.1 70B in 4-bit quantization. Mac M-series chips run smaller models (up to 13B) on unified memory. Minimum viable for internal tooling: any modern GPU with at least 8GB VRAM.
Performance:
Request concurrency is configurable based on available GPU memory and system settings. Without explicit concurrency configuration, requests may queue on a single GPU. For a small team making occasional queries, a single-GPU setup is often acceptable. For sustained concurrent workloads at scale, throughput is not comparable to a dedicated multi-GPU serving engine.
Failure modes:
The runtime does not handle production-level authentication, rate limiting, or monitoring by default. Running the tool on a server accessible on the corporate network without authentication is a security risk. Add a reverse proxy with authentication before exposing it to any user beyond the developer running it.
Option 2: vLLM
What it is:
vLLM is an open-source LLM inference and serving engine optimized for high throughput. It implements PagedAttention, a memory management algorithm that enables efficient serving of multiple concurrent requests on GPU hardware. The engine is the production standard for high-volume self-hosted LLM inference.
Who it is for:
Engineering teams running self-hosted LLM inference at production scale, where throughput and latency SLAs must be met and request volume justifies the infrastructure investment.
Performance:
Benchmark comparisons with other serving frameworks vary significantly by hardware, model, workload, and configuration. Use vLLM's own benchmark tooling with your specific model and request pattern to measure throughput on your target hardware before committing to infrastructure spend.
Failure modes:
The serving engine requires infrastructure engineering to operate: GPU instance management, model versioning, monitoring, and incident response. A team without GPU infrastructure experience will spend 2 to 3 months reaching production-grade reliability. The operational investment is real. Budget for it. The GenAI Protos Private LLM Deployment for Enterprises guide covers the infrastructure stack including monitoring, incident response, and capacity planning.
Option 3: OpenRouter
What it is:
OpenRouter is a hosted API router that provides a single API endpoint to access models from multiple providers: OpenAI, Anthropic, Google, Meta (via hosted endpoints), Mistral, and open-weight models hosted by third-party providers. It normalizes the API format across providers and adds features like automatic fallback, load balancing across providers, and access to models that are otherwise difficult to access directly.
Who it is for:
Teams that want access to multiple model providers through one API, teams that need fallback routing when a provider has an outage, and teams evaluating multiple models before committing to a deployment.
Key use case:
Teams at volumes where API calling makes economic sense who need: access to frontier proprietary models without maintaining separate API keys and rate limit management for each provider, access to open-weight model hosted inference at lower per-token cost than proprietary models, and automatic fallback if a primary provider has an outage.
Failure modes:
The routing service creates a dependency between your application and the model providers. A routing-layer outage affects all your model calls simultaneously. For production systems where uptime is critical, implement a fallback to direct provider API calls. The service does not guarantee latency SLAs in the same way a dedicated self-hosted deployment does.
Side-by-Side Comparison
| Dimension | Ollama | vLLM | OpenRouter |
|---|---|---|---|
| Use case | Dev / low-volume tooling | High-volume production | Multi-provider / evaluation |
| Hardware needed | Consumer GPU | Multi-GPU server | None |
| Setup complexity | Low | High | Very low |
| Concurrency | Configurable; limited by VRAM | High (configurable) | Managed by provider |
| Cost model | Hardware upfront | Hardware upfront | Pay-per-token |
| Data stays on-prem | Yes | Yes | No |
Relevant case study: The Private AI Music Generation API on DGX Spark demonstrates private model serving on dedicated enterprise infrastructure.
When Self-Hosted LLM Is Not the Right Answer
Data security requirements are the most cited reason teams choose self-hosted LLM. Not all data security requirements actually require self-hosting. If your concern is that prompt data is used to train provider models, check the provider's enterprise terms: Anthropic, OpenAI, and Google all offer enterprise agreements that exclude customer data from training. If your concern is data residency (data cannot leave a specific geographic region), most major providers offer regional API endpoints that satisfy residency requirements.
The decision is not binary. Enterprise LLM deployment options span a spectrum: direct provider API, managed dedicated endpoint (provider-hosted but isolated to your account), private cloud endpoint, self-hosted inference on your own GPU infrastructure, and multi-provider routing services. Self-hosting is the right answer when: regulatory requirements specifically prohibit third-party processing of the data, the organization has IP concerns even with contractual protections, or the token volume and utilisation make cloud API costs prohibitive at scale. GenAI Protos' Private AI expertise covers the full architecture options for organizations in each of these scenarios.
Key Takeaways
- The self-host vs API cost comparison depends on token volumes, model size, hardware choice, utilisation and operational overhead. Calculate the TCO for both sides using your actual numbers before committing. For most teams at low-to-moderate volumes, API calling is cheaper once all self-hosting costs are included.
- Ollama is designed for developers and low-volume internal tooling. Its default configuration is not optimized for high-concurrency production workloads; throughput is not comparable to multi-GPU serving engines under sustained concurrent load.
- vLLM is the production standard for high-throughput self-hosted LLM inference. It requires GPU infrastructure engineering to operate reliably.
- A hosted API router provides multi-provider access, automatic fallback, and open-weight model access at lower cost, without hardware investment. Teams evaluating model options can also review the AI benchmarking guide before committing to a deployment architecture.
- Verify whether your data security requirement actually requires self-hosting before committing to the operational overhead.
Conclusion:
The right inference architecture depends on token volume, model quality, concurrency, utilisation, security and operational capacity. GenAI Protos evaluates these variables together and designs hosted, private or self-managed deployments around the organisation's actual workload and control requirements.
