Your legal team vetoed cloud AI. Your security team wants data residency. Your engineering team has one GPU server and a deadline. Now someone in the meeting asks: "Should we deploy a small language model or a large one?" The room goes quiet.
Most enterprises are running that meeting without a framework to answer it. This guide gives you that framework.
SLM vs LLM is not a capability debate. It is a deployment architecture decision. Get it wrong and you either overspend on compute you do not need or deploy a model too weak for the task.
Defining the Categories: No Universal Threshold Exists
"Small language model" is an informal term for a compact model designed to provide useful task performance with lower memory, compute and deployment requirements. There is no universally accepted parameter threshold separating an SLM from an LLM - Meta's official Llama 3.2 model card describes its 1B and 3B models as large language models.
For deployment planning, these operational bands are more useful than a single cutoff:
These ranges are practical illustrations, not formal industry definitions. A 14B model running with Q4 quantisation may fit the same hardware envelope as a 7B model at full precision. Model size is one input to the deployment decision - not the deciding factor on its own.
Why Local LLM Deployment Is the Forcing Function
The enterprise shift toward a local LLM is often regulatory - but the regulatory picture is more nuanced than a simple "cloud is prohibited" reading.
GDPR
GDPR Article 44 regulates transfers of personal data to third countries or international organisations. It does not prohibit cloud processing. Transfers may occur when Chapter V conditions are satisfied. Organisations must evaluate processing location, transfer mechanisms, vendor terms and technical safeguards - not assume cloud AI is automatically prohibited.
HIPAA
HHS explicitly permits covered entities and business associates to process ePHI through public, private or hybrid cloud services when they execute a business associate agreement, perform risk analysis and implement required administrative, physical and technical safeguards. On-premises is one option, not a HIPAA mandate.
FedRAMP
FedRAMP is an assessment and authorisation framework for cloud services used by federal agencies. Whether a use case falls within its scope depends on the specific agency and service use case. It is not a general requirement to deploy AI on-premises.
Where local deployment earns its keep: when the organisation requires a tightly controlled processing boundary, offline operation, specific residency guarantees, or predictable inference economics at high call volume.
Benchmark note: performance figures depend on model, quantisation, serving runtime, prompt length, output length, batch size and concurrency. The ranges below are illustrative reference points; validate against your specific configuration before planning infrastructure.
- Compact open-weight model on a single modern GPU: 20 to 150ms time to first token; 40 to 80 output tokens per second at low concurrency
- Managed cloud API including network round-trip: 200ms to 2s median end-to-end response latency
See how GenAI Protos architects private AI deployments on enterprise infrastructure.
Open-Weight Models: The Deployment Economics Case
The open-weight model ecosystem is why compact model deployment is now a serious enterprise option. In 2026, you can pull a production-grade open-weight model, apply adapter-based fine-tuning to a domain-specific task, and serve it behind an OpenAI-compatible API endpoint - without proprietary model dependencies.
Licensing: not all open-weight models are equivalent
- Phi-4 (Microsoft): MIT licence - permissive commercial use
- Ministral 8B (Mistral AI): Apache 2.0 - permissive commercial use; current recommended replacement for Mistral 7B, which was retired March 2025
- Llama 3.x (Meta): Custom Community Licence - commercial use permitted with conditions; redistribution restricted above 700M MAU
- Gemma (Google): Google Gemma Terms of Use - not a standard OSI licence
Review the licence for your specific use case before deploying in production.
Illustrative open-weight models (mid-2026) [FIX #3 - model-agnostic framing with caveats]
Model catalogues change frequently. Validate against current provider model cards before selecting.
- Microsoft Phi-4 (14B, MIT): Strong reasoning-to-size ratio. Available on HuggingFace.
- Meta Llama 3.x family: Range from 1B edge-optimised to larger multi-GPU configurations. The 11B variant is vision-oriented - do not group with text-only models without noting the modality difference.
- Mistral AI current family: Mistral 7B is listed as retired (March 2025); Ministral 8B is the current recommended instruction-following model. Verify the model card before selection.
Fine-tuning economics: method determines cost
Adapter-based approaches such as LoRA or QLoRA reduce compute requirements significantly compared to full-weight training. Cost and runtime depend on dataset size, sequence length, training method, GPU type and epoch count - estimates range from under a hundred dollars for QLoRA on smaller models to several thousand dollars for full fine-tuning at production data scale.
Distinguish fine-tuning from retrieval augmentation. Many domain-accuracy problems are better addressed first through retrieval, structured output constraints, improved prompts or evaluation improvements before committing to a training run.
When your private AI stack includes retrieval, see the GenAI Protos guide to RAG Applications for retrieval architecture on private infrastructure.
What Private AI Actually Requires From Your Model Choice
Private or on-premises deployment may be preferable when the organisation requires a tightly controlled processing boundary, offline operation or specific residency guarantees. It is one architecture option - not an automatic compliance requirement for every regulated workload.
The model choice drives these deployment considerations:
- Memory envelope: Compact models run on a single GPU; larger models require multi-GPU or CPU offloading. Memory requirement depends on parameter count, precision (FP16 vs INT4), KV cache, batch size and framework overhead - not parameter count alone.
- Data control: On-premise keeps inference within your network perimeter. Managed API deployments require contractual and technical safeguards to achieve equivalent data control.
- Customisation rights: Open-weight models allow fine-tuning, LoRA and quantisation within licence terms. Proprietary API fine-tuning typically does not transfer model weights.
Inference cost: use a formula, not a generalised figure
API cost formula: (input tokens x input rate) + (output tokens x output rate) + tool and retry costs. GPT-4o published rates as of this writing: approximately $2.50 per million input tokens and $10.00 per million output tokens ($0.0025 per thousand input tokens; $0.010 per thousand output tokens). Verify current pricing at the provider's developer documentation before quoting.
Self-hosting cost formula: hardware or instance amortisation + utilisation overhead + electricity + storage + engineering operations + monitoring + model maintenance. Break-even depends on call volume, prompt length, output length and the specific model - not a universal timeframe.
What Teams Get Wrong About SLM vs LLM Selection
Mistake 1: Benchmarking on general capability, not task performance.
MMLU scores measure broad world knowledge. Your task is extracting invoice line items or classifying support tickets. A smaller model fine-tuned and evaluated on your representative production data may match or outperform a larger general-purpose model on that specific evaluation set. Run evals on your data with defined acceptance thresholds - not the public leaderboard.
Mistake 2: Underestimating total cost of ownership.
Cloud LLM API looks cheap until you multiply by call volume and define average token counts per request. Self-hosting looks cheap until you add engineering operations, monitoring and model maintenance overhead. Build a scenario-based TCO estimate using your actual prompt structure, output length and monthly volume.
Mistake 3: Treating all regulated workloads as requiring on-premises deployment.
Private infrastructure can make certain controls easier to implement and audit. Compliance depends on the full system: identity and access management, encryption, audit logging, data minimisation, retention, contracts, incident response and risk assessment. On-premises is one architectural option - evaluate it against the specific regulatory obligations for your workload.
Mistake 4: Skipping evaluation before concluding compact models do not work.
A base open-weight model will underperform a task-tuned version. Before concluding a compact model is insufficient, test with: production-representative prompts, structured output constraints, retrieval augmentation where relevant, and adapter-based fine-tuning on domain data. Measure the tuned system, not the base model.
A Two-Stage Decision Framework for SLM vs LLM
Selecting a model class and selecting a deployment architecture are separate decisions. A compact open-weight model may be consumed through a managed API. A frontier model may run in a private VPC. Conflating the two produces misleading comparisons.
When a Hybrid SLM-First / LLM-Escalation Architecture Is Better
Many enterprises should not choose between a compact model and a frontier model - they should route between them. A hybrid pattern reduces API cost, keeps sensitive data on the controlled path, and reserves frontier capacity for tasks that actually need it.
A practical routing architecture:
- Classification, extraction and routine queries go to the smaller local model.
- Low-confidence responses or high-complexity queries escalate to a larger model.
- Sensitive requests that must not leave the authorised processing boundary stay on the local path regardless of confidence.
- Evaluation and routing logs measure when escalation is triggered and whether the escalation improved the outcome.
Over time, routing logs identify cases the compact model handles reliably and cases where escalation is consistently needed - driving targeted fine-tuning decisions and reducing unnecessary escalation costs.
See how GenAI Protos designs multi-agent AI architectures that incorporate routing between model tiers.
Key Takeaways
- "Small language model" is an informal term with no universally accepted parameter threshold. Use operational deployment categories when planning infrastructure.
- Local inference may be preferable for residency, economics or latency - but compliance depends on the full system, not the deployment mode alone. Neither GDPR, HIPAA nor FedRAMP automatically prohibit cloud AI inference.
- Open-weight model licences differ materially: MIT (Phi-4), Apache 2.0 (Ministral 8B), Meta Community Licence (Llama 3.x) and custom terms (Gemma). Review licence terms before production deployment.
- Inference economics require a formula, not a generalised figure. GPT-4o costs approximately $2.50 per million input tokens and $10 per million output tokens as of this writing - verify current pricing before quoting.
- A compact model fine-tuned and evaluated on representative production data may match or outperform a larger general-purpose model on that specific evaluation set. Validate with your own evals and defined acceptance thresholds.
- Hybrid routing - compact model for routine tasks, larger model escalation for complexity - reduces cost, keeps sensitive data on the controlled path and produces data to drive further optimisation.
Conclusion
The SLM vs LLM decision is two decisions: which model class fits the task, and which deployment architecture fits the data governance and operational requirements. They are separate axes - do not conflate them.
For many enterprise workloads - classification, extraction, internal Q&A, structured output, RAG - a fine-tuned compact open-weight model on controlled infrastructure performs well, costs less at scale and keeps data within a defined boundary. The ceiling matters only for a specific class of tasks. Know whether your task hits it before paying for it.
Build the evaluation framework first: production-representative test data, defined acceptance thresholds, a scenario-based TCO estimate, and a clear view of your data governance obligations. Run the evals. Then make the call.
