Your security team approved the private AI deployment six months ago. The model runs on your hardware. The data never leaves your VPC. On paper, you have private AI security. In practice, the inference endpoint has no auth layer, logs write to a shared volume, and two contractors have read access to the embedding store. That is not secure. That is a liability wearing a data-sovereignty label.
private AI without access controls, audit trails and data isolation is a demo, not a production system. This post maps the three control layers every private AI security stack must get right before it touches regulated data, and the four LLM security surfaces most teams miss entirely.
What Private AI Actually Means (and What It Does Not)
Private AI is inference and fine-tuning that runs on infrastructure you control, with data that never transits a third-party API.
A private LLM on your Kubernetes cluster is private. The same model accessed through a vendor's hosted endpoint is not, even if the vendor claims 'no training on your data.' The prompt still leaves your network. The response still passes through their systems.
The boundary matters because 'private' has no legal definition. GDPR Article 32, HIPAA's Technical Safeguard Rule and SOC 2 Type II all specify controls, not deployment modes. Calling something private AI does not satisfy compliance. Implementing encryption, access management and audit logs does.
The real attack surfaces in a private AI deployment are four: the data pipeline feeding the model, the API gateway in front of inference, the vector store holding embeddings, and the logging infrastructure capturing every prompt and response. Most teams map only the first one and call it done.
LLM Security Risks That Stop Private Deployments from Shipping
LLM security in a private deployment covers four threat surfaces most architects miss.
Prompt injection: treat all inputs and retrieved content as untrusted. A single sanitise-and-log step is not sufficient. The OWASP LLM Top 10 recommends layered controls: constraining model behaviour via system-prompt hardening, output validation, strict retrieval permissions, least-privilege tool access, separation of external content from trusted instructions, and human approval for high-risk actions. Test adversarially before every production release.
Embedding leakage: vector databases store dense representations of your proprietary data. Without encryption at rest and namespace isolation per tenant, one misconfigured permission exposes your entire corpus. Any private LLM backed by a RAG pipeline needs retrieval-layer isolation, not just model-layer isolation.
Model weight theft: on-premises LLM weights represent months of fine-tuning and significant GPU spend. Network egress controls and filesystem ACLs on the artifact directory are non-negotiable. AI data protection begins with the weights, not the inference endpoint. See GenAI Protos's private AI consulting engagements for how we scope model-security reviews.
Log sprawl: inference logs aggregating token counts and user IDs create an uncontrolled secondary data store with no retention policy. Define a log schema before go-live. Strip PII at the collector. Retrofitting audit infrastructure after go-live costs materially more than building it in from the start.
Self-Hosted LLM vs Cloud AI: The Security Trade-off
Choosing a self-hosted LLM means you control the security stack entirely, and you own every failure in it.
Cloud AI offloads model operations but introduces data egress risk. Every prompt sent to a third-party API is a potential data flow under GDPR and CCPA. A self-hosted LLM eliminates that risk and adds operational overhead: GPU provisioning, runtime management, model serving framework patching and vector store maintenance, all yours.
The cost gap is real. Cloud AI operations typically run at lower cost than a self-hosted LLM at equivalent throughput, though the actual difference varies significantly by workload, provider and infrastructure. A self-hosted LLM hitting p95 latency targets under 500 ms for enterprise Q&A tasks requires batching, quantisation (INT8 and INT4 both reduce GPU memory materially) and a load-balanced inference cluster.
Most teams choose self-hosted LLM for regulated industries, finance, healthcare and defense, where data residency is non-negotiable. Cloud AI works for unstructured, low-sensitivity workloads where cost and time-to-value outweigh isolation requirements. AI data protection in a self-hosted environment is entirely your responsibility, the vendor's shared model no longer applies.
Private LLM Data Controls: Encryption, Residency and Isolation
Private LLM data controls operate at three layers: storage, transit and compute.
Storage: use AES-256 to protect model weights, vector embeddings and inference logs at rest, with quarterly key rotation and HSM or KMS integration so every key access is audit-logged. GDPR Article 32 requires technical measures appropriate to the processing risk, it does not mandate specific cipher suites. AES-256 at rest and TLS 1.3 in transit are widely adopted industry-standard choices that auditors and regulators commonly expect to see documented, but they are architecture recommendations, not direct legal mandates.
Transit: enforce TLS 1.3 on all inference endpoints, including internal service-to-service calls inside the cluster. Mutual TLS (mTLS) between your orchestration layer and model server prevents lateral movement if one service is compromised. Internal does not mean trusted.
Compute: AI data protection at the compute layer means isolating inference workloads by tenant or data classification. Kubernetes namespaces provide logical and policy isolation, RBAC scopes, resource quotas and network policies, but they do not by themselves prevent memory sharing between pods on the same node. For sensitive multi-tenant inference, add NetworkPolicies, dedicated node pools or sandboxed runtimes. confidential AI enclaves (AMD SEV, Intel TDX) encrypt the memory space of the inference process itself, blocking host-level access to in-flight prompts. See the GenAI Protos Private AI Expertise for reference architecture.
Data residency: document every significant processing step a prompt takes from user input to model response. This data-flow map should support your GDPR Article 30 record of processing activities, alongside processing purposes, data categories, recipients and retention periods, but it does not replace the complete record.
Enterprise AI Access Management: RBAC and API Gateway Controls
Every inference endpoint is an API. Treat it exactly like one.
Implement role-based access control (RBAC) at the API gateway layer, not inside the model. The model has no concept of permissions, your gateway does. Define three roles: read-only (inference only), read-write (inference plus fine-tune trigger), admin (model swap, config change). Assign roles to service accounts, not individuals.
Least privilege matters more in private AI than in most systems because the model can be prompted to surface data it was not designed to expose. Implement output filtering at the gateway for sensitive field masking. Test adversarially before every release, not just at initial launch.
For on-premises LLM deployments, integrate with your existing LDAP or Active Directory via OIDC, do not build a parallel identity system. Rotate service account API keys on a defined schedule. Log every key issuance and revocation with the requesting identity and timestamp. GenAI Protos's LLM deployment practice covers gateway-level access patterns across major inference stacks.
Confidential AI and On-Premises LLM Audit Architecture
Confidential AI protects data in use, during inference, not just at rest or in transit.
A minimum audit trail for on-premises LLM deployments must capture: user identity (hashed), session ID, timestamp, input token count (not the raw prompt), output token count, model version, p50 and p95 latency, and output filter actions triggered. Do not store raw prompts in audit logs, use redacted or tokenised summaries to avoid creating a secondary PII store.
Compliance retention requirements vary by framework and should be defined with your compliance team, not assumed. HIPAA requires specified policies, procedures and compliance documentation to be retained for six years, but security-log retention for your specific deployment should be determined by your risk analysis, BAA obligations and applicable record-retention requirements, not a blanket rule. For SOC 2, retention periods and storage controls depend on your defined control objectives and examination scope. Tamper-evident or WORM storage strengthens evidence integrity, confirm the required period with your auditor.
For GDPR, you must be able to respond to data-subject access requests without undue delay and generally within one month. Your audit infrastructure should support retrieval of all processing events touching a data subject's data within that window. Build this capability into the logging infrastructure from day one.
Ship all logs to a SIEM that is physically and logically separate from your inference infrastructure. Separation of concerns is not optional in a regulated environment.
What Teams Get Wrong About Private AI Security
Three mistakes appear in almost every private AI engagement GenAI Protos reviews.
Common pitfall: treating the model host as the entire attack surface. What to do instead: map all four surfaces, data pipeline, API gateway, vector store and logging infrastructure, before the first inference call goes to production.
Common pitfall: skipping LLM security controls because 'the deployment is internal.' What to do instead: internal does not mean trusted. LLM security does not stop at the network perimeter. Implement RBAC at the gateway. Log every access attempt, authorised or not.
Common pitfall: defining the log schema after a security incident. What to do instead: define log fields, retention policy and AI data protection controls at architecture review, aligned with your compliance framework. If you wait until go-live to wire up audit infrastructure, you are already behind.
Key Takeaways
- Private AI without access controls and audit logs is a liability, not a security posture.
- LLM security covers four surfaces: data pipeline, API gateway, vector store and inference logs, map all four before go-live.
- A self-hosted LLM gives you full data residency and full ownership of every security failure; the operational overhead is real.
- private LLM data controls require strong encryption at rest, TLS 1.3 in transit and tenant-level isolation at the retrieval layer.
- confidential AI and on-premises LLM audit-log retention must be defined by your specific compliance framework and risk analysis, HIPAA, SOC 2 and GDPR requirements differ; confirm with your compliance team.
- AI data protection covers three layers, storage, transit and compute; missing one layer leaves the others exposed.
Conclusion
Private AI security is not a single control. It is four attack surfaces, multiple compliance frameworks and a log schema most teams do not define until an auditor requests it.
The one move to make first: map your inference data flow end to end. Every hop a prompt takes is a potential audit event. If you cannot trace it, you cannot control it.
