In-VPC AI as an Enterprise Differentiator: Selling B2B Software with Dedicated On-Premise Agents

If you sell B2B software into regulated or security-conscious accounts, you have probably seen this deal pattern. The champion loves the AI feature in the demo. Then the customer's security questionnaire arrives and asks where prompts are processed, which sub-processors see document content, and whether the model provider retains anything. The deal stalls for months, or the AI feature is switched off in the contract.

For CTOs and engineering leaders, the question is no longer whether to ship AI. It is where the AI runs. This briefing makes the case that a dedicated agent running inside the customer's own VPC or data center is a product strategy as much as an architecture choice. It covers the commercial logic, the reference architecture, the hard parts, and a decision framework for whether to build it.

Why the Hosted-AI Default Breaks in Enterprise Sales#

A multi-tenant SaaS that sends customer data to a third-party LLM API creates a chain of trust obligations that the customer must evaluate:

• Data residency and sovereignty. Prompts and retrieved context may cross regions or jurisdictions.
• Sub-processor sprawl. Every model provider, vector database and observability vendor is another entry on the customer's sub-processor review.
• Retention and training terms. Even with zero-retention contracts, security teams often want technical controls, not only contractual ones.
• Blast radius. A single compromised API key in your platform could expose content from many customers at once.
• Auditability. Regulated buyers want to inspect, in their own logging stack, exactly what the AI touched.

None of this means hosted AI is wrong. For most mid-market buyers it is the right default. But for banks, insurers, healthcare providers, defense-adjacent suppliers and public-sector bodies, the data the AI is most useful on is exactly the data they cannot send out. Those customers tend to be your largest contracts.

The commercial reframe#

An in-VPC agent turns a blocker into a tier. Instead of arguing about why your hosted AI is safe, you offer a deployment in which the customer's data never leaves their network boundary. Typical effects teams report from this kind of repositioning, which you should validate against your own pipeline rather than assume:

• Shorter security review, because the data-flow diagram has fewer arrows leaving the customer perimeter.
• A natural premium tier, since the customer carries compute and you carry deployment and support complexity.
• Stickier accounts, because an agent wired into internal systems is costly to rip out.

What "Dedicated On-Premise Agent" Actually Means#

The phrase covers three distinct deployment shapes. Be precise about which one you are selling.

Reference Architecture#

The core design principle is a split between a control plane you operate and a data plane the customer operates. The agent in the customer's environment makes only outbound connections, so the customer never opens an inbound firewall rule.

+--------------------- Vendor SaaS (control plane) ---------------------+
|  Fleet registry | Signed releases | Policy/config | Metadata telemetry |
+----------------------------------^-----------------------------------+
                                   | outbound HTTPS/mTLS only (443)
+----------------- Customer VPC (data plane) --------------------------+
|  Agent runtime  -->  Inference gateway  -->  Model server (vLLM)      |
|       |                     |                                         |
|  Connectors (DB, DMS, ITSM)  Vector store (pgvector) + audit log sink |
+-----------------------------------------------------------------------+

Components#

1. Agent runtime. The orchestration loop, tool definitions and guardrails. Package it as a container and a Helm chart so that it installs on any conformant Kubernetes cluster (EKS, AKS, GKE, OpenShift or a bare-metal distribution).
2. Inference gateway. A thin internal service that fronts the model. It lets you swap models without touching agent code, and it is where you enforce token budgets, PII redaction and logging.
3. Model server. For self-hosted models, vLLM exposes an OpenAI-compatible HTTP API and handles continuous batching. Alternatives include Hugging Face TGI and NVIDIA Triton. For the private-endpoint pattern, this layer is replaced by the cloud provider's managed endpoint.
4. Retrieval store. Postgres with pgvector is a sensible default because customers already know how to back it up and secure it.
5. Connectors. Read-only service accounts into the customer's systems of record, scoped per use case.
6. Audit sink. Structured logs written to the customer's SIEM (Splunk, Elastic, Sentinel) rather than to your cloud.

Packaging: a minimal Helm values file#

Make the security-relevant defaults explicit and overridable. A trimmed example:

400 font-semibold">class=400 font-semibold">class="text-emerald-300">"text-slate-500 italic"># values.yaml (excerpt)
agent:
  image:
    repository: registry.vendor.example/agent
    tag: 400 font-semibold">class="text-emerald-300">"2.4.1"
    digest: 400 font-semibold">class="text-emerald-300">"sha256:<pinned-digest>"
  controlPlane:
    url: https:400 font-semibold">class=400 font-semibold">class="text-emerald-300">"text-slate-500 italic">//fleet.vendor.example
    mtls:
      secretName: agent-mtls
  telemetry:
    mode: metadata-only        400 font-semibold">class=400 font-semibold">class="text-emerald-300">"text-slate-500 italic"># never prompt or completion content
inference:
  provider: vllm               400 font-semibold">class=400 font-semibold">class="text-emerald-300">"text-slate-500 italic"># or: bedrock-400 font-semibold">private, azure-openai-400 font-semibold">private
  endpoint: http:400 font-semibold">class=400 font-semibold">class="text-emerald-300">"text-slate-500 italic">//vllm.ai-system.svc:8000/v1
  maxTokensPerRequest: 4096
networkPolicy:
  enabled: 400">true
  egressAllow:
    - fleet.vendor.example:443

And a Kubernetes NetworkPolicy that turns "no data leaves" from a claim into an enforced property:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: agent-400 font-semibold">default-deny-egress
  namespace: ai-system
spec:
  podSelector:
    matchLabels: { app: agent }
  policyTypes: [Egress]
  egress:
    - to:
        - podSelector:
            matchLabels: { app: inference-gateway }
      ports: [{ protocol: TCP, port: 8000 }]
    - to:
        - ipBlock: { cidr: 10.20.0.0/16 }   400 font-semibold">class=400 font-semibold">class="text-emerald-300">"text-slate-500 italic"># customer connector subnet
      ports: [{ protocol: TCP, port: 5432 }]

Note that standard Kubernetes NetworkPolicy matches IP blocks, not hostnames. To allow only the control-plane FQDN, use a CNI that supports DNS-aware policy (for example Cilium's toFQDNs) or route egress through a proxy. Document this plainly, because security reviewers will ask.

The Hard Parts Nobody Puts in the Pitch Deck#

1. Model quality versus deployability#

Open-weight models that fit on a customer's GPUs are generally less capable than frontier hosted models, and the gap matters most on complex multi-step reasoning. Design for this honestly:

• Choose tasks that smaller models handle well: extraction, classification, summarization, retrieval-grounded question answering.
• Build an evaluation harness and run it against every model you support, so a "supported" model has measured acceptance thresholds.
• Offer the private-endpoint pattern as an upgrade path for customers who accept a managed model.

2. GPU economics#

The customer pays for inference hardware, and they will compare it against your hosted price. A worked example with stated assumptions: suppose a customer provisions two 80 GB-class GPU instances at a hypothetical on-demand rate of 4 USD per GPU-hour, run continuously. That is 2 x 4 x 730, about 5,840 USD per month before storage, networking and staff time. If their expected AI usage would cost 1,500 USD per month on a hosted API, the in-VPC deployment only makes sense when privacy or latency requirements, not cost, are driving the decision. Do the arithmetic with the customer's real figures during scoping, and consider quantized models or smaller instance classes for light workloads.

3. Upgrades and fleet management#

Once software runs in 40 environments you do not control, every release becomes a distributed systems problem.

• Use semantic versioning and publish a compatibility matrix across agent, gateway and model versions.
• Sign artifacts with Sigstore cosign and publish SBOMs so customers can verify what they run.
• Support N-1 upgrades and rollback by default, and let customers pin versions behind their own change-control windows.
• Pull configuration; do not push. The agent polls the control plane over outbound TLS.

4. Observability without data exfiltration#

You cannot support what you cannot see, but you must not see customer content. Emit metadata only: latency histograms, token counts, error classes, model and agent versions, retrieval hit rates. Use OpenTelemetry with an exporter that the customer can redirect to their own collector. Offer a customer-triggered diagnostic bundle that is redacted locally and reviewed before upload.

5. Tenancy inside the deployment#

Even within one customer, different business units may need isolation. Enforce authorization at the retrieval layer, not in the prompt. If a user cannot read a document in the source system, the vector search must not return it. A minimal pattern with pgvector:

400 font-semibold">SELECT chunk_id, content
400 font-semibold">FROM document_chunks
400 font-semibold">WHERE acl_group_id = ANY(:user_group_ids)   -- permission filter first
400 font-semibold">ORDER BY embedding <=> :query_embedding
LIMIT 8;

Synchronize ACLs from the source system on a schedule, and treat stale permissions as a security defect with a defined maximum lag.

Security and Compliance Posture That Wins Reviews#

Security teams respond to evidence, not adjectives. Prepare a standard package:

• A data-flow diagram showing every network path, with the default-deny egress policy highlighted.
• A threat model covering prompt injection, tool abuse, and data exfiltration through model output. Reference the OWASP Top 10 for LLM Applications and map each risk to a control.
• Least-privilege connector scopes: read-only unless a write action is explicitly approved, with human-in-the-loop confirmation for destructive tools.
• Immutable audit logs of every tool call, with the invoking user identity.
• Mapping of controls to the customer's frameworks, such as SOC 2, ISO 27001, HIPAA safeguards or NIS2, being careful to say "supports" rather than "makes you compliant". Compliance is a property of the customer's whole system.

Prompt injection deserves special attention in on-premise settings, because agents there are connected to powerful internal systems. Treat retrieved text as untrusted input, constrain tool schemas tightly, and require approval for actions with side effects.

Build, Buy, or Partner: A Decision Framework#

Not every B2B company should ship a dedicated agent. Use these questions.

A staged path#

1. Stage 0: Harden the hosted offering. Region pinning, zero-retention provider terms, customer-managed encryption keys. This resolves a surprising share of objections cheaply.
2. Stage 1: Private endpoints. Run the agent in your cloud but reach the model over private connectivity, or deploy the agent into the customer's cloud account with a managed model endpoint.
3. Stage 2: Dedicated in-VPC agent. Ship the Helm chart, the control-plane registry and the supportability tooling. Start with two or three design-partner customers, not a general release.
4. Stage 3: Air-gapped. Only when contract value justifies offline bundles, manual update media and slower support cycles.

Each stage should have exit criteria tied to revenue and support load, not to engineering enthusiasm.

Metrics to Track#

• Security review cycle time for deals with and without the in-VPC option.
• Install-to-first-value time in the customer environment. If it exceeds a few weeks, the packaging needs work.
• Support tickets per deployment per month, split between infrastructure and product issues.
• Upgrade adoption lag: the share of fleet within one minor version of current.
• Evaluation score drift per supported model after each agent release.

Conclusion: A Decision, Not a Trend#

In-VPC AI is not a universal answer, and treating it as one will bury a small team in support work. It is, however, one of the few product moves that directly addresses the objection blocking your largest and most regulated deals, and it creates a defensible tier that hosted-only competitors struggle to match.

A practical way to decide this quarter:

1. Quantify how many open or lost opportunities cite AI data handling. If the number is material, proceed.
2. Complete Stage 0 hardening regardless, since it costs little and helps every deal.
3. Select two design partners and build Stage 2 against their real environments, with a metadata-only telemetry contract agreed up front.
4. Set a go/no-go review after the first two production installs, using the metrics above.

If you are weighing this for your own platform, knetwork.live works with engineering teams on exactly this kind of design: scoping the deployment model, building the agent runtime and packaging, and setting up the evaluation and release tooling. A short architecture consultation is usually enough to tell you which stage fits your product and your buyers.