The capacity and control a private AI stack delivers in operation
Once a private AI stack is live, its gains reduce to two properties an evaluator can inspect: capacity as owned compute bounded by measured floors and ceilings and set against metered token prices, and control as governance over data, models, configuration, and change. Neither depends on a provider's calendar. Both show up in operation, on a cadence, in evidence the organization itself generates.
The decision-stage case for going private and local, argued in the language of custody and risk, is covered in what organizations gain when AI runs inside their network. This piece goes one level down. The stack is sized, the hardware is deployed, and the operating team has taken over. In operation, what the organization has gained reduces to two properties: capacity and control. Capacity is what the system can do for the business, bounded by physical compute the organization owns. Control is who decides what the system does next. Both are inspectable on a cadence. Neither depends on a provider’s calendar.
Capacity is what the system can do for the business. Control is who decides what the system does next.
Capacity is owned compute, not an allowance
A metered AI service grants capacity as an allowance. Quotas, rate limits, regional availability, and terms that can change at renewal set the ceiling, and the provider can move the ceiling without consulting the customer. An owned stack has a ceiling too, but it is physical, it was measured before purchase, and the organization sets its operating point against it.
The methodology for establishing that ceiling is covered in sizing an on-premise GPU configuration against the real workload: replay the actual workload, measure the latency, throughput, and memory budgets, and read a floor and a ceiling off the run. What matters here is what those two numbers become in operation. The floor is the smallest configuration that meets the service-level target. The ceiling is the point beyond which more hardware buys nothing. Between them, the organization owns a capacity envelope that no provider can shrink at renewal.
The envelope holds through the duty cycles that define enterprise load. Regulated workloads are not steady-state internet traffic. They spike around quarter-end closing, filing deadlines, and business-hours peaks, and those spikes are known in advance because they follow the business calendar. Capacity sized to the measured peak serves the entire cycle. The apparent idle capacity in the trough is not waste. It is headroom for a peak the organization can already name and date.
The standard objection is elasticity: rented capacity grows on demand, owned capacity does not. Elasticity is priced for workloads nobody can predict. An enterprise AI workload is driven by headcount, process volume, and the business calendar, and all three are forecastable inside the organization. When the workload genuinely grows past the envelope, expansion is a sizing exercise with a known method, run against the same replay harness, and the existing hardware keeps serving production while the evaluation runs.
Marginal economics then point in the right direction. On a metered service, usage growth raises spend, because every request carries a provider-set price. On owned hardware, additional inference trends toward the cost of power and operations, so growing adoption improves the return on capital the organization has already committed. Utilization pressure inverts as well. A metered environment quietly discourages saturating rented compute. An owned stack rewards pushing toward the measured ceiling, because headroom beyond the ceiling was identified as unnecessary before purchase and never bought.
Mapping the token bill against the hardware invoice
The marginal-economics argument above can be made concrete with numbers. Two starting configurations illustrate the range. One is a cluster of two or three NVIDIA DGX Spark machines, desk-side hardware that pools its memory over NVIDIA’s clustering links. The other is a single server with three NVIDIA RTX Pro 6000 cards, the shape a small company racks once and grows into. The tables below state what each configuration costs and what it generates in a month. They then set that monthly volume against cloud API list prices and show when the hardware pays for itself from the avoided spend. A final table holds the organization’s yearly cloud budget constant and asks what that money buys as tokens or as hardware.
What the hardware costs and delivers
| Configuration | Investment estimate | Generation speed | Context window |
|---|---|---|---|
| 2x NVIDIA DGX Spark | $13,000 | ~170 tok/s | ~800K tokens |
| 3x NVIDIA DGX Spark | $19,500 | ~220 tok/s | ~1M tokens |
| Server with 3x RTX Pro 6000 | $52,000 | ~470 tok/s | ~1M tokens |
Speed and context figures are estimates for sustained, concurrent team use. For comparison, the metered tiers of frontier cloud APIs accept roughly one million tokens of context. A cloud subscription seat delivers a 200K-class window in day-to-day use, so the windows in the table run four to five times as wide.
Speed with a wide window
The configurations sustain roughly 170 to 470 tokens per second in concurrent team use, with context windows of 800K to 1M tokens. That is about four to five times the window a cloud team seat delivers in practice.
What the same tokens cost from cloud APIs
| Provider | Tier | Model | Input $/1M | Output $/1M |
|---|---|---|---|---|
| Anthropic | Flagship | Claude Fable 5.1 | $10 | $50 |
| Anthropic | Upper-mid | Claude Opus 5.5 | $4 | $20 |
| Anthropic | Mid | Claude Sonnet 5 | $2 | $10 |
| Anthropic | Small | Claude Haiku 4.5 | $1 | $5 |
| OpenAI | Flagship | gpt-6-astra | $10 | $50 |
| OpenAI | Upper-mid | gpt-5.6-sol (promo, guaranteed through 2026-11-21) | $4 | $20 |
| OpenAI | Mid | gpt-6-sol | $2 | $10 |
At a typical mix of five input tokens for every output token, the list prices above blend to about $20 per million output tokens at the mid tier, about $40 at the upper-mid tier, and about $100 at the flagship tier. Both vendors halve their list prices on Batch APIs for latency-tolerant work.
There are three ways to buy tokens: subscription seats, the metered API priced above, and owned hardware. As of September 2026, a seat costs $20-25 per month, about $100 on premium tiers. In day-to-day use, a seat delivers a 200K-class window on a tap that resets every 5 hours and closes again each week. Owned hardware has no tap. Its windows run 800K to 1M, and the only ceiling is throughput.
A budget or a ceiling
A cloud subscription seat at $20-25 a month, about $100 on premium tiers, buys a 200K-class window on a tap that resets every 5 hours and closes each week. Self-hosted windows run 800K to 1M with no weekly budget, bounded only by throughput, and ten premium seats at $100 cost $12,000 a year, about what the whole two-machine starter cluster costs once.
What the hardware produces in a month
The table below states each configuration’s monthly output at business-hours duty, and what that volume would cost from cloud APIs at the three blended anchors.
| Configuration | Output tokens per month | Mid tier | Upper-mid tier | Flagship tier |
|---|---|---|---|---|
| 2x NVIDIA DGX Spark | about 135 million | about $2,700 | about $5,400 | about $13,500 |
| 3x NVIDIA DGX Spark | about 175 million | about $3,500 | about $7,000 | about $17,400 |
| Server with 3x RTX Pro 6000 | about 370 million | about $7,400 | about $14,900 | about $37,200 |
Business-hours duty here means ten hours a day on business days, roughly 22 days a month. Run around the clock, the same hardware generates roughly three times as much: about 440 million, 570 million, and 1.2 billion output tokens a month.
Where the hardware pays for itself
Months until each configuration repays its investment from avoided token spend, at business-hours duty:
| Configuration | Mid tier | Upper-mid tier | Flagship tier |
|---|---|---|---|
| 2x NVIDIA DGX Spark | 4.8 | 2.4 | 1.0 |
| 3x NVIDIA DGX Spark | 5.6 | 2.8 | 1.1 |
| Server with 3x RTX Pro 6000 | 7.0 | 3.5 | 1.4 |
Even at mid-tier pricing, every configuration pays for itself in seven months or less. Against flagship pricing, the Spark pair repays in about a month and the server in about six weeks. Electricity is included in the payback and moves each figure by less than a tenth of a month. Around-the-clock duty shortens every payback roughly threefold: to 0.3-1.5 months for the Spark pair, 0.3-1.7 for the three-unit cluster, and 0.4-2.2 for the server.
The meter stops running
The two-machine starter cluster repays itself in about five months at mid-tier token prices, the three-card server in about seven. After that, inference trends toward the cost of power.
What the same yearly budget buys at 15, 50, and 100 users
Take the organization’s yearly cloud spend as a constant, then ask what that money buys either way: tokens from an API, or hardware that can generate them. The table below runs that comparison at 15, 50, and 100 users.
Each user is assumed to generate 25,000 output tokens per business day at business-hours duty, a full agentic-assistant workload. Each size is mapped to the tier that covers its demand with durable headroom: the Spark pair at 15 users, the three-unit Spark cluster at 50, the RTX Pro server at 100. The mappings leave headroom on purpose, at 6% to 16% average utilization. The comparison counts what the hardware can generate, not what the workload demands.
The year-one column compares the hardware’s yearly token capacity at business-hours duty against the tokens the same budget buys from an API. That multiple holds at every price anchor, because the budget is by definition the price of the demand tokens. The three-year column compares tokens per dollar on both sides: what the hardware generates per dollar of purchase price plus electricity, and what the API returns per dollar of spend.
| Users | Yearly cloud cost: mid / upper-mid / flagship | Hardware cost / tokens a year | Capacity for the same money, year 1 | Tokens per dollar over 3 years: mid / flagship |
|---|---|---|---|---|
| 15 | $1,980 / $3,960 / $9,900 | $13,000 - about 1.62B tokens a year | 16.3x | 7.3x / 36.6x |
| 50 | $6,600 / $13,200 / $33,000 | $19,500 - about 2.09B tokens a year | 6.3x | 6.3x / 31.6x |
| 100 | $13,200 / $26,400 / $66,000 | $52,250 (estimate) - about 4.47B tokens a year | 6.8x | 5.0x / 25.2x |
The 50-user row carries the plainest result. At the mid anchor, three years of cloud spend comes to $19,800, and the three-unit Spark cluster costs $19,856 over the same period, purchase price plus electricity. That is the same money within 0.3%, and the cluster can generate 6.3 times the tokens.
The mid anchor also carries the honest year-one gaps. The 15-user budget of $1,980 and the 50-user budget of $6,600 cover none of the three configurations in year one. Buying the mapped hardware outright needs an additional $11,020 at 15 users and an additional $12,900 at 50. At 100 users, the mid-tier budget of $13,200 covers only the smallest configuration. The server that row maps to needs an additional $39,050.
At the flagship anchor, the 50- and 100-user budgets buy the hardware outright in year one, with $13,500 and $13,750 left unspent. The 15-user flagship budget stays $3,100 short of the Spark pair in year one. The upper-mid anchor lands between the two columns, at 14.6x, 12.6x, and 10.1x over three years. The three-year multiples do not depend on the per-user assumption. They hold at any workload profile, lighter or heavier.
Capacity that grows with you
Measured against the same yearly cloud budget, the hardware can generate 6.3 to 16.3 times the tokens in year one. Over three years it returns 5 to 36.6 times the tokens per dollar, at any workload profile.
The caveats that travel with these numbers
All prices are third-party figures in USD, captured at the time of writing. Cloud list prices and promotions change, and hardware prices vary by seller and region. The speed, capacity, and payback figures are estimated aggregates for planning, not an offer or a quote for delivered systems. Re-run the arithmetic against the real workload before committing to either side.
Control is four governable layers
Control is a looser word in vendor material than it deserves to be. In operation it reduces to four layers an organization either governs or does not.
Data
Data governance starts with location. Inference runs inside the network, so questions about which systems processed a query have one answer and it is local. There is no vendor-side telemetry on query content, because there is no vendor side. Retention follows the organization’s own schedules. The audit trail is generated at the point of work, under access controls the organization already operates, which is what keeps finance ledgers reportable and reconstructable on demand. For technology platforms, tenant data and source IP sit under the same separation controls the platform already owes every customer contract.
Models
Model governance means the organization decides which model version serves production. Hosted providers deprecate models on their own schedules and can change behavior behind a stable endpoint name, which turns version pinning into a service-level hope. On an owned stack the serving version is pinned explicitly. Upgrades happen on the organization’s calendar, after the replacement has been evaluated against the organization’s ground-truth data rather than a public benchmark. The hardware stays put through the change: better models arrive as software and run on the same machines. Open-weight models with commercially friendly licenses keep the option set wide, and the licensing posture remains the organization’s own.
On an owned stack, the serving version is pinned explicitly. Upgrades happen on the organization’s calendar.
Configuration
Configuration governance covers everything that shapes behavior at run time: sampling parameters, routing rules, guardrails, retrieval sources, prompts. On a private stack these are versioned artifacts under the organization’s change control. A given output can be traced to the exact configuration that produced it, and the same configuration can be replayed. On a hosted endpoint, part of the configuration belongs to the provider, and reproducibility depends on the provider’s versioning policy rather than the customer’s.
Change
Change governance is the layer that decides when any of the above moves. On a private stack, releases pass through the organization’s own approval gates and on-call rotations, the same ones every other production system already passes through. No provider can retire an endpoint underneath a running workflow. Migration work happens because the organization scheduled it, not because a deprecation notice arrived.
The two properties reinforce each other
Control makes capacity legible. Because versions and configuration are pinned, throughput and latency measurements mean the same thing quarter after quarter, and the telemetry that proves it belongs to the organization rather than to a vendor dashboard. The measurement discipline is the one laid out in the metrics that make an AI investment defensible after go-live: accuracy, cycle time, avoided cost, and adoption, read against the baseline taken before the build.
Capacity makes control affordable. Evaluating a new model, a changed guardrail, or an expanded workflow happens on the same owned hardware, against the real workload, without a meter running on the experiment. Governance decisions therefore rest on measured evidence the organization produced itself.
The test for any candidate architecture
For a technical evaluator, the distinction condenses into two checks. First, ask who sets the ceiling: is maximum throughput a measured property of hardware the organization owns, or an allowance a provider can resize? Second, ask who signs the change: do model, configuration, and release decisions pass through the organization’s gates, or a vendor’s? On a private stack both answers land on the organization’s side of the table. Capacity and control then stop being procurement arguments. They become operating properties, inspected at every review cadence, on evidence the stack itself produces.
Ask who sets the ceiling, and changes to AI driving your company into the future.