Sizing an on-premise GPU configuration against the real workload

A hardware configuration that a procurement committee can defend is not a number from a vendor. It is the intersection of three measurements taken on the organization’s own workload shape: latency, throughput, and memory. Each constrains the other two, and the defensible configuration is the smallest setup that satisfies all three at the required service level. A sizing exercise that reports any one of the three in isolation is merely a forecast. It measures how hardware performs on a generic benchmark rather than what the deployment will demand under the actual workload it was purchased for.

Why the configuration is measured, not estimated

The reference architecture a hardware vendor publishes is calibrated against public benchmarks. These benchmarks are designed to compare hardware across vendors, not to surface the failure modes of a specific workload. The setup derived from a spec sheet naturally flatters the spec sheet. The hardware stack an organization actually needs emerges only when replaying real traffic - its concurrency patterns, context-length distribution, and request mix - against a target configuration. Sizing without that replay turns workload risk into procurement optimism. That optimism later surfaces in the audit committee as budget overruns or service outages.

The methodology below sizes infrastructure by replaying the real workload against candidate configurations in an in-house GPU lab and measuring the three budgets. It produces two clear bounds: the floor (the smallest setup that meets the service-level target) and the ceiling (the point beyond which additional hardware yields no measurable service gain). Procurement decisions are made between these two numbers.

Capturing the workload shape

A hardware configuration is only as accurate as the workload used to size it. The first step captures the real production workload rather than a synthetic substitute:

• the request mix, including the long tail of regulated and low-frequency queries that uniform benchmarks omit;
• the context-length distribution generated by actual users, including peak requests that stress memory rather than averages that flatter it;
• concurrency patterns driven by business operations (such as quarter-end closing, daily reconciliations, or filing deadlines) rather than steady-state averages;
• the latency sensitivity of each request class, establishing specific service-level targets rather than a single blanket threshold.

The replay harness reconstructs this shape as a standardized load profile applied across candidate configurations. Because the workload profile remains fixed, every variance in measured budgets stems directly from the hardware configuration.

The latency budget

Latency is measured as a distribution rather than an average. Sizing is governed by tail latency - typically the 95th and 99th percentiles per request class - because service degradation and audit concerns appear first in the tail. An average hides bottlenecks by blending fast routine requests with stalled edge cases. Tail latency exposes those edge cases before deployment.

Latency is the primary constraint because it limits how long a request can wait before its prefill and decode stages are queued. A setup that hits throughput targets by queuing requests into large batches will show strong aggregate throughput alongside unacceptable tail latency. That configuration looks convincing in a vendor presentation but fails in production. Measuring the latency budget against real request distributions eliminates unviable configurations early.

The throughput budget

Throughput is the sustained request rate a configuration maintains within the target latency, not its theoretical peak before saturation. Sizing is governed by the saturation point: the concurrency level where latency begins rising faster than throughput. The target operating point must sit safely below this inflection point, leaving headroom for operational peaks.

Throughput and latency interact directly through batching. Increasing batch size improves throughput. However, beyond the saturation point, larger batches increase tail latency disproportionately. The defensible operating point is the batch size that maximizes throughput while strictly honoring latency targets under test. The throughput sustained at this point forms the second budget.

The memory budget and why it usually binds

For autoregressive transformer inference, memory splits into three distinct components: model weights, activation buffers, and the key-value (KV) cache accumulated as context expands. Model weights and activation buffers are fixed by model architecture. The KV cache, however, varies dynamically with three factors: concurrent in-flight requests, input context length, and model depth.

This dynamic growth makes memory the most frequent bottleneck. Sizing based on vendor-supplied average context lengths handles routine queries but fails on long, complex requests. Sizing based on average concurrency fails during operational spikes. The KV-cache budget - concurrency multiplied by context length at the target service level - converts throughput goals into physical VRAM requirements. Measured during replay, the memory budget defines the maximum batch capacity before requests are queued, spilled, or rejected. It is typically the constraint that dictates hardware scale.

Reading the configuration off the run

Once all three budgets are measured against a fixed workload profile, hardware selection is driven by data rather than guesswork. The floor represents the minimal configuration (accelerator count, VRAM tier, and host topology) that satisfies latency and throughput targets without exceeding memory limits during peak concurrency. The ceiling marks the threshold where additional hardware yields no measurable service-level improvement. Capacity beyond the ceiling provides no operational benefit.

High availability and failure-domain margins are calculated explicitly above the floor. Adding an extra accelerator or host node ensures single-device failures cause graceful degradation rather than system outages. Accounting for this margin as dedicated physical capacity, rather than assuming operational slack, ensures resilience is genuinely engineered.

Why the measurement runs on in-house hardware

Measurements run on dedicated GPU infrastructure configured to mirror the on-premises hardware stack targeted for procurement. This approach avoids two common sources of bias. Cloud-metered environments reward higher resource consumption, subtly tilting recommendations toward the ceiling. Conversely, vendor testing environments are optimized for standard synthetic benchmarks, masking the edge cases that drive hardware requirements up. Testing on independent, dedicated hardware loaded with actual workload profiles produces unbiased floor and ceiling baselines for procurement.

The artifact the procurement is priced against

The deliverable is a comprehensive sizing dossier. It maps each candidate configuration to its measured latency, throughput, and memory performance under the workload profile, establishing the floor, operating point, and ceiling. The procurement committee bases purchasing decisions on empirical test data rather than vendor claims. The final specification includes the floor configuration plus failure-domain margins, bounded by the verified ceiling. Workload parameters - concurrency, context lengths, and query mix - are documented so the sizing can be re-evaluated whenever business requirements shift.

This empirical approach directly reinforces the Building and Validation stage. Prototype behavior on physical hardware provides verifiable justification for capital expenditure, reducing uncertainty across subsequent procurement cycles. Sizing by any other method produces an estimate that must eventually be reconciled with reality after capital is already committed.