Enterprise AI Infrastructure TCO: A Worked On-Premise Business Case

Enterprise AI infrastructure total cost of ownership (TCO) is the cost of delivering an agreed AI service over its operating life. It includes the computing equipment, facility, software, people and services required to make that service usable. Comparing a GPU purchase with an API token price misses most of that boundary. A credible on-premise business case starts with equivalent work, measures the system that performs it, and tests how the result changes when demand, equipment life or operating costs differ from the plan.

Our on-premise AI inference guide explains the deployment decision. This article develops the calculation behind it, using a fully disclosed example that a technical team and a finance team can reproduce together. The example is illustrative: its equipment prices, capacity, tariff and alternative-service rate are assumptions, not market quotations or ModulEdge prices.

TL;DR

• Define the service before costing the hardware: model quality, workload, input/output sizes, latency, availability and data controls must match across alternatives.
• Count complete installed costs and operating costs. Separate existing sunk costs from incremental spending, and show shared-cost allocation explicitly.
• Measure useful output at the required service level. GPU utilization, scheduled activity and accepted business tasks are different quantities.
• Use energy measurements or an explicit load model. Annual power usage effectiveness (PUE) helps estimate annual facility energy; it does not size the utility connection.
• Calculate a base case, downside cases and a break-even point from your own inputs. There is no universal utilization threshold at which on-premise AI becomes cheaper.
• Keep the spending decision separate from ROI. Lower infrastructure cost creates value only if the resulting service does useful work.

Establish a comparable unit of useful work

Choose a unit that remains meaningful when the deployment changes. For document extraction, it might be a document whose fields pass validation. For a coding assistant, it could be an evaluated task completed within a defined interaction budget. For a model-serving platform, input and output tokens remain useful technical measures, but they need a model, quality target and latency boundary.

The FinOps Foundation's unit-economics guidance distinguishes resource measures from business measures. Applied here, cost per token describes part of the production process; cost per accepted task connects that process to what the organization needs. Track both where possible. A model that generates inexpensive tokens but causes more retries or human correction may cost more per successful outcome.

Write a workload card before requesting prices:

Compare deployments against the same card. A batch service with an overnight completion window is not economically equivalent to an interactive service that must answer immediately. A provider's advertised model name is not proof that quality, context handling or operational controls match the local alternative.

MLPerf Inference provides useful discipline: benchmarks specify datasets, quality targets and serving scenarios rather than treating all throughput figures as interchangeable. Its results can inform a shortlist. Your own acceptance workload still determines whether the purchased system delivers the required service.

Draw the cost boundary before building the spreadsheet

Separate one-time investment, recurring operating cost and consumption-driven cost. Then assign an owner to every line. An expense does not disappear because it sits in another department's budget.

The modular data center cost guide addresses physical-infrastructure pricing boundaries. The enterprise model must add the IT and operating layers around that facility scope. Where a supplier offers both components, require an inclusion schedule so the same UPS, commissioning service or support contract is not counted twice.

Existing assets need two views. An incremental view asks what additional cash the proposed service requires. A fully allocated view also assigns an appropriate share of existing facilities, staff and shared platforms. The first supports a near-term placement decision; the second exposes whether the service can sustain its operating model. Label the view rather than presenting one as the only legitimate answer.

Convert a performance test into capacity you can budget

Benchmark the complete proposed system with its production software and representative demand. Record accepted results per hour, failure/retry rate and measured AC power alongside latency. A throughput number without those conditions cannot support a cost-per-task denominator.

NVIDIA's inference-benchmarking guidance uses metrics including time to first token, inter-token latency and request throughput, and shows how concurrency changes the latency-throughput relationship. The procurement implication is straightforward: buy the capacity available inside your service envelope, rather than a maximum obtained under different conditions.

Reserve capacity must be explicit. If maintenance requires one node to be unavailable while the service continues, establish the required throughput in that operating state. Do not cost four nodes, use the throughput of all four, and then promise that the same capacity remains during a node outage. Our GPU data center planning guide follows the corresponding hardware-to-facility interfaces.

Keep three measurements separate:

1. GPU utilization: a hardware/runtime metric whose interpretation depends on the tool and workload.
2. Productive activity: the fraction of time the fleet delivers the validated service at the assumed rate.
3. Useful output: tasks that meet the acceptance criteria, after retries and rejection.

The worked model below uses productive activity. It does not assume that a GPU-utilization percentage equals the same fraction of useful throughput or electricity demand.

Worked example: a three-year enterprise inference service

Assume an enterprise is evaluating a defined document-processing service. For this calculation, assume 2,000 accepted tasks per productive hour at the agreed completion-time and quality targets, including the specified reserve arrangement. Installed equipment consists of four servers and shared infrastructure. A real proposal must establish that throughput through testing. The throughput and power figures below are illustrative inputs for the complete arrangement; they are not measured results or specifications of a named GPU or ModulEdge product.

Use a three-year evaluation period, 8,760 hours per year and 50% productive activity. The simplified two-state energy model assumes 18 kW total IT power while productive and 8 kW while idle. Actual production should replace this approximation with measured interval data, including maintenance and seasonal effects.

The facility allocation is a scenario input, not the price of a complete data center. Zero residual credit makes the calculation easy to inspect; it does not assert that all assets expire after three years. A real model should use separate lives, refresh dates and residual assumptions for servers, network and facility assets.

Calculate output and energy

At 50% productive activity:

Productive hours = 8,760 × 0.50 = 4,380 hours/year
Accepted tasks = 2,000 × 4,380 = 8,760,000 tasks/year
Average IT power = 18 × 0.50 + 8 × 0.50 = 13 kW
IT energy = 13 × 8,760 = 113,880 kWh/year
Facility energy = 113,880 × 1.30 = 148,044 kWh/year
Electricity cost = 148,044 × $0.15 = $22,206.60/year

The US Department of Energy defines PUE as total annual facility energy divided by IT equipment energy. It is an energy ratio for a stated boundary. Applying 1.30 here is a modelling assumption, not a measured efficiency claim. It says nothing by itself about peak cooling duty, UPS capacity, battery recharge, transients or utility-connection size. The PUE guide explains that distinction in more detail.

If the facility cost already includes an electricity service charge, replace the energy line with that contract's charging method; do not add both. At a shared site, an average site PUE may also differ from the incremental energy attributable to this deployment. Keep the allocation method visible.

This example models only a flat per-kWh energy charge; additional utility charges are zero in the calculation. A real bill may include demand or capacity charges, fixed fees and time-varying energy rates. DOE's utility-rate guidance distinguishes these components. Apply the actual billing rules to the relevant load and interval data. If a blended rate already includes a charge, do not add that charge again.

Calculate annualized TCO and unit cost

Spread the $700,000 initial investment over the three-year comparison period for this first, undiscounted unit-cost view:

Annualized capital = $700,000 / 3 = $233,333.33/year
Annualized TCO = $233,333.33 + $145,000 + $22,206.60
               = $400,539.93/year
Cost per accepted task = $400,539.93 / 8,760,000
                       = $0.04572/task
Three-year undiscounted cost = $700,000 + 3 × ($145,000 + $22,206.60)
                             = $1,201,619.80

This spreading is a comparison convention, not an instruction about accounting depreciation or tax treatment. The example excludes taxes, financing, escalation and residual receipts on both sides. It assumes installation costs are complete and no refresh is needed within the three-year window. Each excluded item becomes a visible input when it matters to the actual decision.

Compare an alternative service on the same boundary

Suppose a separately evaluated managed service meets the same workload card at an illustrative $0.06 per accepted task, plus $12,000 per year of customer-side platform, connectivity and administration cost. This is a hypothetical equivalent-service rate, not a current provider price or a token-to-task conversion.

At 8.76 million tasks, its annual cost is:

$12,000 + 8,760,000 × $0.06 = $537,600/year

The local case is $137,060.07 lower per year on the annualized view, or $411,180.20 lower over three undiscounted years. That difference depends entirely on the disclosed assumptions. It is not a general on-premise savings claim.

For an API alternative, build the charge from the actual input/output-token distribution, cache-hit assumptions, tool calls, retries and batch/priority tiers. For rented GPU capacity, use the billed reservation or running hours, support, storage and transfer charges. Neither is automatically equivalent to a per-task managed service. Data-control and recovery requirements can also exclude an alternative before price becomes the deciding factor.

Cash timing remains different. The local case spends substantial capital before service begins; the alternative spreads payments over use. For investment approval, discount the actual dated cash flows, including refresh and residual value. NIST's building life-cycle-cost methodology provides a formal reference for comparing initial and future building-related costs. Select the organization's applicable financial assumptions rather than importing federal discount factors into an enterprise case.

Test demand before declaring a break-even threshold

Holding the illustrative fleet and assumptions fixed gives the following results. Each row changes productive activity, not the hardware count or the agreed throughput per productive hour.

Much of the local spending is fixed, while the comparator's assumed charge increases with accepted tasks. Under these particular linear assumptions, break-even occurs at approximately 36.7% productive activity. A different service rate, equipment price, staffing requirement or throughput result moves that point. It is not a GPU-utilization target or an industry threshold.

The reason is visible algebraically. Let u be productive activity. Annual local cost in this example is approximately $391,998.93 + $17,082 × u; annual alternative cost is $12,000 + $1,051,200 × u. Setting them equal gives the scenario's crossover. If demand cannot fit the installed capacity at the required latency, the model needs another fleet configuration rather than an activity value above 100%.

Run at least four more sensitivities:

• Lower accepted throughput: test a harder document mix, longer contexts or a stricter quality target. Keep the acceptance test unchanged when comparing deployments.
• Higher operating effort: include the staff actually needed for support coverage, model updates and incident response.
• Different refresh timing: add a replacement event instead of assuming software progress or resale will solve it.
• Delayed readiness: include interim hosted capacity and overlapping operating costs if the local facility or application launches late.

Separate infrastructure economics from application value

A positive cost comparison does not establish ROI. The benefits side needs evidence: time saved after review, improved service capacity, reduced error or another measurable outcome. Avoid valuing every generated answer as a successful business task. Log acceptance, rework and abandonment so the denominator reflects what the organization receives.

Nor does a low energy bill establish a good infrastructure choice. A smaller model, more efficient serving software or a different batching policy may reduce both required hardware and facility capacity. The RTX PRO 6000 server guide illustrates why exact server configuration matters; the edge inference guide adds location and operating constraints.

For expansion, model increments. Buying capacity ahead of demand can secure an operating requirement, but it also changes the cost carried by early workloads. Keep committed demand, forecast growth and optional future phases separate. A staged infrastructure plan should state which future investments are avoided if demand does not arrive.

Turn the calculation into a procurement brief

The useful output is an assumption register with evidence and an owner for each input. Attach the workload card, benchmark configuration and logs, server/rack schedule, power measurements, facility scope, operating plan and commercial comparison. Mark supplier quotes with their validity date and currency. Assign a sensitivity range to inputs that remain uncertain.

Then test physical feasibility. An attractive calculation cannot make a room support the required power, cooling or service access. Conversely, an existing suitable room may avoid a new facility investment. Compare those options before selecting ModulEdge's product range, and use the configuration process to establish the site and equipment interfaces. The infrastructure scope should follow the validated server plan, while AI software operations and business outcomes retain their own owners.

For larger inference fleets, MDC500 is one facility route to assess where the actual rack schedule and site match its scope. This is a separate assessment route, not a sizing recommendation for the example's 18 kW fleet. Its price is not embedded in the model. Bring the measured workload, rack schedule, site limitations and cost boundary to the assessment so that the facility comparison addresses the proposed deployment.

Frequently asked questions

What happens to unit cost in a month with no accepted output?

Record the spending and zero accepted tasks separately; division by zero does not produce a meaningful cost per task. Staff, support, idle power and committed capacity can still incur costs. For the annual view, include that month's costs in the numerator and only accepted annual output in the denominator. Investigate whether the gap reflects planned reserve, unavailable service or missing demand.

How should existing cloud reservations enter a migration comparison?

Show the commitment that remains payable during the comparison period, even if the workload moves. Credit resale, cancellation or reassignment only where the contract and replacement use support it. Separate unavoidable payments from future renewals that the move actually avoids. Overlap spending can make migration unattractive now even when the steady-state local service has a lower unit cost.

How do I compare servers and facilities with different asset lives?

Use a component-level cash-flow schedule. Put each purchase, refresh, maintenance change and defensible residual receipt on its expected date within a common evaluation period. Explain how remaining asset value is treated at the endpoint. A long-lived facility allocation should not silently receive the same replacement cycle as a server, and residual value should not become an unsupported credit that determines the outcome.

Can interactive and overnight batch services share one unit cost?

An aggregate can support budgeting, but retain separate service-tier economics. Each tier needs its quality, completion-time and capacity commitments. Assign shared costs consistently and test the combined schedule, including what happens when interactive demand consumes capacity planned for batch work. Cheap batch output should not conceal failure to meet the interactive service, or subsidize it invisibly in a procurement comparison.

How should a deployment starting halfway through the year be modelled?

Separate installation, acceptance and production dates. Model output and energy for the actual operating periods, while retaining staff, commitments or interim hosted services incurred before launch. Do not halve every annual cost automatically: some fees are paid in full or start before useful output. A staged model also exposes seasonal tariffs and the cash overlap between old and new arrangements.

What changes when growth requires another server or facility phase?

Create a new configuration and recalculate its capacity, reserve, power and cost. Spending can rise in steps when another server, network tier, support contract or cooling increment becomes necessary. Do not extend the existing linear activity curve beyond its validated capacity. Compare the expansion date and retained headroom with the alternative service's actual volume tiers and commitments.