Training vs Inference Cost — The CFO's Guide to AI Economics

Training a frontier language model requires significant compute infrastructure over weeks or months. GPT-4 class training is estimated to have cost OpenAI over $100M in compute. Gemini Ultra, Google's frontier model, required similarly large investments. These are capital expenditures that appear in the balance sheet of a handful of technology companies. If a mid-sized vendor claims to have trained a proprietary foundation model, ask for evidence: compute spend, model evaluation documentation, and benchmark performance. Unsubstantiated claims of proprietary training at frontier quality are a diligence red flag.

Inference is computationally expensive because AI models require significant memory and computation to produce each output. A large language model processes every token (roughly, every word) in both the input and output through billions of mathematical operations. At the token prices charged by API providers, the cost per document, per conversation turn, or per query adds up quickly at enterprise volume. Build a token consumption model before signing any usage-based AI contract: estimate average input length, average output length, and projected monthly volume. The product of these three numbers is your monthly AI inference budget.

AI inference has near-zero marginal cost per additional unit once infrastructure is in place. Once a model is deployed on GPU infrastructure, serving the ten-thousandth query costs essentially the same as serving the tenth — electricity and bandwidth are the only truly marginal costs. This is categorically different from professional services, where serving ten times as many clients requires ten times as many people. Model AI investment economics at multiple volume scenarios. Underestimating future volume in the business case systematically understates the ROI of infrastructure investment.

API access to AI models is the fastest path to deployment and the most flexible for variable workloads. With API access, you pay per token, require no GPU infrastructure, and benefit from the provider's model improvements and security management. The trade-off: per-token cost is highest at all volume levels, data leaves your environment with each query, and you are subject to the provider's uptime, terms, and pricing decisions. API-first is a rational starting point — but include a self-hosting evaluation trigger in your AI governance framework: at what monthly spend does the self-hosted break-even case become compelling?

Latency in AI systems is measured in seconds or sub-seconds and varies dramatically by model tier and infrastructure. A small, cached model may respond in 200ms — imperceptible latency. A frontier model processing a long document may take 20–40 seconds — noticeable and frustrating in an interactive product. The latency gap between these configurations can be an order of magnitude in cost. Define your latency requirement before selecting an AI model tier or architecture. Latency requirements are a product decision that constrains cost optimisation.

AI TCO has six components that must all appear in the business case. Component one: model cost (licensing, API, or infrastructure). Two: data cost (preparation, labelling, storage). Three: integration cost (engineering to connect AI to existing systems). Four: quality assurance (evaluation, testing, red-teaming). Five: human oversight (the humans required to review AI outputs in consequential contexts). Six: governance and compliance (audit trails, regulatory documentation, legal review). Require all six TCO components to be present in any AI business case before approval. A case missing any component is incomplete and should be returned for revision.

The most powerful cost lever is model tier selection — the choice of which model to use for which task. Deploying a frontier model for all queries is like staffing every customer interaction with a specialist consultant. Routing simple, well-defined queries to smaller models and reserving frontier capability for complex cases reduces average cost per query by 50–80% with minimal quality loss on the simple majority. Maintain a task-quality matrix: for each AI use case, define the minimum acceptable quality threshold. Engineering selects the cheapest model above that threshold. Review quarterly.

Structure the AI financial narrative in three time horizons for any board presentation. Year one: capital and implementation. Model the licensing or infrastructure cost, integration engineering, data preparation, and initial governance setup. This is the CapEx-equivalent phase. Year two to three: variable operating cost growth. Model inference cost scaling with usage, ongoing retraining, and quality assurance. Year four and beyond: architecture maturation. Model the cost efficiency gains from architectural optimisation and volume-based pricing. Require three-year phased financial models for all AI investments above board materiality threshold. Single-year models should not pass the CFO review gate.