"Should we run our own model or use an API?" is really four questions wearing one coat: a data question, a cost question, a latency question, and an operations question. Answer them in order and the decision usually makes itself.
Start with data, because it can end the discussion
Start with the actual data policy and contract. Health or financial data does not, by category alone, force self-hosting; the permitted processing environment depends on your obligations, provider terms, controls, and deployment. If a binding rule excludes the proposed external processor, that option fails until the constraint is resolved. A signed DPA or regional endpoint alone does not prove suitability. Record the allowed data classes, retention, training use, subprocessors, and deletion requirements, and have the responsible owner approve the interpretation.
Then volume, because it flips the cost math
Many hosted APIs meter tokens, while dedicated managed endpoints can charge for provisioned capacity. Self-hosting combines hardware or GPU rental with operations and application costs. Compare options that pass the same quality and latency checks. Estimate input and output tokens separately, idle capacity, caching, retries, network costs, redundancy, and engineering time. Low or irregular utilisation can favour consumption billing; steady demand can improve self-hosted utilisation. Neither outcome follows from token volume alone.
Then latency and control
Self-hosting gives you control over deployment, model artifacts, queue policy, and upgrades. It can reduce some external network or provider dependencies, but your own network hops, queueing, capacity limits, and outages remain. Managed hosting may offer capabilities or operational support that your team values; an open model may meet a narrower task well. Measure end-to-end tail latency at expected concurrency and pin model and serving versions where supported. Do not infer answer quality or latency from hosting location.
Finally, operations — the question teams underweight
Self-hosting means you now own GPU drivers, inference-server upgrades, autoscaling, monitoring, and the 2 a.m. page when throughput collapses. If you do not have someone who wants to own that, the "cheaper" option is more expensive in the way that matters.
Compare complete bills for the same service
Put both options on the same service boundary. Include integration, monitoring, incident response, and quality evaluation on both sides. Add hardware, power or rental, patching, model deployment, and spare capacity where your team owns them. A hosted endpoint still needs an application owner, failure handling, and data controls. Compare a monthly total and identify which costs are estimates rather than measured expenses.
An arithmetic illustration, not a current price quote: suppose an API path costs 200 currency units in fixed monthly work plus 4 per million workload units, while a self-hosted path costs 1,000 fixed plus 1 per million. The simplified break-even is (1,000 − 200) / (4 − 1), about 267 million units per month. Define a unit with the same input/output mix and accepted quality before using this equation. If the local server cannot sustain that throughput or needs a second replica, the assumptions change and the calculation must be redone.
A decision tree you can actually follow
- Does the proposed processor violate a documented data requirement? Exclude that deployment until an approved alternative is established.
- Is demand low or irregular? Compare metered API and managed-endpoint costs before reserving hardware.
- Is demand steady, quality acceptable, and an operations owner available? Measure local capacity and price both complete systems.
- Does the saving disappear after redundancy, idle time, and maintenance? Keep the operationally simpler option and record the revisit trigger.
A common and sensible end state is hybrid: prototype on an API, route sensitive or high-volume paths to a self-hosted open model once the numbers and the team justify it. Whatever you choose, write down the assumption that drove it — token volume, a compliance clause, a latency target — so you know what change should make you revisit.
Put a date on the decision
Whichever path you choose, schedule a one-month review before the decision hardens into infrastructure. For an API path, compare actual token use with the estimate, count rate-limit incidents, and check whether the data terms still match your use case. For a self-hosted path, count GPU idle time, failed deploys, model reloads, and the number of hours spent on operations instead of product work. The right decision can change when volume becomes steady, when a privacy requirement appears, or when the team loses the person who understood the inference stack. A good LLM architecture decision includes the date when it should be challenged again.
Validate the capacity assumption
Before treating the arithmetic as a deployment decision, replay an approved workload at the expected peak concurrency. Record accepted responses per second, time to first token, tail latency, queue depth, and recovery after a process restart. Include a long-input case because memory and throughput can change with context size. If a candidate misses the required quality or response-time limit, exclude it from the cost comparison rather than pricing an inadequate service as though it were equivalent.