The inference spectrum is the same on all three clouds

Four tiers, one pricing boundary

AWS recently framed inference as a spectrum with four discrete operating models. Credit where due: the framework and the AWS column below are theirs. What I want to add is the observation that the same four tiers exist on Google Cloud and Microsoft Azure. The vendors use different names and draw the pricing lines in slightly different places, but a buyer evaluating all three is looking at one shape. Treat the four tiers as a comparison lens, not a product taxonomy the vendors share.

The tiers, from left to right:

• Fully managed. A token-priced API. The vendor owns the model server; you observe tokens, latency and cost per request.
• Managed inference. Dedicated, auto-scaled endpoints running weights you chose. Billed by instance-hour plus a platform fee.
• Managed cluster. Kubernetes with inference add-ons: routing, node recovery, model-server automation. Billed by instance-hour plus a cluster fee.
• DIY inference. Raw GPU VMs and a scheduler. You own everything, including the pager.

The structural insight is where the money changes shape. The first tier bills on tokens. Dedicated capacity to its right bills for allocated compute, whether or not traffic arrives. All three clouds draw that line in roughly the same place, with two exceptions worth knowing: serverless GPU products bill on usage and scale to zero, and batch endpoints bill per job. Neither charges for idle hours.

The cross-cloud crosswalk

Read this table row by row rather than column by column. Product names are as of October 2026; verify against each vendor's pricing page before you reuse it.

The fifth row is the one real divergence. Google and Microsoft both sell a serverless GPU product that scales to zero, which blurs the line between managed inference and a managed cluster. AWS has no equivalent at the time of writing; SageMaker Serverless Inference exists, but its documented exclusions include GPUs.

Knowledge Bases, Guardrails and AgentCore on AWS; Agent Engine and Search on Google; Agent Service, AI Search and Content Safety on Azure all sit around the model endpoint. Compare them as retrieval, safety and orchestration capabilities, not as inference tiers.

Google collapses two tiers under one brand

Google folds the first two AWS tiers under one platform. That platform was Vertex AI until April 2026, when Google announced Gemini Enterprise Agent Platform as its successor; the Vertex names persist in the API endpoint and in most documentation, so you will see both. Model Garden provides both the token-priced APIs (Gemini, Claude, Llama) and the self-deploy path onto dedicated endpoints, which bill like SageMaker: by node-hour plus a management fee.

GKE is Google's answer to HyperPod, and it is the more mature of the two Kubernetes stories. Inference Gateway with KV-aware routing, Inference Quickstart profiles, and the TPU serving stack are production features Google itself uses to serve Vertex AI. That is a meaningful signal about how hardened they are.

Cloud Run with GPUs sits between tiers 2 and 3 and has no clean AWS analogue: serverless, scale-to-zero, usage billing, with regional and configuration limits.

Microsoft has the richest catalog and the thinnest cluster tier

Microsoft's first tier is the richest in model-of-record choice. OpenAI models arrive here first, and the Provisioned Throughput Unit (PTU) option gives the reserved capacity AWS sells as Bedrock Provisioned Throughput and Google sells as Vertex AI Provisioned Throughput.

The second tier is the Azure Machine Learning managed online endpoint: the direct SageMaker Inference analogue, GA, billed at the VM rate with little or no platform premium. Foundry Managed Compute is a different product. It is in preview, bills hourly per accelerator rather than per VM, and draws on its own quota. Microsoft's own docs advise against production use while it is in preview, so do not treat the two as one line item.

KAITO is Azure's HyperPod story, and it is narrower in scope. It automates GPU node-pool provisioning and model-server setup on AKS from a workspace CRD; confirm the current add-on's routing and recovery capabilities before you count on them, because at the time of writing those remain the operator's problem. One practical note: Foundry was formerly Azure AI Foundry, and before that Azure AI Studio. Readers will encounter all three names in documentation.

Where the three clouds differ

Once you accept the spectrum is shared, the differentiators narrow to three things: catalog, silicon, and how deep the managed-cluster tier goes.

For an observability program the practical consequence is where the telemetry boundary sits. In tier 1 the vendor owns the model server and you observe tokens, latency and cost per request. From tier 2 outward you also inherit GPU utilization, KV-cache hit rate and replica health as metrics you have to collect yourself.

What the same workload costs on each cloud

Two worked examples at list price, US regions, October 2026. Silicon and model classes are not identical across columns, so read these as order-of-magnitude, not quotes. They exclude storage, egress, guardrails, agent runtime and the engineering and on-call effort of operating anything past tier 1. A fair comparison for your own case needs the same workload, quality target, latency target, region and traffic pattern in every column.

Example 1 · Tier 1, token-priced

A support chatbot at 100M input and 20M output tokens per month on a mid-tier model.

Example 2 · Tier 2, instance-priced

One 8B open-weight model on a single entry GPU, running 24×7 (730 hours per month).

Two lessons from the numbers

First, in tier 1 the model you pick moves cost more than the cloud you pick. Swapping Haiku for Sonnet doubles the Bedrock line, while the three mid-tier models across three clouds land within a 2.5× band of each other. Procurement effort spent on which cloud is misallocated if nobody is governing which model.

Second, in tier 2 the managed premium over the raw VM is modest: roughly 20 percent on SageMaker, about 15 percent on Vertex, and near zero on Azure ML managed online endpoints. The expensive part is not the platform fee. It is that the instance bills every hour whether or not traffic arrives.

That makes the break-even question one of throughput. The cheapest box in Example 2 costs about as much as five Example 1 chatbots on Google. Whether it can actually serve five at your latency and quality target is an empirical question, not something the hourly rate tells you. There is no universal utilization percentage at which dedicated capacity wins; the crossover depends on the tokens per second the GPU delivers for your model, your traffic shape, and the operating cost you take on.