Bringing Neoclouds to the Enterprise

Since I wrote about Meta renting out its spare GPU capacity back in July, neoclouds have steadily picked up pace among enterprises. CoreWeave, Nebius, Lambda and Crusoe are all now showing up on enterprise shortlists, and it’s not hard to see why.

First of all, buying data-center GPUs outright can mean waiting months for delivery, and a hyperscaler quota increase isn’t guaranteed. A neocloud’s GPUs are already racked and ready, so a team can often get H100s in just a matter of days. 

Second, neoclouds are built solely for GPU work, so you don’t need to piece a GPU training cluster together yourself. Networking, storage and managed Kubernetes or Slurm come set up for training and inference.

And finally, pricing. Lambda lists on-demand H100s at $3.99 per GPU-hour, Nebius at $4.50 and CoreWeave at about $6.16, while Google Cloud’s H100 instances run about $11. Spot and committed pricing narrow that gap on both sides, but for a GPU-heavy training or inference job, a neocloud often makes more economic sense than a hyperscaler.

All of this, however, does not come without a warning. If you add neoclouds the same way most enterprises added their second and third public clouds, one team at a time with governance bolted on afterward, these GPU-first clouds can quickly become black boxes. 

Very expensive black boxes.

What’s inside the black box

Let’s start with cost. When several pods share a GPU, each one reports the same device-level utilization number. You can see that the GPU is busy, but not which workload is using it or how much of it that workload actually needs. 

On the clouds you already govern, that usage at least lands in your existing tagging and chargeback. On a neocloud added on the side, it shows up on a separate invoice with no attribution to the teams or workloads behind it.

With how fast enterprises are scaling AI, if you can’t see exactly what you’re spending, your bill could easily be bigger than you (or finance) expected. 

StormForge GPU optimization attributes GPU use to individual workloads on Kubernetes and breaks the cost down by cluster, namespace and workload, so you can see which teams and products are driving the spend. It also recommends better node fits, like moving a training job that only needs half of an eight-GPU node onto a four-GPU one. 

Then there’s access. Your existing clouds run through your identity provider, with the same roles, quotas and tagging rules applied everywhere. A neocloud usually starts as an account someone opened with their work email and a few API keys pasted into a pipeline. Those rules get rebuilt by hand, if they get rebuilt at all, and they drift over time. When the engineer who set it up moves on, nobody’s quite sure who still has access.

Getting to yes

Governance has a reputation for slowing developers down. Part of the reason is that each cloud you add tends to arrive with its own console, its own deployment automation, accounts and its own copy of your policies, so your team ends up rebuilding governance from scratch every time. 

Instead, each provider should be one more place your teams can provision into, under the catalog, approvals and policies you’ve already defined. Your rules stay the same even as the list of providers underneath them changes. And the whole thing has to be as fast as signing up on their own, or your teams will keep going around it. 

We built CloudBolt CMP so public clouds and private infrastructure and each plug in as a provisioning target under one set of governance, and teams get self-service with approvals and policy built in. Requests come in with an owner and tags attached, so the spend lands in chargeback instead of on a separate invoice nobody can trace. 

Questions to ask before you sign

1. What’s the full cost? The GPU-hour rate is what gets a neocloud approved, but egress, storage, idle capacity and contract minimums don’t show up in it.

2. Which workload is this for? Start with the specific job the neocloud is supposed to speed up, not a general migration.

3. How will you govern it, patch it and enforce compliance rules? If the answer is separately and by hand, budget for that work up front. 

4. Who owns the forecast? Name the person accountable for the spend before the first invoice lands.

5. How healthy is the provider? Several neoclouds carry significant debt and rely on a few large customers. Ask for incident history and know how you’d move the workload if you had to.

To see how CMP brings new providers under the governance you already have, start for free. And if GPU cost on Kubernetes is the bigger worry, watch the Kubernetes GPU optimization demo. Heading to KubeCon? Stop by booth #207.