Data Center 2.0 —Untangling the software stack

When you stream a movie, post on social media, use an AI chat bot or load a website you

are tapping into a massive invisible machine.

But that machine isn't just hardware.

It is a carefully layered stack of software all working together in concert.

Software is kind of the intelligence backbone of data centers and AI factories.

So software is key to really keeping up with the fast-paced change happening and accelerated

AI compute.

But we are throwing the stack up in the air.

For me, this is as much of a tectonic shift as the internet was in 1999, or mobile was

in 2007.

You know, all the cloud has been over the last 20 years.

This is one of those big tectonic shift.

Welcome to the 590 series data center 2.0.

The show where we break down what's changing in the infrastructure that powers modern

business.

I'm Diana Gooverts.

And today we are unpacking what the data center software stack really is.

And why AI workloads are forcing a rewrite from power to cooling all the way up to applications

and services.

But before we dive in, let's have a quick word from our sponsor.

We talk a lot about GPUs in the race for AI.

But what about the backbone supporting them?

Every AI model needs a reliable home for data ingestion, APIs, and application hosting.

That's where Vulture VX1 cloud compute comes in.

Vulture VX1 offers an 82% performance per dollar advantage over traditional hyperscalers.

By moving your core workloads to Vulture, you reclaim up to a third of your budget.

Call it an AI dividend to reinvest back into your models.

Deploy globally in seconds at Vulture.com.

Software is what makes a modern data center tick.

It designs it, monitors it, secures it, and increasingly it has to optimize it in real

time as AI pushes rock density and power levels to extremes.

But the phrase data center software can be a little bit confusing because it blends

two worlds.

A single-personality software, so power cooling, building systems, and IT software, so servers,

networks, virtualization containers, observability, and security.

So what makes up the software stack is really a lot of things.

It's kind of those element managers, as I've called them.

There's going to be a power system in there.

There's going to be water systems.

there's gonna be kind of physical things,

electricity, HVAC, moving air through the data center.

So that's kind of where the Schneider Electrics,

the Honeywells, you know, that kind of layer.

Then once you get up past that kind of infrastructure element

level, then you start to get up into things

that we would recognize from IT vendors like Cisco,

like HPE, like Dell, like Lenovo, like SuperMicro.

That is where you're kind of managing servers,

you know, a classic example would be from somebody like HPE

with their ILO and Computops management.

This is pretty low level kind of,

it's the box on, is it working, is it connected,

is the link working, you know, that kind of level.

Then you start to move up into looking at the workload itself.

So that might be at the container level,

it might be the virtual machine level.

It's looking at traces, logs, metrics,

as kind of where the observability vendors start

to come in, the splunks, the diner traces,

the data dogs, the new relics of this world.

So kind of as you go up through the stack,

you're trying to monitor and observe what's going on,

whether that's from the real physical infrastructure

in the data center, right up to the workload,

and then you start to get to the application level.

So application performance monitoring

is the supply chain, up and running.

You can't answer that question

unless all the parts below it

are being monitored and observed.

On the facilities side,

Carlini uses more OT-flavored terms

and emphasizes separate silos.

So power monitoring, cooling, and BMS,

plus the growing need to unify them.

The data center software stack is really complicated.

When data centers were smaller,

you really just had an IT room that you were monitoring

and there's software that's dedicated to that,

which is usually decent software.

On the power side, you have power operations

of power monitoring software

that monitors all the way from utility,

all the way into the IT room.

And then you have the cooling software,

which is usually part of the building management software.

You have these silos with the software

that are operationally very, very good

at doing the different domains.

And that's where the big challenge is

because all of these domains are interdependent on each other,

we wanna start moving the software

to be more unified across all of the domains.

So to recap, here is the practical model.

Facilities plus IT infrastructure,

plus workload platform,

and observability security and application outcomes.

Got it?

The hard part is that each layer usually comes

with its own tools

and integration tends to break at the seams.

It's a bunch of screens,

it's a bunch of kind of interfaces to manage this stuff.

There isn't one sort of ring to rule them all.

You're gonna be in Schneider's tools,

you're gonna be in Honeywell's tools,

you're gonna be in this vendor's tools,

you're gonna be in Cisco's tools,

you're gonna be in HP's tools.

We are still a long way away from a single pane of glass

where you can manage at a data center level.

We're trying to get all of the data

from the different domains,

and we're encouraging companies

and giving them the tools to put all the data

in central repositories or a single location.

Because when you have the data from the different domains

and different places, it's really hard

to run analysis on that data.

So the first step is to try to get it

to a single location.

Things have really, I think, stepped up

in the last few months as far as advancement.

And a lot of this is being driven

by the fact that the densities,

especially in the AI factories with accelerated compute,

are almost doubling every year

with the latest evolutions and generation

from the NVIDIA GPUs.

The higher the density, the lower room for failure

because things will heat up very, very fast.

If you have a hiccup in your CDU,

or if you have some water pipe spread as a leak

or something, you're gonna overheat

and very, very short period of time.

So the importance of management

in these types of environments has never been more critical.

And when the physical side gets tighter,

the software side has to get smarter, more autonomous,

more security aware and better at understanding

what's normal and what's not.

The future stack isn't just about more dashboards.

It's about more automation, more prediction

and more control loops

because humans can't manually steer the ship anymore.

The way that AI is impacting the data center

is pretty foundational.

We were on a really good track with PUE,

sort of a measure around data center performance.

So if you go back three or four years,

we were starting to crack that problem.

We were starting to get really efficient

with how data centers moved their hot and cold aisles.

Then somebody puts GPUs in every server

and we're back to pretty foundational challenges again.

These consume a lot of power, consume a lot

and push out a lot of heat.

So that whole trend we were on has been booked back

and we're now scrambling.

So much show that if you look at Dell technology world

last year, Michael Dell was announcing on stage

as part of his keynote, a rear door heat exchanger.

That's pretty nerdy stuff,

but it was that important that it made it onto the keynote.

This is front of mind.

And what's happening is all the smart people

in the industry are having to look at all the layers

in this stack to re-architect it

because we've thrown a new component in there

that's fundamentally changed the dynamic.

The software has become a lot more intelligent.

It has become intelligent in terms of being

a bit more autonomous and certainly in terms of dealing

with one of the key constraints

of using AI at scale, which is security.

But beyond the ITOT stack that we've been talking about,

there is the software services stack

and that's where companies like NeoCloud

are looking to differentiate themselves from hyper-skillers.

NeoClouds originally pitched themselves

as bare metal providers, experts at running the very systems

we just discussed and the GPUs, CPUs

and storage systems we talked about in previous episodes.

But that may not be enough anymore.

Just being a pool of GPU capacity and renting GPU capacity,

that has got a shelf life and I'd argue

that shelf life expires at the end of this year.

It doesn't cut off at the end of December,

but it's largely a 2026 thing.

I think the race for the nebias is and the core weaves

and the lambda labs and those guys is,

they've got a race this year and put their foot flat

and floor to get away from just being seen

as renting capacity.

How do they do that?

It's been able to use the GPU as an anchor point

for those new AI-driven applications.

They've got to make the pivot from training

to inference capacity.

If the NeoClouds can become the place

where you run inference, there's a huge opportunity.

Jamie Thomas argues that the ceiling for NeoClouds

is often economic.

If a provider can offer a compelling cost equation

and real value, well, then the market will let them keep priming.

It will all be around economics.

So if they are able to offer an economic equation

to the client that is compelling,

I don't know that there's any limitation

in where they can go.

There is a lot of scale that's being achieved

by the main cloud vendors to allow them to do things

at the right cost dimension because of their scale

that I think will be a reality of the competition.

But if someone comes up with a different innovative approach

and it's cost effective and it brings value,

I think there's opportunity there.

Before we wrap, there are a few misconceptions worth retiring.

First is the idea that the stack is somehow solved.

Stephen from HyperFrame says the opposite is actually true.

We are in the middle of a tectonic shift

and anyone claiming that they've got it all figured out

is selling you comfort.

The IT and the OT kind of worlds have their own management platform

and they'll never emerge.

And I think we're starting to see that.

The industry has to see that software evolve.

To be able to not only do what it's core function is

but to extend from just power to power and cooling

or from just going to cooling and power and then IT room.

And then as you talked about moving up the stack.

So I think the myth that you need dedicated software

for each domain, I think that myth is going to start

to be dispelled as more and more the software starts

to be more interoperable.

We're still so, so, so early.

People think we've been going at this now for three years

and that we've got it all figured out.

We are in the first innings of a seven game world series.

You know, if you think about the maturity of the public cloud

20 years after it was launched,

that is probably a good rubric to think about

where we are on the maturity curve of AI.

So three years into a 20 year journey.

That's a bad ride, I think.

Second, this shift isn't just technical.

It's operational.

It's a human.

And Jamie's hot take is that the workforce has to get proactive

and AI literate to help run what's coming.

In terms of where things are headed from data center software,

one of my real hot buttons is with the workforce.

So in other words, we've created a workforce

in many cases in the IT industry

that it tends to be a bit more reactive.

And now we need to create the workforce of the future

and the workforce of the future needs to be a much more

proactive workforce.

And to do that, they need to be AI literate.

So I'm a really strong proponent

that we need to create an AI literate IT workforce

so that we can then support the evolution

of the technology that we're all supporting.

In a nutshell, the data center software stack

is becoming less like a set of tools

and more like a coordinated system,

spanning design operations and outcomes.

And AI is accelerating the timeline

for getting that coordination right.

That is all for today.

Make sure you like and subscribe wherever you listen

or watch to catch the latest episodes in our series.

And until then, we will see you next time.

Bye.