Are websites getting "fat"? Page weight, HTML size & Googlebot limits explained

Hello and welcome to a new episode of Search of the Record, a podcast coming to you from

the Google search team.

I am Gary today and I think I'm joined by our Martin, who I call Morty.

Hello.

Hello.

Yes.

I'm here.

Morty is here.

I don't know Rick.

I don't know.

Why do I call you Morty?

I call you Morty, but I don't know why.

I do not remember.

I think at some point you started singing Otanbaum, but with like, O Mortimer, O Mortimer.

We blows in time.

Nuh.

Oh, again.

Oh.

So nice.

You're not here to talk about my eye color, though, I think.

I mean, we could talk about it for half an hour, for sure, but I think not.

I'm afraid that while our customers would be very interested in that topic, I am not.

So I don't have a topic.

So I looked at what you prepared.

Yeah.

And it's weird.

You don't like my topic, I know.

So you coming in hot with a question, looking at the notes now.

The first question is, are websites getting fat?

Yeah.

I think that's a reasonable question.

Okay.

I think this question is not even meaningful.

Ooh.

Look at you.

Why is it not meaningful?

Because it does not matter in the context of a website if it's fat.

Why?

In the context of a single page.

Yes.

Oh, nice.

But in the context of a website, it really doesn't matter.

Ah, that's an interesting one.

Cool.

Okay.

So we can talk about either things or both things.

We'll see if time permits.

Yeah.

We can talk about websites.

And we can talk about web pages.

Do you want to do that?

So I think first we step back a little on another word.

And that would be fat.

Okay.

Fine.

Yeah.

That's a bit.

Okay.

Let's start again.

Are websites or web pages getting fat?

In that context.

That's a really nice one because I struggle with it.

And I kind of posed it in a stupid and not great way.

And I used fat for that because heavy, big, large, I don't know which term to use because

I realized recently when I asked on Blue Sky that people are understanding different things

when it comes to call it page size.

Okay.

What is page size?

It's a surprisingly difficult answer.

Well, an individual page size can be defined.

Yeah.

Yeah.

Because the raw bytes, that's easy.

But that's not what everyone talks about when they mean size.

That's what some people talk about when they mean size.

Right.

And that's the rabbit hole I fell into.

So what you are thinking here when you say fat is basically the raw bytes from the

parent HTML, plus the resources that need to be brought in, plus an immediate.

Maybe.

Okay.

I started at a slightly different position where I asked myself like, so back in the days

when I started making websites, okay, I had dial up internet so I couldn't be online

all the time.

So some of that happened on my hard drives locally, with a file and a text editor, then

I needed to transfer that between the computer that was actually hooked up to the internet

and my laptop because sometimes I was working on my laptop and I had a really old, really,

really old laptop.

It didn't even have USB.

It sounds super stupid, but the only reasonable way I could transfer data was with floppy

disks.

Okay.

And I realized that some I was living in the same era.

So yeah, I distinctly remember this.

Yeah.

But like John, for instance, also lived or lived in the same era and he is like, who thinks

of floppy disks.

So you know, I thought I explained I'm pretty sure our boss John Miller was born roughly

when the furrows were still running around.

But anyway, he was probably here when electricity was invented, yes, but hi, John, if

you're listening, but I realized that I usually, even if I built like relatively elaborate

things, except for if I had to transfer images, then the floppy was problematic.

But otherwise with just HTML and maybe some CSS and maybe even some JavaScript, it worked

fine.

And a floppy disk would fit more or less the whole website minus images, which was fine

because images sometimes back in the days, I had no notion of copyright.

So I just like hot linked effectively to images of other websites.

Yeah.

So it wasn't really like transfer size that I had to deal with all they needed web space

for.

But then I read the 2025 web almanac, which is great.

We're going to link that in the description as well.

It's a great look at how the web is changing and how websites are changing.

And they found that in 2015, the median mobile home page was 845 kilobytes.

And that's the data that needed to be transferred over the network into the browser onto someone's

disk.

That's how they define page weight.

And that is just a parent page.

That's just a home page.

That's just like they are only looking at home pages, I think, from the crawl.

Okay.

But that's just a parent page.

It's not the resources.

Yeah.

See, that's where I'm not so clear about the definition of page weight.

That's a really interesting.

They have a paragraph where they are trying to like explain what they mean by page weight.

That's why I used fat to just kind of like blatantly call out that I don't understand the differences

in what these things are.

So they say page weight also called page size is the total volume of data measured in

kilobytes or megabytes that a user must download to view a specific page.

In my book, that includes images and whatnot, because I have to download that to see.

And that's why I was surprised to hear that 2015, that was 845 kilobytes.

That to me was surprising.

Why?

Because I would have assumed that with images, it would be more than 800 kilobytes.

Ah, I see what you mean.

Right?

Yeah.

But anyway, 2015, 11 years or longer depending on when you hear this, 845 kilobytes

July 2025, the same median page is now 2.3 megabytes.

Holy, that's a three times growth.

Yeah, and it goes roughly from a little more than half a floppy disk to almost two.

It's almost two floppy disks, almost two.

Yeah, floppy disk was able to norm and the normal circumstances 1.44 megabytes, I think.

That surprised me because I knew it was growing.

I knew we were also doing more complicated things.

I mean, obviously, if you like basically build AutoCAD, like a software that does computer

aided design in the browser, then yeah, that's not going to work in 800 kilobytes.

Yeah.

Wow.

But then again, on the median, I would assume that a lot of it is just websites and that

is quite big.

Yeah.

When you ask people what they think if this is big or not, you start getting very different

answers depending on how they think about page size.

And there is no one true definition of it.

So I think based on what you said, that we probably need to do a better job at explaining

what page size is.

And if you look at our documentation for about crawlers, I think about Googlebot, but I

could be wrong.

We say that by default, we fetch 15 megabytes of the content or of the raw bytes from a specific

URL and then we stop.

And that's per URL.

So basically, if you reference stuff in your HTML, then those have their own 15 megabyte

limit.

Right?

Yes.

And that, to me, from a search perspective, not from a user's perspective, that makes more

sense.

From a user's perspective, it probably doesn't make sense because, well, they don't care.

Yeah.

Like, why would they care about it?

It's like, what do I need to download?

I need this document full stop.

Yeah.

I mentioned that we could propose a change in the documentation for the, should you be

archive for the web amannac to better explain what page weight is.

I mean, I wouldn't change their documentation because for their purposes and intents, I think

they're explaining what it is and they're defining it rather well.

I, I'm just surprised that with all the resources that go onto my disk at the end, it used

to be only 800 kilobytes, 800 something kilobytes in, in 2015.

Sure.

But I don't think that the page weight or the page size or whatever, let's phrase it

differently.

When I think about page size, depending on the context, think about different things.

Yeah.

And I'll realize that when I asked this question publicly, that different people had very

different notions of how they understood page size.

Depending on the layer you're looking at, it gets confusing as well because there's

also compression.

Ah.

So some people are like, ah, but this website downloads 10 megabytes onto my disk and I'm

like, yes.

And they're like, what?

The internet is slow here and I'm like, but maybe if you look at what actually goes over

the wire, you might find that this is five or six megabytes, not the whole 10 megabytes

because you can compress things on the network level and then you decompress them on the

client side level, which again, if you are strapped for space on your phone, for instance,

which in the past, my phones typically, because I took so many pictures and videos and stuff

and I was really, really bad at deleting things.

I'm kind of like a digital hoarder sometimes, especially when it comes to my phone.

I ran out of disk space as simple as that.

And so like, you want me to install an app, no, this app is like seven megabytes.

I don't have seven megabytes.

Yeah.

Compression is nice for reducing the transfer time, but I don't care as much.

I care more about the space on my phone that it actually occupies.

So I have a different angle on this, depending on what I'm looking for myself.

And then it gets worse.

Have you looked at the HTML version of the HTML standard?

No.

It's funky.

I never really fully understood why they have a one page version and they have a multi-page

version.

OK.

But if you look at what you have on your disk once you download the single page version

of the HTML specification, you end up with 1414 megabytes of HTML.

That seems reasonable.

It's text with a bit of extra sprinkles on top.

That's huge.

And it's not even a fancy website.

Like, no, I think you are talking about the what working group, the whatwg.org or something,

whatwg.

Right?

Yes.

Like, if you look at the spec, the HTML spec on whatwg, it is not a fancy website.

It is really just text with some default styling, nothing else.

So yeah, and like 14 megabytes, that really means raw content, basically.

That is wild.

And I'm not sure if they have images in there.

I don't think they do that.

Maybe they have like some explanatory diagrams or something, I don't know.

But 14 megabytes, that is a lot.

And I know that I've seen websites out in the wild that are quite heavy, just the HTML

that gets transferred as well.

And I know some extreme cases, and I've heard of extreme cases through the blue sky question

that I asked and the blue flies, where people are inlining images.

So you can turn an image into effectively a very long string of characters and put that

straight into your HTML, quote unquote, to avoid having to do another network request

and fetch that image.

So you're embedding images and stuff into the HTML, and then obviously end up with like

a 50 megabyte HTML file.

So while you were talking, I found the single page version of HTML living standard.

And when you finished your sentence, that's when I clicked the link to one page version,

and it just finished loading.

So basically that was like what, about 45 seconds?

Yeah.

More or less?

Yeah, yeah.

And I'm on a fast connection.

So it gets wilder.

If you use print to a PDF in Chrome, you get a 96 megabyte file, I think, or I got one.

And then if you look at the PDF version that they offer, it's, I think, 15 megabytes.

Oh.

Wild.

I mean, different kind of compression.

Yeah, yeah.

I think it's interesting to see like the PDF version of web pages, oftentimes are either

a lot smaller or a lot larger.

And this one is like, you can tell like this is really dense content.

Do you think stuff like the reader mode alleviate anything on the user side?

Ooh.

Hey, don't really know what the reader mode, I know what it does from a user's perspective,

but I don't know what it does in the background.

I was thinking about it a little bit a while ago, and my conclusion was that it doesn't

change much because like from a loading perspective at least, because you still have to first

get the resources.

And then the phone or the app does some magic locally to remove fluff and present you with

a more readable version of the page, but at least makes the internet kind of sort of more

bearable to enjoy.

Yeah.

Wow.

I used two weird words together.

But like trying to think about it from like holistically, do you think that this even matters?

Like what's the weight of a web page or a website is, which we still haven't identified

by the way.

Correct.

Yes.

Because like, for example, for me at home, I have a 10 gigabit connection.

Yeah.

I have a 10 gigabit fiber here as well.

Yeah.

And I don't feel the pain.

Like for example, if something is, I don't know, what did you say, like 96 megabytes

or whatever, for a PDF file, it will still come down in relatively short period of time.

And at that point, it's dependent on the server that's sending the stuff not on my connection.

Yeah.

So to me, it really doesn't matter that websites are getting bloated.

On my phone also, I have a permanent 5G from Swisscom, my phone company.

So again, it is super fast.

It really doesn't matter.

It just comes down like that in a snap.

Oh, that's nice.

Oh, sweet summer child.

I'm so happy for you.

Yeah.

Right.

But like I live in a bubble, right?

Yes.

But I do travel a lot.

And then if I go to less developed countries, for example, then suddenly it becomes problematic

because like the cell phone towers would advertise that they have 5G.

But in reality, they have like a really weird, I don't know, 3G or something.

I don't even, not even 3G.

Yeah, I guess it depends on where you are living and what kind of connection you have as well.

It's wild.

I remember when we went diving in Antarctica, we were on the ship where we had a meter satellite

connection.

And I believe if I'm not mistaken, what did I pay for 100 megabytes?

100 megabytes, which again, the spec for HTML is 15.

Yeah.

So just to keep that in mind, I think like $20 or something.

Yeah, yeah.

So it was expensive, yeah.

And that made me think about things like I activated data saver mode on my phone.

I set the connection to a meter.

I did not open things like most of the social media apps I just never opened because they

would suck this 100 megabyte dry in minutes.

So I think it still matters.

But it begs the question at what level is what kind of weight acceptable?

Right.

And I don't have a good answer to that.

And then on Blue Sky, I got into this conversation about HTML size.

So in this case, there's a document that is 15 megabytes.

And we both agree that pretty much most of these 15 megabytes are actually useful content.

Okay.

Because there is not much in terms of images or style sheets or whatever.

It's just the pure HTML and it's minimal markup around maximum content.

Yeah.

And then people were like, yeah, well, in that case, it's kind of okay to have 15 megabytes

of HTML that you need to deal with.

And we both know that our browsers, even on our relatively latest technology kind of laptops

and fast internet connections take a while to process that 15 megabytes of HTML.

Actually, I wonder how much goes over the wire because I'm pretty sure this compression

involved.

Oh, God.

Okay.

I tried to also open this page in reader mode and that kind of crashed the browser tab.

Great.

In this case, it's kind of acceptable.

But then what if the markup is only overhead?

And I mean, like, what do you mean?

It's like, well, you know, if it's like five megabytes, but it's only very little content

is that bad.

Is that worse as in this case, the 15 megabytes?

And I'm like, that's tricky because then we come into this weird territory of the ratio

between content and markup.

Yeah.

And I said, well, but what if a lot of it is markup that is metadata for some third party

tool or for some service or for regulatory reasons or licensing reasons or whatever.

Yeah.

Then that's useful content, but not necessarily for the end user, but you still kind of have

to have it.

It would be weird to say that that is worse.

Yeah.

Then the page where the weight is mostly content.

I know.

You know, this is interesting because like historically, I have some beef with structured

data, for example, because I mean, this is a very long thing or long grinding thing.

At one point, early in Google's life, Sergey Brin, one of our funders, said that computers

should be able to figure out anything from the text that they received.

So site owners slash back then webmasters wouldn't need to provide anything extra.

And the context was something around spam.

Anyway.

And then structurally that comes in and structurally that is specifically not for users.

That is specifically for machines.

And depending how much structured data you add to a page, it can increase the blood considerably.

Like if you look at in our documentation, like what kind of structured data Google supports,

it's a lot.

Yeah.

Like you can fill a page easily with just the links to the documentation.

And then if you keep piling on structured data on your pages, basically stuff that users

will never actually see them, what?

Like is that good?

Is that bad?

For the site, it's probably good or it can be good because it can bring in additional

traffic through whatever magical stuff search engines might be able to show based on the

structured data.

But for the user, I don't know.

Yeah.

Well, this begs the question.

And it came up in discussions that the ITF, ITF being the internet-engineering task force,

basically, who created the standards for the internet, whether we should think about

separating, like holistically in general, separating metadata stuff from user visible content.

Like what if we would have something like for clients, HTTP clients that you trust,

you would expose an API sort of end point on your site where you just give the metadata

that they would want, right?

Yeah.

Like in an ideal world, that is not the most insane idea.

Like for example, if you had the HTML standard, like all the text for an LLM is irrelevant.

Like it will process the stuff differently than you and I.

True.

Can you read?

I can read sometimes.

You can read, right?

Okay.

Just checking.

And you could have something like append parameters, like question mark output equals

JSONLD.

Mm-hmm.

Let's say.

Yeah.

And then if the client is trusted, say through cryptographic authentication or something like

that, then you would serve them the machine, purely machine readable version, even in binary

format, doesn't matter.

And then unfortunately, this is a utopistic thing, because not everyone on the internet

is playing nice.

That's true.

We know how much spam we have to deal with on our blog.

We say somewhere that we catch like 40 billion URLs per day, that's spam, or some insane

number.

I don't remember exactly, but it's some insane number and definitely billions.

Will that just exuberate the amount of spam that search engines receive and other machines

receive?

Maybe.

Like I would bet $1.05 that will actually increase the amount of spam that search engines

and elements and others ingest.

So yeah.

Yeah.

And I think that's an interesting, like there was this principle that was quite popular in

the early 2000s, early 2010s, hypermedia as the driver of application state or something

like that, where the idea is to kind of like have separate URLs with different representations

depending on who consumes it.

Yeah.

And I think generally that makes sense.

And you can do this to whatever degree, but I'm not seeing it happening as much anymore.

I don't know how much sense it makes, because one of the things that we had when

what is it called mobile first in mobile first indexing.

One of the things that came out of that was lots of documentation based on stuff that

we observed when we were trying to switch over sites.

And one of the actually do we still have the documentation for mobile first indexing

that might actually have this still if it's there.

But one of the things that we noticed is that there's no parity between the mobile version

of a website and the desktop version of a website if there's different URLs for the

two.

So mobile would be like martenshp.com slash m slash content.

And then the desktop version would be martenshp.com slash content.

And for a very large number of pages that we observed, there was no parity between the

content or no exact parity.

Content was missing.

That was the worst because then you're not going to rank for the stuff that your desktop

version was ranking before.

And then we had outreach campaigns about it.

But then links were also missing.

To me, that's harder to explain.

That's bad.

And the navigation was different, the whatever was different metadata was missing.

Atreflang was missing.

Link elements were missing.

And yeah, it was a pain in the lower back.

Lower back.

Thank you.

Lisa or producer makes me really, really good stuff.

And yeah, so even for trivial stuff like a mobile version and the desktop version of

a web page served under different URLs, even for trivial stuff like that, there can be

large differences.

Yeah.

So wait, I want to go back to the first question.

Okay.

The question that got us here, basically in this really, really deep and dirty, dirty hole

or a website's getting fat.

I think if we rephrase it to our web pages getting larger, I think that is true.

Yeah.

The answer is yes.

Is that a problem?

I think that depends on the context because we identified that some people are constrained

by either slow internet connections and that can hit us here in Switzerland as well.

I know in some parking garage, I have very bad reception.

Oh, yeah.

You are in an atomic bunker and then you don't have reception.

It's like, oh, I will cry your river Martin.

Correct.

I was very surprised and not very happy with it.

Correct.

So internet connection speed is still playing a role.

And I think in the last couple of years, the increase in transferred by size has outgrown

the increase in transfer speeds.

If you look at the median transfer speed increase for mobile connections, I think so.

As a hypothesis, I would have to check the data, but I feel like that's the case.

The other thing is there are mechanisms to alleviate this like caching.

So of course, you get the full painful hits in the first place, the first time you visit

the website.

But for instance, the HTML specification uses caching.

So when I reload, I don't have to like download all the 15 megabytes again.

Compression on the transport level on the network level helps as well because then you

have less data that you need to transfer.

You still have as much data to store on the user's device or on the crawler's end.

So yeah, that's only helping you so far, but there are ways to make this less painful.

There's also this lazy loading thing where you can say, like, okay, so there's a lot

of heavy content, like lots of images, for instance, right?

You're only loading those that you actually scroll to or go to and possibly see rather

than loading all of them in the first place.

But I still think it's worthwhile thinking about how much data are we throwing around.

Yeah.

But the website versus web page distinction.

Do you think or do you agree with me now that the original question is meaningless?

Yes.

Okay.

That's good.

You caught me there.

That's a very good catch.

Okay.

Well, it's Friday today.

So this just made my day because someone finally agreed with me.

Hey.

Do you think that we need to do anything to reduce the size of pages on the internet?

That's a tricky one.

And I mean you and I as people who like the internet and work on standards and stuff.

I think yes, I think we are wasting a lot of resources and I mean we had that in another

episode where we said that we know that there are studies that show that websites that are

faster have better retention and better conversion rates.

And speed is in part also based on size because the more data I ship, the longer it takes

for the network to actually transfer that data and the longer it takes for the processor

of whatever device you're on to actually process it and display it to you.

So and again, like if storage is a problem, then you're excluding some people there as

well, I think yes, I think that makes sense.

I also think that not everyone is you and I myself know this.

I'm now building like a new website and I'm looking specifically for a workflow that makes

it easier for me to deal with images, for instance, because yeah, so far I've just been

like, here is this 59 megapixel image.

I know you're on a smartphone, but I don't care.

Like here is the five megabyte image, good luck, which is not the right way to do it.

May the odds ever be in your favor?

Yeah, exactly.

You know, for that specifically, internally we have a linter that prevents submission to

the developer sites that we are using for the search documentation if the image is over

one megabyte.

Yeah, interesting, arbitrary limit in my opinion, but yeah, well, I like to think that they

came up with that number using some methodology, not just like I woke up and chose violence.

Yes.

And like I was thinking about it a lot because it's like, should I ask for an exception?

Because our images often are like seven megabytes or so.

And then I was looking at the compressed versus the less compressed versions of the images.

And my non-blue eyes could barely tell the difference.

I know what you mean, yeah.

And at that point, on a big screen, and at that point, does it matter that we lost some

pixels or some shades in pixels?

Yeah, yeah.

Because our images are not fine art, right?

No.

For fine art, sure.

So what's our takeaways?

Our websites slash web pages getting fat.

Are they?

Yes.

I think pages yes.

Connections are getting faster.

So does it matter?

I think it still does.

Okay.

How do we fix it?

Maybe we talk about it in another episode.

Yeah.

Sounds like a plan.

Because there's a lot that you can do.

And I'm pretty sure that every little bit helps.

Not just with search engines.

And we shall talk about that a little bit later.

But also with your users.

Yes.

Users definitely do like snappy websites.

And bloaty pages don't help with that.

Agreed?

I agree.

Your honor.

You have to say your honor.

I agree your honor.

Thank you.

Objection.

Objects, and I'm not honorable.

All right.

Well, Martin, thank you for the chat.

Thanks a lot for you.

This was our second chat today, because we recorded two episodes.

And I will leave you for two days, three days,

because it's the weekend.

And on Monday, I'm not going to the office,

because I don't want to.

Yeah, especially if there's still snow.

So I would like you to remember this face and this voice

and have nightmares, please.

I thank you for the lovely conversation.

But thank you, nonetheless, for chatting with me.

Thanks.

If you like this episode, please like and subscribe,

wherever you get your podcasts.

And have a very nice day, please.

And thank you.

Bye-bye.

We've been having fun with these podcast episodes.

I hope you, the listener, have found them both entertaining

and insightful, too.

Feel free to drop us a note on Twitter at Google Search C,

or chat with us at one of the next events we go to,

if you have any thoughts.

And of course, don't forget to like and subscribe.

Thank you, and goodbye.