In my previous post “Why You Need To Market Your Website” I mentioned the Web contains literally billions of web pages and millions more are added daily. Just how many is unclear.
Well a couple of days ago Google stated that their systems which process links on the web to find new content hit a milestone, recognising 1 trillion (as in 1,000,000,000,000) unique URLs! Note the word unique.
They go on to say “we found even more than 1 trillion individual links, but not all of them lead to unique web pages. Many pages have multiple URLs with exactly the same content or URLs that are auto-generated copies of each other. Even after removing those exact duplicates, we saw a trillion unique URLs, and the number of individual web pages out there is growing by several billion pages per day.”
What blows me away in particular is Google’s estimate that several billion pages each day are added. That’s just mind boggling & shows the growing level of competition.
To put this in context. The first Google index in 1998 had 26 million pages & by 2000 the Google index reached the one billion mark. In my previous post I mentioned a study released in 2000 estimated 7.3 million unique pages were being added daily. I went on to say that given the huge growth of the web since 2000 & boom in blogging it’s likely the figure is now very much higher. Well I was right, but I’m still astounded at Google’s estimate. (I don’t dispute it – I just have difficulty counting that high :-)).
So how many unique pages does the web really contain? Google themselves say “we don’t know & strictly speaking, the number of pages out there is infinite with things like web calendars may have a ‘next day’ link, and we could follow that link forever, each time finding a ‘new’ page. We’re not doing that, obviously, since there would be little benefit to you. But this example shows that the size of the web really depends on your definition of what’s a useful page, and there is no exact answer.”
Let’s just agree there’s a lot & the number’s growing. Which means ranking well in search engine results is going to be more important than ever and more challenging to achieve.
Mark Sceats
Mark is a Partner and Senior Consultant at SureFire which he founded back in 2002. Prior to establishing SureFire he worked for KPMG Consulting. Today Mark heads up SEO, embracing the challenges that can come with complex website implementations. Outside of work, his interests beyond his family are running, snowsports, diving and fishing (badly).
