How Much of the World's Knowledge is Readily Available on the Internet?

I figure it's about 2%. This is an astonishing number to me -- meaning knowledge available right now to any human with an internet connection.

Though my estimate was a complete shock to a young nephew who believed that ALL human knowledge was available on the Internet.

What's your estimate?

What counts as knowledge?

Does it include all thoughts people have? If so, including people who are dead?

unfortunately an unknown amount of knowledge though history has been lost?
for example. how exactly were those giant megalithic stones, building blocks, moved? some weighing an estimated 1000 to 1600 tons.

As a percentage of living human knowledge, as in common knowledge rather than things like personal history, I’d guess about 5% give or take depending on how deeply you look, is readily available. For all human knowledge on the internet, which may require some intense digging, that might come closer to 50%, but likely significantly less.

The categories of things are varied. If you include some categories, the estimate gets larger, others and it gets smaller.

If I include the placement of chairs in this room, the internet will know absolutely nothing. The same with the actual shape of the river.

Is this “human knowledge” or “natural knowledge”, though?

Words can only contain so much—in the end they become misleading.

How are you folks coming up with those percentages?

Is that the amount that has been loaded or entered somewhere?

I'm with the crowd, very little of it is. No guess of a number. I'd label a lot of what the net does know as first level information. "A goat is a ruminant" Factual, surface knowledge. "Goats are hard to keep fenced in."  Surface.  Goat motivations, the motivations of a particular goat, what about abnormal goats? (Gotta admit, most goats are abnormal!!)
Surface, first level info it has. "Knowledge" no.

Yep. I maintain that every description of reality is a simplification, and a lot of the information online is a description of a description of reality, if not even farther removed from the real thing. Even the simple-seeming stuff, where you can probably find a million "how-to" videos or articles online in about five seconds. Even if you read or watched all of them, I'm willing to bet that there's a lot you still won't know about the subject, because it's too subtle to be put into words, or because the only way to that knowledge is to do it for a few years, or because just a few people in the world have realized that particular thing and they're not the ones writing online tutorials...

Throughout much of time, certain knowledge was a prized secret passed down from one person to another. Huge amounts of that knowledge was lost when the next generation wasn't there to learn it from childhood, often because it wasn't seen as significant in a changing world.

It's interesting what archeologists are discovering with modern tools, only to have the discoveries raise more questions, rather than give answers.

Christopher Weeks wrote:What counts as knowledge?

Does it include all thoughts people have? If so, including people who are dead?

Phew, thorny question. But fair. If you don't ask the right question you don't get the right answer.

I rather like the definition of knowledge at oxfordlearnersdictionaries.com: "the information, understanding and skills that you gain through education or experience."

I like to think of knowledge as curated information and skills that expand and enhance understanding and inform useful or meaningful actions.

Knowledge has more "signal" than "noise." A snarky Reddit post that does not add value or advance human understanding qualifies as "noise" to me.

Raw data is not knowledge, but rather is raw material that can be converted into knowledge.

Dead people still speak wisdom and insight. The entirety of gutenberg.org surely has more signal than noise.

How's that?

I just read that about 10-15% of the world's texts have been digitized, and not all of that is available to the general public. Some requires pay, such as some scientific journals, as an example.
Also, there is much knowledge that is not in text, it's in people's heads.

It may seem like the internet is infinite, but I wonder how much is redundant? You only need one web page to explain how the heart pumps blood, but I bet there are thousands or millions of attempts, as parts of larger learning materials for different audiences.

A bit of a tangent here, but something I have always wondered is just how MUCH information was lost in the as-of-yet never satisfactorally explained burning and subsequent loss of numerous books in the library of Alexandria. Then you have whole other compiled sources of information like the tens of thousands of clay tablets amassed by Ashurbanipal of Assyria in his time. And this is just written down knowledge, no other medium included.

I have absolutely no idea what percentage of all the world's knowledge is actually available on the internet, much less readily available. To tell you the truth I struggle to ever stumble back upon some of the articles, tutorials, how-to sequences, etc that I once worked through just a decade or decade and a half ago. It just seemingly isn't there anymore (and some of it may literally not be on the regular internet any longer, if a server went down or websites expired, etc).

Interesting topic. Huge topic really, haha. And how much is lost over and over and over again each time whole chunks of the world put themselves through yet another onslaught of war, only to lose whole waves of people spanning multiple generations, and if enough of the "right" people are lost each time, people possessing especially more keen or observant minds, or just possessing certain skillsets deemed more "desirable" for their particular time, only to never be rediscovered or happened back onto in one or even a few lifetimes?

Cody Hahn wrote: To tell you the truth I struggle to ever stumble back upon some of the articles, tutorials, how-to sequences, etc that I once worked through just a decade or decade and a half ago. It just seemingly isn't there anymore (and some of it may literally not be on the regular internet any longer, if a server went down or websites expired, etc).  

Me too. I get this weird feeling that the amount of readily available knowledge online is actually decreasing.

Cody Hahn wrote: I have absolutely no idea what percentage of all the world's knowledge is actually available on the internet, much less readily available. To tell you the truth I struggle to ever stumble back upon some of the articles, tutorials, how-to sequences, etc that I once worked through just a decade or decade and a half ago. It just seemingly isn't there anymore (and some of it may literally not be on the regular internet any longer, if a server went down or websites expired, etc).

On this topic I spent about half an hour on the build it solar site recently and found that the majority of the links have been dead for over a decade... All great resources, around half still thankfully available due to the wayback machine. Now that's one site that would remove a huge chunk of knowledge on the internet if it was taken down...

Me too. I get have this weird feeling that the amount of readily available knowledge online is actually decreasing.

Which is more than a little depressing because we should in many ways have it better than ever before, and I sure hope we haven't already hit some sort of sweet spot as far as compiled information and reasonably easy and ready access to said information by the masses (those whom are interested in the first place, that is). Surely not already....................haha.

*This post was edited to clarify that the top/first sentence in this post is actually Douglass'. I botched the "quote" function when trying to reply to his sentence such that it looks like I wrote that sentence, when in fact that is Douglass' sentence, and that's all there is to it.............LOL.

Alex Howell wrote:On this topic I spent about half an hour on the <a href="https://www.builditsolar.com/index.htm">build it solar site</a> recently and found that the majority of the links have been dead for over a decade... All great resources, around half still thankfully available due to the <a href="https://web.archive.org/">wayback machine</a>. Now that's one site that would remove a huge chunk of knowledge on the internet if it was taken down...

I have had similar experiences on the WayBack Machine, and I too am grateful for what can still be reaccessed on that thing from time to time.

Douglas Alpenstock wrote:Me too. I get this weird feeling that the amount of readily available knowledge online is actually decreasing.

I think (hope!) that's not really true. And I think that Google is primarily responsible for that sensation by completely screwing the pooch on search in a quest for revenue. Others are certainly involved, but Google is the 800-lb gorilla.

If you take a page with instructions for screwing a machine screw into a nut. Even if they're thorough -- "righty tighty", helical inclined planes interlocking, hold each bit between thumb and forefinger, slowly engage while turning, etc. Even if there's some page with all of that spelled out. What about that feeling you get when the threads engage or the other ones that happen as you seat the nut and start to get tight and then also when you overtighten? I think that's knowledge that either doesn't lend itself to the media at which the internet excels or we're too lazy to convert it to the written word because it would be a lot of work. And that's just one incredibly simple operation.

I'm a software developer. There's a giant tidal wave of information about how to do most particular tasks in my field available on the net. Less so about when. Less yet about why.

I guess I'm poking at the difference between knowledge and wisdom and I don't know if they're different things or what.

I'm going to hard disagree here...

Accessible knowledge is unfathomably vast now - bewilderingly so. Sure, accessing deep knowledge requires some skills and dedication to break through the surface noise. But once you do you can learn more, more quickly, than at any time in prior human history.

Thinking back to my own life - I learned beekeeping as a teenager back in 1995. At that point in time learning was very much dusty old tomes, or master-student relationships. There were beginning to be more accessible "backyard beekeeping" type books but you were utterly dependent on someone standing alongside you and showing you the way. The internet has transformed that utterly - every single possible facet of beekeeping, from the most basic to the most detailed and arcane is now out there. You have the full richness of conflicting opinions being discussed and evaluated, and many of the old myths that were passed down orally can be debunked. And those dusty old tomes are available on there as well... The learning process has been completely disrupted.

This is just one field that I am personally intimately familiar with - it seems to be repeated all over.

As a parent, I'm currently watching - in some position between awe and mild apprehension - my 13-year-old's progress in learning computer coding. He has been able to self-teach in the space of a handful of months enough code that he is writing his own games, messing around with the basics of AI, has a functional understanding of computer graphics, including using and applying ray tracing, made a key logger for his laptop... I'm clinging on for dear life and trying to keep up. I grew up at a time when home computers were just becoming established, and would have killed for the access to knowledge that he has now. My biggest challenge is to make sure he stays on the ethical side of life!

I wonder if you're really disagreeing. I think there's an unprecedentedly astounding amount of knowledge available on the internet. And I also think it's a small(ish) proportion of all the knowledge there is.

Perhaps our disagreement is about what you consider to be knowledge?

I share some of my son's obsessively inquisitive traits. I can't recall a topic in recent years where I haven't been able to go down the rabbit hole a scratch that itch to investigate and learn.

I think this graphic is helpful.



I would suggest that the vast majority of the area of that diagram is readily accessible on the internet. Of course, you might need years of study to recognise it and understand it, but it is likely there. The central core area is very well represented.

So if we are talking about what is not accessible via the internet, I'd ask more explicitly - what knowledge do you mean?

That illustration is neat.

Michael Cox wrote:So if we are talking about what is not accessible via the internet, I'd ask more explicitly - what knowledge do you mean?

The example I used above is:

...that feeling you get when the threads engage or the other ones that happen as you seat the nut and start to get tight and then also when you overtighten...

There is knowledge that you only get by doing a thing. I can read someone's account of eating wildish potatoes and the feeling on their tongue and in their gut from the solalanines, but I don't really know what that's like until I've done it a few times. Or what it's like to fall in love. Sure, there are a million testimonials, stories, poems, songs, and movies about that and none of them are knowing what it feels like.

A while back I put my name into a background checker. I was surprised to see an exceptionally long list of past addresses. I finally realized that many of those were motels I had stayed at during my travels. With that observation, I find myself thinking about accurate information, useable knowledge, etc.

I wanted to say 2 percent like topic starter. Just like the Epstein files btw ( it's a joke guys, don't flip).
Guess we'll never know. But i'm having a discussion with a friend who got really arsy about a topic he 'knew' everything about because the wholiness AI said a thing, so then it must be true, despite my experience of twenty year rendering. But i'm right. Cement does bond on cement. Not perfectly in many cases and it will fall off, but it can last 50 years or more.lime can fall of earlier.The AI grazes the internet and finds all drama stories people write about and not all the cases in which it went ok and concludes ok it can work.

There is so much information I can grab in 5 seconds, it has been both transformational and bloody scary.

That information can also disappear in 5 seconds.

Libraries are getting rid of paper books, only to discover that electronic books require a yearly subscription that costs them more than managing their paper book collection.

Yes, paper books can burn, or get flooded, but electrons can vanish so quickly, or worse, be corrupted in 1 minute. I love the internet, I love permies, but both are a little scary to this really old dinosaur who's glad to have a paper copy of Edible Forest Gardens, Webster's Dictionary, Rodale's Encyclopedia of Organic Gardening and a few other classics, sitting within view on my bookshelf!

I guess you start with what percent of the world’s knowledge is in print.  I would guess less than 1%.  Then how much of that is somewhere online.  I would guess 1% of that 1%.  Then how much would you find with a google or other searches.  I would guess 1% of that 1% of that 1%.  I’ve done in depth research on many topics.  You go to the library and find all the books that fit your interest.  Then you read those.  Then you write down all the references in the back that seem to apply.  Then you ask the librarian to get a few of those, because you have limited time with them.  Rinse and repeat until you have your answers or until the answer is that no one has published the answer.

A lot of the worlds knowledge has been save in books.

How many of these books have been saved to the internet?

Anne Miller wrote:A lot of the worlds knowledge has been save in books.

How many of these books have been saved to the internet?

I would have guessed more than half.  But a quick google show 10% for the library of congress is scanned as of today and I assume it is ahead of most others.  Big libraries have been making a major push to get the stuff scanned at least.  Most are not text searchable yet but only scanned pictures.

Now I agree with Jay's comment about about things disappearing and being changed etc.  I am beginning to think I need storage for everything I view so I can actually go back to what I read.  And that would mean I need my own AI cataloging things too.

I would guess there is more knowledge there than shows though.  But much of it is not readily available for various reason.  

We can all probably list thing from farm or homestead stuff that many people do not know.  For example say I talked about working with a bar digging post holes and laying it down on the ground and suddenly 10 minutes later it is too hot to pick up.  Guessing maybe 10% of the audience here is thinking something along the lines of "hey stupid, stab it in the ground pointed at the sun or pointed where the sun will be"  But betting for most that is a new process.  It works because you only leave a few square inches to absorb the sun's heat while maximizing it radiative surface area, so basically it air cools to air temperature.  Where is that actually going to show in a search function?  AI probably improves the chances of finding that one but likely still not easy yet for some it is common knowledge.

Pay walls for access to much of the internet is another problem

Information overload with potentially bad information is another part of the problem.

Another problem is algorithms not showing needed information.  For example 10 or 15 years ago I watched a solar water heater video by  Ben Gravely that made a whole lot of sense.  Found it a few times while dreaming back then and then suddenly couldn't find it for many years.  Assumed it had been removed.  Then about a year ago I stumbled into it again.  Information wise it was actually better than what I remembered.  But for that whole time in between poorer but newer or more popular information had kept it hidden.

My controversial take on this is that knowledge without action is almost meaningless. What good is becoming a vet if you never see any animals? Research rocket mass heaters and never build one? Learn electrical generation and never build a generator?

I love learning and quit a ton of good jobs because I stopped learning and got bored. But knowledge without action or skill is just plain self-defeating for society and for individuals. It even has a name: the Duncan Krueger effect. Basically it states that the more intelligent a person is the less THEY feel they know. Why? Because as much as they know, they also know they have so much more to learn. And even if they know the topic they know others are more skilled then they are in executing what needs to be done.

If you need a permie example look up YouTube on rocket mass heaters. AI has latched onto them and there is a ton of informational videos about them, but I also noticed they are getting vital information wrong. What is happening is AI is feeding on itself and slowly convincing itself that the poor information is correct.


Another example? I needed a cover for my latest novel. I got the same woman in different configurations, but 80 percent of the pictures I got were of that one woman. Hair a different color. Mirror posing of her etc, but it was mostly her. I mean hundreds of pictures but 80 percent was of her.


We are not gaining knowledge we are getting lazier that is all. Why learn a new skill which takes time and effort when we can pseudo-learn from the internet and then argue with one another on finer points so it seems we are more knowledgeable than we are?

But even with politics it really is not about division. The former ultra conservative coworker was just like my ultra progressive mother in law: they both spent hours on news, podcasts and the internet to find that strange angle so both could tell me, “see you don’t know this but”… it was not about the information it was about them knowing some unique slant that no one else knew about. I am fine not knowing. I’m glad, it’s why I don’t have a TV in this house.

The secret sauce is SKILLS. That is knowledge and proficiency personified.

I can say this with authority because I am a Gert.

Books written between 1923 and 1990s, maybe even some recently have not been ebooks or digitized due to copyrights.

To date, we have removed over 500,000 books from lending on archive.org

https://blog.archive.org/2024/09/21/lending-of-digitized-books/

2% seems generous if we look at it from a historical point of view.

At our university archives, the staff said about 0.01% of the collection at any university is available online, and usually hidden behind a pay wall. That was a few years back.

Talking with a librarian who works in this area 2 years ago, it's probably up to 0.08% now.  

If we look historically, at the less than 5% of documents that survived time before standardized type, maybe 0.01 to 0.05% of that are online (private collections don't tend to digitized and university and other public archives have a very small fraction of what's available).

If we are looking at information of say 1800 to 1990, maybe 20%.  Maybe less.  The physical copies lasted better.  But it requires perceived value for the efforts of the Gutenberg project or the like to digitize it.  And a lot of copyright issues removed much of that catalog.

Post 1990 knowledge, it is estimated about a quarter to half (again, citing librarian and what I learned in university) of what humanity has produced and discovered during that time period is online.  Most behind login or pay walls.

From a university point of view, to have knowledge that no one else has, is a great income source.  To look at some of these documents, we had to sign an agreement we wouldn't take images or share the information within the document outside the university.   But if someone were to pay for research access, their use rights would be different.

r ransom wrote:2% seems generous if we look at it from a historical point of view.  

Excellent and illuminating post! Thanks for taking the time.

I did an AI google search with this result;
'While the internet hosts an immense amount of data—estimated at over 64 zettabytes in 2020—only a small, hard-to-quantify fraction of total human knowledge is readily available online.
Estimates suggest that only about 15% of knowledge published in the last 200 years is available online, with the majority of human experience, physical archives, and historical documents still existing in offline, physical formats.
A significant portion of human knowledge exists in books, archives, and personal experiences not yet digitized.
Some estimates suggest over 90% of real knowledge remains in physical form.'

A friend owes a favour so is giving me unrestricted access to the NSA's database- Ima take off next week to read the internet and will get back with what I figure out...