Friday, February 08, 2008

The accumulated amount of knowledge will saturate over time

The accumulated amount of knowledge will saturate over time. Not because there is no more knowledge to aquire for man kind, but because we will not be able to keep track of what knowledge is already known.

I recently read that 90% of all scientists that has ever lived, are alive and working today. That would mean that the amount of knowledge generated each year is enormous compared to the amount of knowledge generated 10 or 50 or 100 years ago. As we also know, it has been a while since anyone was able to hold all known knowledge in ones head. But today, it is even hard for computers to hold all this knowledge, or even the scientific community as a whole.

So far, I would guess that we are somehow able to obtain the knowledge we want to know, if known, before pursue the task of researching it. Either by doing some comprehensive literature search, or talking to colleagues in the community. However, as the amount of knowledge accumulates, it will be much harder to do this. Google Scholar and similar services helps the literature search by indexing the vast amounts of publications of scientific knowledge. However, if these services does know improve, and change dramatically, they will only suffice for so long. Namely, unless the way we handle and store knowledge change fundamentally, the amount of knowledge we can keep within reach can be just as large.

If we can not find and aquire the knowledge we are curious about, or need, by either searching through the literature, or asking colleagues, we therefore will be forced to research it ourselves. And if this knowledge is already known, we will not increase the amount of knowledge known to man, but just duplicate it - we will invent the wheel again. Therefore, even though it seems that the amount of knowledge being built by the scientific community, is increasing exponentially (following the size of the community increasing exponentially), it will just do so for a while, before saturating and everyone starts to duplicate all knowledge.

Hopefully this is no problem, as the amount of knowledge about data mining is also increasing exponentially, but I think it is still worth considering a thought.