Every day I am more amazed by what is going on in the digital world and how it’s changing.
In 10 years we have manged to create the most complex “thing” humans have ever built - which in that short but complex time has never broken down.
It spans the world, has millions of users, billions of connections, trillions of bits of information and almost all of the world depends on it for commerce, communication, entertainment etc. In my opinion it is without a doubt the most important and influential development in our history.
What I find even more incredible is that the majority of what we have built we can’t easily find and we have huge amounts of “Dark Data” lurking out there un-indexed. The Internet is growing at a stupendous rate as people become more accustomed to applications like Flickr, You Tube, Facebook, Myspace etc more data and information is being added into the system.
In October 2005 Eric Schmidt from Google said that, “…a recent report shows that there is about 5 million terabytes of information in the world”. Google at that time had only managed to index around 170 terabytes. If Google was to index the remaining 3.3 terabytes at the same rate it would take about 300 years, and that’s assuming that no one added any more to it. Since then the amount of information in the world has continued to grow exponentially. This massive increase is raising some interesting problems, questions and opportunities about who and how we index all of that data if it’s going to be of any use.
There are also questions being asked about if this “unbreakable system” we have, can actually cope with so much data and there is a risk that there will be so much that it generates something being coined as “The Exaflood” (a film title there i think). This is where there is so much data it clogs the whole system and it grinds to a halt on a global scale. Is this possible? I think that you only have to look at what happens to the UK connection speeds when the US comes online to get the answer.
In 2006 International Data Corp estimated that we would generate about 160 Exabytes of new data that year. That was 2 years ago, since then many more video sites have come online, along with growing economies like China and India. (India’s film industry is also bigger than Hollywood’s so there is plenty to go at film wise in terms of getting it online).
Now when I first heard the term Exabyte it did not mean that much to me so let me explain.
One exabyte is 1,000,000,000,000,000,000 bytes (dependant on whether you are sticking to decimal – click the link earlier, to see what I mean but it’s a big number).
If you were to take that 1 Exabyte of data and turn it into DVD quality video footage then it would take 50,000 years to watch – and that’s only 1 Exabyte!
Ironically (and not helping the problem) this video from You Tube gives you a great explanation of the potential of an Exaflood
What is obvious to me, is that being able to make sense of all of the data online, from a business and consumer stand point, is going to be critical and we need to find ways to simplify it all.
