Are You Drowning in Data?

• By Terry Calhoun
• 03/08/07

Tech analyst John Gantz estimates that in 2007 the world will for the first time generate more "data" than it has storage space for. But that's not really the major kind of crisis it might seem. After all, a lot of the data are things no one intends to store in the first place. (Witness, for example, live, 24-hour video feeds of building construction on many campuses. Nobody plans to store all of that video!)

I wrote recently about a book in which one narrative subplot was the digitization of the entire British museum and library, and then storage of basically all of the world's knowledge, culture, and history on a single chip. That chip was generated, fictionally, in 2025 and had a capacity of 128 petabytes. Gantz calculates that in 2006, humans generated 161 exabytes (164,864 petabytes) of digital data.

Wow.

My first hard drive was a 20 MB external drive to an early Macintosh desktop machine. I was absolutely thrilled by that capacity. Heck, I was thrilled the first time I used an IBM Selectric typewriter  with a magnetic card that could store a few typed lines. Now I'm a bit frustrated by the 80 GB capacity of my current laptop because I have recently had to take a bunch of images off of it and store them on a central server.

Having less storage space than I have data is not something I am personally worried about. As a data storage consumer, I have watched my costs for storage go down and my options go up. I am delighted that my laptop, my camera, and my phone each have a  2 GB storage chip inserted and that the chips didn't cost a fortune. My concerns are finding stuff I have stored and worrying about losing access to data stored in archaic formats. How about you?

Disappearing data
On the largest scale of finding or forever losing things, those of us who use the Web constantly have gotten used to the fact that favorite "places" can disappear. We have very little control about who deletes what off of some server on the other side of the world. Like many, I often "print to PDF" an important resource that I locate on the Web, and store it on my own hard drive. I sure hope that PDF never becomes an archaic, unreadable format in my lifetime.

Recently I learned that the website of the joint conference that SCUP and others held last year, the Campus of the Future conference, had its plug pulled. I wasn't happy about that. I am the type of person who never throws anything away. But it wasn't my decision. Sure enough, since that happened, a short while ago I've had to respond to two queries from people who were looking for information that had once been on that website. Luckily for me, the Wayback Machine saved the day.

On the more personal scale of finding or forever losing things, have you ever experienced a complete hard drive crash, due to which you lost all of your data? It's happened to me three or four times, a couple of times before I had quality backup available. Each time, for the first moments and hours, it felt like a major catastrophe. Then, a few weeks later, things were rolling along fine with no apparent long-term impact from the data loss.

I've had the same experience with accidental loss of e-mail archives as well. I have never deliberately erased from storage a single non-spam e-mail message. By all rights, I should have several million stored messages to tap into. But it seems that, like their clunky and slow search capacities, e-mail clients in general assume that the user is going to delete, delete, delete. Just try collecting 50,000 messages in a single folder in Thunderbird and see what happens!

A data time machine
But, you know, experience has taught me that these kinds of losses are generally a blip in the overall scheme of things. We all manage to ensure that the really important things we need professionally are backed up and stored. I am still waiting, however, for the implementation of user-friendly, fully-functional eportfolios. Not that the work done by the ePortfolio Consortium folks isn't good. It is. But I'm approaching 60 years of age, and it doesn't seem likely that I'll ever benefit from such a thing. (I hope those words come back to haunt me some day.) But surely someone will offer a comprehensive, guaranteed, automatic, never-filled, user-friendly virtual storage that just collects and collects and collects your stuff and, over time converts it into modern, usable formats, right?

Oh, wait, check out the new Mac OS X feature coming along this spring, the "Time Machine."

More important to me than sometimes losing data is finding things when I need them and not wasting time putting things away. Back about the same time that I was delighting in my first 20 MB hard drive, I had an epiphany about filing digital things away. I learned about the Command-F function. I went in days from being a complete geek about files, filing things away, hierarchical file structures, and the like, to complete relaxation.

If you saw my laptop's desktop, you might, as has happened many times, gasp in shock. At various points in time I'll have enough icons on there that it would take 10 times as much screen space as I have to display them all. My filing system is mostly geared toward taking everything left on my desktop and putting it into a folder dated the day on which I did so. In fact, I will do so later today. Everything on my desktop right now will soon be in a folder called "Desktop 030707," and experience shows me that I will not have a single problem getting to something I need.

I predict that some day in the not too distant future, intelligent agents will decide for us where to put files we are working on, and find them for us when we need them. Just like I now longer need to put an asterisk before and after a bolded word, like I had to when using the early word processor Wordstar, I won't have to spend time worrying about where to file something.

As "storage consumers," just like we no longer need to know how to spell when we write, we also won't have to worry our minds with the details of our hierarchical file structures, and there will be no need to consciously navigate through them. And despite what some might think right now, we'll have plenty of storage space, too.

NEW! Reader forums!
Do you have questions about this article? Want to comment on it? Join Terry and friends in our new IT Trends forum at http://campustechnology.com/forum/forum_topics.asp?FID=19.