The storage dilemma: what is worth keeping?

Am I the only one who feels bemused by the obsession we have with storing everything? I suppose it shouldn’t feel strange that at a time when we are generating more and more data and information in all kinds of formats, we have become increasingly concerned with making sure we store it all somewhere securely.

But where’s the value? This is the point where people talk about big data, data analytics and all those other technologies that they argue are best designed to extract value and generate insights from those vast pools of data currently being produced. But let us not forget the data already out there on paper which hasn’t been captured and digitised in a format that makes it accessible, readable and searchable.

“The editing process resulted in some things not being put on paper. Even some of the things that were put on paper were neglected or rejected over time as more recent developments proved them to be erroneous, misguided or dangerous”

 

advertisement

 

Fair enough, we are not actually starting from the best place so it is much easier to process and store digitised data, information and documents than their paper-based equivalents, but how quickly can we convert and capture the older formats that are more at risk of fading into oblivion than the digital content we are now generating?

And is there a point at which the capture process becomes a waste of time? For example, let us think back to a time before people started using paper to capture and store data and information, to a time before papyrus even. All there was before that was the oral tradition where stories and information were stored in the minds of people who then told them to chosen successors in the next generation to act as repositories for stories and knowledge.

But it was not all knowledge. It had been selected, filtered and edited. Extraneous information deemed surplus to requirements was discarded and forgotten. The holders of the knowledge had decided, quite wisely, that people in the future were unlikely to be interested in knowing exactly what such and such a great chieftain had chosen to do on a particular Wednesday afternoon in July. They had made a judgement that some information just was not worth keeping.

The same thing happened with paper which provided the opportunity to store more information and knowledge but not everything was written down. The editing process resulted in some things not being put on paper. Even some of the things that were put on paper were neglected or rejected over time as more recent developments proved them to be erroneous, misguided or dangerous. Very few people cared that those things had not been written down or cast aside after they had been put on paper.

So why should digitised data and information be any different? The main reason appears to be that too much stuff is being generated too quickly to make a considered decision on what to keep and what to throw away. Another rationale is that if you have the technology to access all of that information then you might as well use it because you never know when the information you choose to discard could prove to be useful.

Essentially, we are giving up the right to edit in real time in favour of storing everything. Why? Initially because we couldn’t process and edit the stuff fast enough and we needed to keep it in case it proved necessary for something in the future. Now, it is because the technology exists that can search it for what we want or need so we do not need to edit it.

From a technology perspective, what is not to like? People bought the IT that generated the data, information, documents, whatever and they bought the hardware to store it all. Then, after they did all that, they had to buy the technology to give that data value by making it accessible and searchable and capable of providing analysis and insights at a much finer level of detail than we ordinary humans could achieve. The end result could be beneficial but how we got there still seems counter-intuitive.

It is slightly better than hoarding, obviously, but there is no denying that it started off uncomfortably close to it. And there is still the concern that, like hoarding, we could end up letting what we store take over without getting enough back from it to justify the space it occupies.