Cheap and plentiful data storage can be both a blessing and a curse. It creates the illusion there is infinite space available for data on networks. This intensifies human pack-rat tendencies to hoard data instead of evaluating what's really needed, so much so that demand for storage is growing by 50 per cent annually in some areas, far outstripping the rate of storage cost reductions.
All this accumulating data has impacts far beyond storage costs, cascading into all aspects of IT operations and management.
"That's the trivial part of the problem," says Roy Wiseman, CIO for the Region of Peel, pointing out there would still be a big problem even if petabytes cost pennies. Time, resources and energy are being consumed to house, back up, archive, secure and manage vast quantities of junk data along with the good. "This is unsustainable," he says.
Waste is not the only issue. Public sector CIOs are expected to respond instantly to requests for information contained in their systems, says Wiseman.
"If someone says, I need you to produce any document in your system that refers to X, there will be billions of files to be searched to find it: e-mails, drafts, duplicates, multiple versions, all considered corporate records," he says. "The information management component is fairly scary relative to what people are expected to do."
Digital preservation for eternity
Hoarding data at exponentially increasing rates is a universal problem across all sectors. But what makes the problem trickier in the public sector is the fiduciary responsibility to preserve public records for very long periods of time, says John Webster, storage analyst at Nashua, N.H.-based Illuminata Inc.
"Thirty years is a long time in the private sector," he says. Not so for government records, and some types such as legislative documents need to be preserved for centuries.
Electronic storage media are constantly evolving and being replaced with new forms. But all are magnetic-based, and magnetic fields degrade after a few decades. While most public entities have a migration strategy to shift critical documents from obsolete media to the next incarnation, no one has figured out how to deal with software obsolescence, says Webster.
"If you want to read data back in the future, will there be software that understands it? The CIO of Massachusetts says this is his biggest issue."
Wiseman agrees the hardware component of digital preservation is well understood, but not the software issue. For example, documents created in defunct software such as WordStar can't be read by other programs today. "So you may have the information stored, but is it accessible? We haven't come to grips with maintaining the capability to read it over time."
Another tricky aspect is the need for public disclosure, and to be in a position to respond to any request for information, says Webster. "In the public sector, the attitude is, 'You'll never know what you'll be asked for, so you may as well save it all.'"
The move to make government services available online has also resulted in increases in the volume of transactional data that could be subject to public access, he adds. Data can be gotten rid of within reasonable timeframes in the private sector, if policies for deletion are developed that can be legally supported, he says. "But I'm not sure they have that luxury in government."
Tiered storage is only a stop-gap measure, adds Webster. Moving less frequently accessed data from servers to cheaper secondary storage media does reduce costs.
"It may get you bigger bang for your storage buck, but if you're not figuring out what you can delete in that process, then all you're doing is creating a great big back-end that keeps growing. And that's getting bothersome not just from a technology and management standpoint, but also energy consumption," he says, pointing out data centres everywhere are scrambling to keep up with space, cooling and power requirements as more and more computer gear gets added.
Continued: The limits of technology
Related content:
Integrating IT for collaboration
Information will exceed storage capacity, says IDC
Storage, It is really about filing cabinets
Information management comes to the table
EMC protects, preserves priceless documents
icon.
