New storage technology opens up archiving options
- By Jane Morrissey
- Jul 06, 1997
Choosing an archival storage medium used to be a straightforward task. Users who needed high-capacity at the lowest price went with tape. Others who needed an unalterable format or random access chose optical disk while those who needed speed and price turned to magnetic disk.
All that has changed. Now the market is flooded with options that overlap in terms of price performance capacity reliability and other key requirements. The technology has blurred distinctions between the categories of offline and online storage.
Offline refers to the traditional large data archives that organizations need to maintain but do not need to make generally available. In contrast organizations want online data to be readily available and they are willing to pay higher costs for higher-speed media.
However storage vendors now talk about near-line storage for customers whose requirements straddle the traditional categories keeping costs down but offering quicker access to archived data. Within every product category there is new technology plus different media formats within each technology leaving users faced with the daunting task of predicting which will have staying power.
Government agencies' need for enormous capacity at a low expense is a given according to vendors. With advances in storage technology agencies with massive requirements - such as the Social Security Administration the Patent and Trademark Office and the Defense Department - are altering their existing storage systems sometimes radically to accommodate the electronic data explosion.
"In a lot of the large federal agencies data growth over the next four to five years could jump [from] 400 to 700 percent " predicted Gary Sobol scientific application manager for Storage Technology Corp. Louisville Colo.
As agencies have become more technology-literate their requirements have kept pace. Users previously content with offline access are hungering for online or near-line access times and they are expecting it at much lower costs than in the past. That's where products such as EMC Corp.'s Extended On-Line Storage systems (EOS) come into play.
EMC describes EOS a disk-based high-capacity storage and retrieval system as an intermediate data repository between high-speed online disk arrays and offline tape libraries and optical jukeboxes.
"You have higher expectations on the part of the federal agency customer and then you have less of a difference in cost between online or near-line vs. off-line " said Carolyn Hyde federal district manager for EMC Hopkinton Mass. "A lot of times this is what has driven our customer to EOS. EOS won't totally replace offline storage but I think you'll certainly see overlap between people looking at both approaches."
Such new options make it difficult for agencies to keep pace.
Even the National Archives and Records Administration which accepts several media formats from agencies for its giant national archive is at a crossroads in evaluating what is next for its own electronic storage needs."It's a conundrum for us " said Michael Miller NARA's director of records management programs in College Park Md. "Once we take it we are responsible for making sure people can access it down the road. Technology is littered with false starts and media that people can no longer access."
For NARA backward compatibility is paramount - backward meaning an incredible 40 or 50 years. That may help explain why even with today's technology NARA processes 10 times the number of paper and microfiche records compared with electronic ones although electronic media are expected to rise. NARA largely limits the formats in which it will accept such records to 9-track tapes IBM-compatible 3480 tape cartridges and CDs although data is transferred off CDs and onto tape.
There is an upside to all these options - more competitive choices than ever before. "The bottom line is that no one media type fits all " said Gil Van Schoor director of federal sales and marketing for Eastman Kodak Co. Rochester N.Y. a leading maker of 14-inch optical disk and CD systems. "But the more the merrier. The end user will have a product-rich environment to select technologies that best suit the application as it evolves through time."
Where Tape Is Still King
When archival storage needs to reach into the multiterabyte range there is little debate that magnetic tape is still the most cost-effective solution users said. Cost per megabyte always tape's strong suit continues to decline into smaller fractions of pennies. Already high transfer rates from disk to tape and tape cartridge densities continue to improve vendors said.
Even tape's downside - its sequential access method which makes for longer response times - is improving. With access times of a few seconds for certain file sizes tape libraries using robotics can reach near-line status."We've had just outstanding service from tape " said Phil Becker director of systems user services and facilities for SSA. "It's cheap and the technology and the investment has lasted. It's been very reliable I can't make the same claims for changes in the optical arena particularly for mainframe storage."
SSA uses 200 000 Storage Technology Corp. (StorageTek) 3490e cartridges with capacities between 1G and 1.5G each for archival and backup of all its master files. Those files contain records such as payment and benefit histories that need to be available for years for such tasks as annual redetermination of benefits. Online access times for master file queries are usually 4 seconds or less compared with 35 seconds for older records held offline on tape according to SSA.
Although tried and true for years as the best mainframe storage medium tape solutions have some liabilities. The absence of backward compatibility with some of the new higher-capacity drives is one of them.
NASA's Center for Computational Sciences (NCCS) Greenbelt Md. has come up with an inventive solution. In 1991 the center invested in two StorageTek tape silos for its mainframe and supercomputer mass-storage needs for earth and space research. At the time StorageTek offered the highest capacity - 1.2 terabytes per silo - which has since doubled twice. NCCS' system is now at more than seven silos which take in about 1 terabyte of new data per day with NCCS predicting daily growth of 2 terabytes by 1999.
However IBM has since introduced its Magstar 128-track tape capacity of 10G vs. StorageTek's 36-track at 800M. Wanting to retain the investment in StorageTek robotics but exploit the better capacity of the IBM drives last November NCCS "actually bastardized the StorageTek silo " said Nancy Palm head of NCCS.
"We've stuck an alien device an IBM Magstar tape drive in the silo.... Using the robotics the benefit is a 12-fold increase in capacity " Palm said. In 50 percent of the space it takes to house the silo NCCS can store about five times the amount of data she added. Cartridge exchange times are up 50 percent at about 20 percent less the cost.
Managing a system of multiple formats is increasingly common. One agency that began using StorageTek's 50G Redwood tape cartridges in April is now performing a balancing act to support its older media.
"We like Redwood's higher capacity and higher transfer rates better than 3490 tape " said the head of mass storage for a national research center who requested anonymity. "But Redwood's not backward-compatible with 3490. We are optimizing transfer rates and response times: Only large files go onto Redwood small files to 3490 and really small files to disk."
Even within the attractively priced tape category there are still cost considerations. The Naval Air Warfare Center in Patuxent River Md. is using Exabyte Corp.'s 8mm Eliant tape drives to store research information associated with its tests on naval aircraft. As the center becomes more centralized it's eyeing an upgrade to Exabyte's higher-end Mammoth drives."We haven't really embraced Mammoth yet because the media price is still pretty high even with the recent price cut " said Charles Palmer the enterprise network manager at the center. For now he will stick with the 14G Eliant tape rather than the 40G Mammoth.
The Big Squeeze
Skipping or moving off optical disk in favor of Redundant Arrays of Independent Disks (RAID) storage is a growing trend. As magnetic disk prices continue to drop and tape functionality increases the optical disk market is under pressure from both ends.
"It's really like a vise " said Anders Lofgren an industry analyst at Giga Information Group Cambridge Mass. "The price/performance of optical disk is not increasing such that it can get squeezed on performance from above and on price from below."
Several agencies that adopted optical disk when it was the only option for a reasonably priced random-access archival system are moving toward RAID with tape as a backup. "The government agencies are using RAID more often because the prices have come down and capacities have gone up which is only going to increase your requirement for backup " said Linda Kempster a storage media specialist with IIT Research Institute Lanham Md. which works with federal agencies on large information systems modernization projects.
The Patent and Trademark Office which went with 12-inch optical disks in the late 1980s is migrating to an EMC RAID system. PTO needs to give its examiners access to every patent granted and trademark registered since 1972 to determine if new applications are unique. As the optical disk system grew PTO's requirement for faster access times grew along with the cost of managing the aging optical system.
"It's gotten to the point where it's more expensive to maintain those than to buy new ones " said Ken Giese director of PTO's Acquisition Management Division. "To get the speed right now we need to split the database up over a lot of disks and search them " he explained. "[RAID] allows us to consolidate the data into less physical space and our studies have shown that magnetic storage is now cheaper than optical storage for this."
Giese said PTO bought 25 terabytes of magnetic disk storage at about $1 million per terabyte five years ago the cost would have been $10 million per terabyte.
Still despite the emerging options the write-once read-many (WORM) optical disk is still a top choice for agencies requiring unalterable records some vendors said.
That is the case with the Joint Engineering Data Management Information and Control System (JEDMICS) which is the standard repository for all DOD engineering data for weapons systems.
JEDMICS opted for Kodak 14-inch WORM disks to ensure for example that each revision of an aircraft's design is preserved. JEDMICS now has 44 jukeboxes at 34 sites and platter capacity has grown from 6.8G to 25G. Access times average 10 seconds but JEDMICS is introducing caching in front of the WORM system to avoid delays accessing often-used data.
So far moving from paper to the optical disk system has saved $5 million to $11 million at each site by compressing administrative lead times. "If an F/A-18 fighter plane goes in for repair it used to take 264 days to overhaul it " said Michael Mooney a vice president at Litton/PRC Inc. the Reston Va. systems integrator on the project. "With JEDMICS it's less than 100 days."
Meanwhile the promise of the Digital Video Disc (DVD) and other ongoing advances in CD technology threaten to encroach on larger-format optical disks and tape.
DVD which will hit the market this year offers 4.7G of storage per disc with the promise of a 17G capacity once the technology matures.
While CD-ROM's universal standard is most attractive for data distribution it also has potential as an archival medium. "A lot of people got really excited when they heard [that NARA] was going to accept CDs " IIT's Kempster said. "But they accept it like an envelope and take out the contents. They do not consider CDs [to be] archival media."
That will likely change as CD-Recordable and CD-Rewritable systems become more prevalent with optical leaders such as Kodak and Cygnet Storage Solutions Inc. pushing for better capacity media quality and ensured compatibility with DVD.
"You will be able to have CD readers that can read the CDs you have today 10 years from now " said Wayne Ausberger Cygnet's vice president of marketing in San Jose Calif. "I don't think anyone can say that either about optical or tape."
For now there are plenty of existing options that have served the government well and will keep improving users said. "In the newspaper and technical reports you read everything [that storage technology] is supposed to be " said one federal marketing manager who requested anonymity. "But in the real world a lot of this stuff doesn't matter. You fill the box up you access it and the data is there."
Nonetheless he hopes the Holy Grail is on the horizon. "If DVD is everything they say it's going to be everyone's going to be darn glad when it gets here."
Morrissey is a free-lance writer based in Denver. She can be reached at firstname.lastname@example.org.
* * * * *
At A Glance
Status: Storage vendors have expanded the range of options for archiving to the point of blurring distinctions between offline and online categories.
Issues: Low-cost tape remains the prime candidate for traditional archiving but disk and even CD-ROM solutions can offer faster access at still reasonable prices.
Outlook: Excellent. The promise of DVD and other technologies suggests agencies' archiving options will continue to expand.