The Need for Digital Archiving Standards
- By Michael Looney
- 02/27/03
Campus tour guides at Yale University
are known to tell a story about Yale's Beinecke Rare Book and Manuscript Library,
one of the world's great document repositories and home to a copy of the Gutenberg
Bible, the first Western book printed from movable type. Standing outside the
Beinecke, the guides describe a remarkable mechanism that, should the terrible
need arise, would cause its glass-encased central tower and its 780,000 volumes
to withdraw deep underground, tucked away from any possible threat of destruction.
It isn't true, but when visitors see the illuminated pages of the Gutenberg
Bible or peruse the papers of Samuel Clemens, the idea of a subterranean vault
sounds prudent.
Librarians and archivists at colleges across the country couldn't agree more.
Storing important works so future scholars have access is a vital part of what
many university libraries and museums do. But as print-based publishing is outpaced
by an onslaught of digital material, traditional archival methods are facing
challenges.
The job of digitally storing and sharing that content is increasingly complicated.
The Web, as just one example, is the largest living document ever created. At
four billion public pages (and another 550 billion pages accessible via the
"deep Web"), it is 55 times larger than the entire contents of the Library of
Congress. Only 10 years old, the Web already is a fundamental resource for students
and faculty, who find they are moving from a print-based world to one saturated
with ever-changing digital content. The Web adds seven million new pages every
day, but on average those pages disappear in 44 days (Lyman 2002).
Fortunately, universities and research libraries are committed to keeping up.
They increasingly are incorporating digital information sources into their collections
and curricula, while making digital records of physical archives. Yet they are
also finding that the old, universally accepted ways to catalog and access information
often no longer apply. Archives aren't purely physical places for the archivist,
librarian or faculty member conducting research or teaching.
The Standards Conundrum
We can all be thankful that, through the ages, we've collectively decided what
form a book should take. In Western civilizations, we know that we can find
the title on a book's spine; we safely assume the contents will be printed on
sheets of paper, ordered from left to right; we know how to use the college
library with varying degrees of success.
Now, imagine if these little details weren't decided at all, leaving it to
individual publishers to decide what a book really is. Or, if each college librarian
created his or her own cataloging method, forcing us to completely re-learn
the process of locating information from one library to the next.
It would, of course, be chaos, which is precisely what standards are designed
to prevent. Nowhere is this more relevant than in the creation of digital archives,
whose future is dependent upon standards—community standards that define common
procedures, and technology standards that uniformly enable digital storage and
retrieval. Pinpointing best practices and technologies for digital archiving
is a core initiative of RLG, a consortium of 160 universities, national libraries,
and other institutions. "Without standards, there is really no hope for digital
archives to be usable many years from now," notes Merrilee Proffitt, program
officer at RLG.
The discussion of standards can get overwhelming in no time. For instance,
consider the quandary of the digital archivist, who must determine the right
processes and technologies to create digital records of printed works, films,
audio tapes, or images. Those creating archiving structures also must define
easy ways for students, with little or no training, to quickly peruse their
options and locate exactly what they need—preferably without first having to
open a single digital file.
These issues tie directly into questions about technology. What format that
exists today will sustain digital records for 20, 50, even 100 years? Content
must be easy to capture, and it must be viewable using tools that are readily
available. What's more, a file created in 2003 must be viewable in 2099.
SMU Dives into Databases
Law students at SMU's Dedman School of Law are undergoing their own digital
revolution, in the wake of a similar transformation among leading professional
law firms. Adjunct Professor Steve Kardell, who is also a corporate governance
attorney in Dallas, supplements classroom materials with content from
the online LexisNexis database and Web pages captured via the Web capture
feature in Acrobat 5. "It's amazing for students to see how much content
is out there in addition to standard classroom material," says Kardell.
Acrobat also simplifies the process of keeping archived materials up
to date. "These are living documents, so we need a way to keep them current,"
says Kardell, who also is evaluating requirements for a permanent SMU
digital law archive. "With Acrobat, authorized editors just go into the
portal from time to time and update the material. You can highlight and
annotate material in ways that people don't realize."
In the transient world of computer technology, that's a tall order. For instance,
there's ASCII, the only electronic document format recognized by the National
Archives and Records Administration. ASCII d'es a fine job of recording and
displaying text—so long as it isn't Asian text, which requires more than ASCII's
set of 128 characters. In fact, ASCII's usefulness stops well short of many
materials headed for digital archives. It can't, for instance, accurately render
a Web page featuring photos and reports from the World Trade Center attacks
on Sept. 11, 2001.
HTML, the language used to display Web text, g'es one better by formatting
text into layouts and identifying areas for photos or links. But not all HTML
is created in precisely the same way—an HTML page may appear differently on
your Web browser than on mine. When researchers and students need to study an
item exactly as it appeared, the limitations of HTML become apparent.
In settling on the right technologies, archivists must match expected longevity
with visual acuity. Peter Ullmann, a key participant in standards efforts at
Adobe, puts it simply: "What's going to give you the sense that you're looking
at something real?"
Though lacking a universally accepted standard, many schools already are hard
at work building digital archives for curricular and administrative use. Typically,
they establish their own processes and select the technologies that best meet
their needs. Many of these systems incorporate the MARC (MAchine Readable Cataloging)
system developed in the 1960s by the Library of Congress. The MARC system provides
electronic access to bibliographical information for a library's inventory,
and may serve as a model for much of the "metadata"—or contextual information—that
will tag digital archives of tomorrow.
But that's tomorrow. Many universities already have successfully created distinctly
different digital archives with Adobe Acrobat software, a low-cost authoring
tool that easily generates any document in Portable Document Format (PDF) (see
"Electronic Archives at Whitman College"). PDF is a broadly accepted, open specification
for final-format documents that can be viewed using freely available Acrobat
Reader software. PDF retains the format of the original document or Web page,
so elements like pagination, photographs, and hyperlinks remain true to the
original.
Electronic Archives at Whitman College
Professors at this small, liberal arts school in southeast Washington
often require their students to use materials that are not available in
the public stacks at the 385,000-volume Penrose Library. "A professor
might have material he's gleaned over the years that he wants his students
to have access to," says Michael Quiner, director of administrative technology
at Whitman. "The old approach is to make photocopies, and then students
go to the Reserve Desk to read them." For greater flexibility, Whitman
administrators devised eReserve, a growing digital archive of reserve
materials viewable online from students' dorm rooms.
In two years, the eReserve program has archived some 400 articles as
PDF files that authorized students can read using a standard Acrobat Reader.
In fact, Quiner says, several professors have started their own efforts
to archive class material in PDF so students can conduct research online.
To College Librarian Henry M. Yaple, the benefits extend beyond
allowing a student to read documents at all hours: "It's a matter
of preservation," Yaple says. "eReserve allows a lot of people
to read the documents without handling the primary material."
Yaple's argument rings especially true with documents like Yale's
Gutenberg Bible, whose rare pages could be viewed digitally by
any religious studies student without ever touching the book.
Where We're Headed
While organizations like RLG work to define digital archiving standards, certain
technologies are likely to find themselves at the forefront of the debate. Their
prominence suggests that they will play at least some role as standards evolve.
One of these technologies is XML (eXtensible Markup Language), which is becoming
vastly popular for many applications. XML allows information to quickly come
together from various locations to form a Web document that can be easily read,
and features an advanced approach to tagging content so that its components
appear in their logical order once they reach their destination. XML appears
to be an excellent candidate for supplying the technical backbone of a digital
indexing system. "XML schema language can provide the universal structure that
allows any school to look at technical metadata," explains RLG's Proffitt.
Maintaining accurate page format of the paper document, however, is not among
XML's many strengths. This d'esn't matter if a student is reading the text of
Dr. Martin Luther King Jr.'s historic "I Have a Dream" speech. But XML is at
a disadvantage when a student must view an original document.
While XML excels at transporting information, PDF excels at displaying visually
rich information. PDF preserves the pagination integrity of original documents,
even when they are viewed on PDAs or next-generation wireless phones. Digital
archiving is a marriage of data and documents. The two must live together, and
for a very long time.
Adobe's recent development around PDF recognizes this. Acrobat 5.0 exports
XML along with PDF, resulting in an XML-tagged document that retains its pagination
no matter how it is reviewed. This would allow a journalism student to look
up a story from yesterday's Los Angeles Times on her handheld. She knows the
story appeared on page 16 in the print version. If the page is stored in PDF
and tagged with XML, that's exactly where she'll find it on her PDA.
As new display devices become popular, these capabilities will be necessary
to find and view records that were archived years before. Wharton's Kendall
Whitehouse already has proven this is possible with PDF, viewing a document
archived in 1995 on three different platforms: a desktop computer, a Palm OS
handheld, and a Compaq iPAQ Pocket PC. When Whitehouse archived the document,
those two handheld devices did not even exist (see "Wharton Students Straight
to PDF").
Wharton Students Straight to PDF
In 1993, Wharton administrators began digitally preserving all school
publications, catalogs, course materials, and faculty research papers.
Yet the school's efforts don't stop at archiving. Some faculty members
also have their students electronically submit their assignments as PDF
files, which are then annotated with comments and corrections and re-posted
for the student to view.
According to Kendall Whitehouse, Wharton's director of advanced technology,
a successful archiving format must faithfully represent the original work
without requiring complicated technical back flips. "A lot of formats
depend upon the material being created in a certain way," says Whitehouse.
"But Acrobat and PDF are completely agnostic. You can view the files on
Windows, Mac, Unix, and handheld devices. None of these require manual
altering or modification."
Archives must be easily viewable for years to come, a constraint that
automatically narrows the technology field. "Our original PDF documents
from 1993 have actually improved with age because the Acrobat Reader has
evolved," Whitehouse says. "It's hard to list other formats that are both
backward and forward compatible."
An indexing feature called e-Binding also allows educators to combine multiple
content in various formats—images from a photo essay, maps from an atlas, spreadsheets,
and text documents—into a single PDF file. And several legacy documents can
be batch-processed to create a multi-layered work that can be searched for key
words or phrases. For faculty, this offers the chance to create an online "course
pack" for students that is easily indexed, searchable, and updated over time.
While PDF itself has become a de facto industry standard, two industry organizations
are jointly working to establish an official archiving standard based on PDF
technology, called PDF/A (see "Making the Case for PDF/A"). The groups are working
to see PDF/A recognized by the International Standards Organization as a global
standard for document archiving.
For now, digital archivists seem focused on tackling the issue of electronically
documenting old and rare printed works, or capturing a Web page before it changes
only a few hours later. And as immensely useful as digital archiving standards
undoubtedly will be, educators point out that a ubiquitous system for higher
education won't likely replace fixtures like research librarians anytime soon.
"Tracking down information correctly is a tricky business, and it requires a
skilled professional to do it," says Whitman College Librarian Henry Yaple.
"That's what librarians do." With solid digital archiving standards, that job
may become considerably easier.
Making the Case for PDF/A
A recent study estimates that the world's total production of information
amounts to about 250MB—some 100,000 pages—for each man, woman, and child
on earth. Printed documents comprise only .003 percent of the total (Lyman
and Varian 2000).
99.997 percent of all information is digital—and it's growing fast.
Some futurists anticipate that someday the world's knowledge will double
every 900 days. The Census Bureau, for example, has accumulated 600 million
pages of information from the 2000 Census that it will be transferring
to the National Archives and Records Administration (NARA)—equaling
10TB of data. That's more than five times the amount of data that NARA
has captured and fully processed in its entire 30-year history.
Yet because of their historical value, billions of documents need to
be managed, preserved, and made accessible for future generations. This
daunting task requires a solution that recognizes the wide range of information
systems, technologies, and formats in which records are generated.
To a growing number of industry groups and users, one solution is PDF—a
broadly accepted standard for the delivery of final-format documents.
More than 20 million PDF documents are publicly available on the Internet,
and almost half a billion copies of the free Acrobat Reader have been
downloaded. PDF retains the content, look, and feel of the document exactly
as it was created, ensuring document integrity and security, while also
allowing documents to be searched. In fact, some countries already have
accepted PDF as an archive standard. However, PDF has evolved to provide
a number of functions that, while beneficial to users who share documents,
are not ideal for long-term archiving: password-based security of documents,
optional (rather than required) embedding of specific fonts, the ability
to embed multimedia in other formats, and the ability to launch other
applications from within PDF.
Consequently, a subset of PDF—PDF/A, with the "A" standing for archive—is
being developed for archiving and preserving digital documents. PDF/A—a
joint initiative by the Association for Suppliers of Printing, Publishing
and Converting Technologies (NPES) and the Association for Information
and Image Management, International (AIIM)—will address the growing need
to electronically archive documents to ensure preservation of their contents
over an extended period of time. PDF/A will also ensure that those documents
can be retrieved and rendered with a consistent and predictable result
far into the future.
PDF/A proponents (a working group comprised of industry, government,
and academic institutions working with AIIM and NPES) are aiming to have
PDF/A officially recognized by the International Standards Organization
(of ISO 9000 fame) within approximately 18 months. Their efforts are directed
at solving a serious and increasingly urgent problem. The lack of a recognized
and accepted electronic standard for records preservation—particularly
as new generations of hardware and software have made previous digital
technology obsolete—has led to the loss of significant amounts of
valuable information over the past several decades. Military files from
the Vietnam War, records from the Viking Mars Mission, Census Bureau data
and land use records have been lost due to the inability to read data
formats and the deterioration of magnetic tapes used to store that data.
The list of organizations that mandate or use PDF as a de facto standard
is growing to include the U.S. Courts, the National Science Foundation
for grant submission, and the Food and Drug Administration for drug submissions.
A common PDF/A standard will give librarians and educators the confidence
that their records could be readily accessed far into the future.
Resources
Lyman, Peter, "Archiving the World Wide Web." LOOP: AIGA Journal of
Interaction Design Education, December 2002, Number 6. Retrieved from
http://loop.aiga.org/content.cfm?ContentID=100
on Jan. 22, 2003.
Lyman, Peter and Varian, Hal R., "How Much Information," 2000.
Retrieved from www.sims.berkeley.edu/how-much-info
on Jan. 26, 2003.
