Digital Dilemma 2
The AMIA TECH REVIEW asked Milt and Andy to summarize their findings from the interviews and surveys with US archives and archivists; what follows is an abstract of "The Digital Dilemma 2."
That report focused on entities that had resources and built-in ecosystems. The Library of Congress, through its National Digital Information Infrastructure Preservation Program (NDIIPP), joined with the Academy to sponsor a follow-up report on the perspectives of independent filmmakers and non-profit audio-visual archives
Several issues highlighted in the new report may be of significant interest to archives and archivists. They are extracted and reproduced here, although for a fuller understanding of the issues affecting filmmakers and archives, it is suggested that one should read the entire report. "The Digital Dilemma 2" is now available on the Academy's website: www.oscars.org/TDD2 .
Through surveys, interviews and a representative cross-section of archives for case studies, it is clear that there is a severe lack of resources. This includes not only funding (what else is new?) but the need for a clear definition of "archivist" in the digital world. While archivists managing analog collections require knowledge of film and audio technologies as well as library science, the new "digital age archive" also requires skills in Information Technology (IT), in digital media systems (both hardware and software) and in how to handle spinning media (hard drives) before the data it contains is no longer accessible.
If A-V archives are to fulfill their mission of preserving moving image and recorded sound for future generations, they need to address the new challenges presented by digital materials. Some archives are using digital technologies to "rescue" problematic analog titles. Others digitize analog materials for distribution. But for most archives, the hard drives that contain this important digital data merely get shelf space, and not the active management they require for continued data accessibility.
With digital materials, there is a rapid devaluation curve. Whether the digital data is financial, audiovisual or otherwise, the life of all digital content starts at the top of the curve (100% playable) and decreases to zero (non-accessible.) The time span of this curve in the digital world is just a few years, not the 100-year life benchmark used for analog items.
This benchmark is based on two distinctly different considerations: 1) analog film has a proven 100-year life when stored in appropriate environmental conditions, and 2) the minimum 95 year copyright protection period. Content creators (and their families) can benefit from their efforts for at least that period of time, assuming there's content to be accessed.
There are many anecdotal horror stories about losses of material in months to just a few years. Hard drive manufacturers generally warranty their products for five years, but that warranty covers only the hardware. Filmmaker Magazine, which cooperated in a survey of independent filmmakers on this subject, cited the experience of Tom Quinn. He spent several years making a film about the famous Philadelphia Mummers and their annual parade, and he made a DVD (compressed, of course) of the movie to show his family and friends one weekend. During the viewing, he noticed a section he wanted to re-edit. When he went back to his desktop editing system a few days later, he found his hard drive containing the original digital source material had failed. His work was gone.
It's certainly easy to make a digital movie today. Almost anyone can do it. Just look at YouTube or America's Funniest Home Videos. Digital technology lowers the barrier to be sure. It also increases competition to get seen by audience, a prime goal of virtually all filmmakers. Film festivals, which used to be the major route to distribution, offer limited opportunities for screenings. A survey of the top film festivals revealed that the average acceptance rate for screenings from submissions was under 5%. That also means many worthwhile films and documentaries may not be seen unless they make it into an archive and are preserved.
There is no system at present to harness all this creativity and save it for the long term future, although nonprofit audiovisual archives have managed to amass substantial holdings of independent films and documentaries.
According to the National Film Preservation Board, there are approximately 550 public moving-image archives in the United States and its territories, and an additional 310 archives worldwide. Surprisingly, none of these archives were mentioned as an archival destination by any of the independent filmmakers, documentarians, marketers, distributors and film festival organizers surveyed or interviewed for this report, with the exception of the UCLA Film & Television Archive and its partnerships with the Sundance Institute and Outfest.
Audiovisual archives, especially the nonprofit public archives, came into being because archivists, curators and historians accepted that moving image and recorded sound materials were as legitimate a part of society's cultural heritage as books, documents, photographs and other media. As a result, the universe of audiovisual media preservation extends well beyond Hollywood's borders. In the last few decades, it has become apparent that many of these archives have custody of films that have been ascribed "permanent" value, i.e., enduring historical and cultural significance, even if these films did not generate much revenue in theatrical release.
The National Film Preservation Act, first passed in 1988 and updated several times since, codified the importance of motion pictures "as an art form and a record of our times." The Act and its subsequent revisions established the National Film Preservation Board, the National Film Registry and the National Film Preservation Foundation (a public/private partnership), as well as articulated a national plan to preserve the country's film heritage.
"Digital preservation" from the audiovisual archivist's perspective must first begin with the basic concepts of archiving, of which preservation is just one part. Archival work requires actions such as appraisal, arrangement and description (cataloging), preservation, management and providing access.
For analog materials, preservation is an umbrella term that includes conservation (storage in archival conditions), preservation (cleaning and reformatting) and restoration (proactive work to return the item to as close to its original state as possible). The critical conservation step means essentially "do no harm." If analog materials are stored in a cold, dry environment in appropriate containers, their life expectancy will be extended with minimal human intervention. Also known as the "store and ignore" approach, this relatively passive strategy is not possible with digital media.
Recent definitions of "digital preservation" by professional associations stress file management and related actions. If anything, the term "digital preservation" is a misnomer. Some use the term "digital archiving," which is different from what traditional IT staff considers "archiving" (regular backup to digital data tape with or without verification steps). Others prefer "data curation," which emphasizes the active management of a digital file throughout its life cycle. Whichever term is used, preserving digital media is an active process that comprises all stages of traditional archival work and important additional actions, some of which are very complex. Some of these additional actions must be taken even before the digital audiovisual data is created - selecting file formats and storage media, for example, and considering data-handling workflows that facilitate downstream preservation. In traditional analog archiving, archivists typically face substantial backlogs of incoming materials, but they can appraise and catalog them well after they are received, because analog objects are generally able to remain in an input queue for long periods without decaying.
By contrast, digital collections must be appraised and cataloged at the time of their creation, because there is insufficient time, resources and technical information available to process them for guaranteed long-term access once the digital collection reaches the archive. The fundamental difference is that the unmanaged life expectancy of digital materials is much, much shorter than that of their analog counterparts.
Several other actions are required to preserve digital media: copying to new media and file formats ("migration"), maintaining redundancy and verifying data integrity, as well as scheduling and managing the actions themselves. A further consideration is that assessing and cataloging digital collections require specialized hardware and software tools that may vary from collection to collection, and these tools require specialized technical skills to operate. Here, the term "digital preservation" is used to refer to both reformatting (using digital techniques to preserve analog or digital originals) and preserving the digital files themselves.
The growth of the Internet and the public access opportunities it provides, as well as the increased monetization opportunities that come with access to archival collections, are the primary drivers of digitization of analog audiovisual materials.
Since the original analog source material is not discarded, there is no need to invest the resources required to preserve the digital copies; the digital copies can be regenerated if necessary for as long as the analog source material is accessible.
Digital files for
which there are no analog source materials are handled differently: they
are treated as preservation master copies and therefore must be managed
according to best digital preservation practices to maintain their viability.
If preservation actions are not taken, the files become unreadable and
are effectively lost.
Practices: Data Backup, Verification and Migration:
The archivists interviewed for the case studies were asked to identify what they thought were the biggest challenges facing archives that intend to preserve digital audiovisual files for hundreds of years:
The archives that stored files on their parent institution's enterprise storage system had varied levels of influence on preservation and management policies. In survey responses and case studies, the archives that stored files on another department's equipment stated a preference for setting their own archiving policies, but many met resistance when they tried to do so. For example, some archivists favored storing a set of digital backup tapes off-site and migrating the files to a new generation of digital data tape every five years. This conflicted with traditional IT backup practice, which assigns a 30-year life expectancy to the tape stock and does not necessarily recognize the need to re-verify and backup data before the end of the physical media's useful life.
It is important to note that digital data tape manufacturer specifications for a 30-year life expectancy apply to the physical media only. It is currently unknown how long recorded digital data will last with the "store and ignore" approach, but the consensus in the user community is that it will not last anywhere near 30 years.
Furthermore, no surveyed archive would consider storing digital data tapes for that length of time, because today's data storage hardware and software reach technology obsolescence every five to seven years, and digital recording media historically become obsolete after two of these replacement cycles.
Several archives noted
the importance of working cooperatively with their institutions' IT departments,
if there was one and to forge a relationship in which each department's
strengths would be utilized. In an ideal scenario, the archive would recommend
digital preservation actions according to archival principles, and the
IT department would recommend how to implement those actions. This was
considered a practical
Many expressed concern
that resource, organizational and technical issues would hinder their
ability to design and implement comprehensive digital preservation programs.
When asked what they thought should or could be done to assist them in
developing adequate digital preservation programs for their audiovisual
materials, the archivists responded with some recommendations and others
were proposed by the staff and writers.
image preservation format standards:
It is worth noting that as of this writing, there are multiple efforts underway that are expected to lead to a set of digital moving image preservation format standards: the Federal Agencies Digitization Guidelines Initiative Audio-Visual Working Group, the Image Interchange Framework project and the Interoperable Master Format project.
or cooperative digital reformatting facilities:
Several of the surveyed archives also suggested building a digitizing "co-op," where several archives would pool their resources to build a shared facility.
with IT departments:
fee-based digital preservation relationships with neighboring institutions:
metadata and file management tools:
They would like to see software tools designed specifically for audiovisual archives rather than production-oriented DAMS or systems built from generic database management software. They also want their digitizing software to produce usable metadata in accepted schema such as PBCore and PREMIS.
for audiovisual archivists and managers involved in digital preservation:
Nonprofit audiovisual archives that are receiving or generating digital materials have the same problems that were reported for major studios in The Digital Dilemma. However, the archives' problems are compounded by severe resource limitations. This is not to say that the nonprofit community has no significant digital preservation projects in development. For example, the Corporation for Public Broadcasting (CPB) spearheads the American Archive, a comprehensive effort to locate and preserve documentaries and other programs created for public radio and television stations. It is important to note that the CPB has an operating budget in excess of $400 million, which while unusual for an organization in the nonprofit audiovisual archive community, makes the CPB well-suited for such a leadership role.
A replacement for
film as an archival medium:
Create film separation
masters as archival masters:
Enable the enterprise
to develop a rational digital preservation strategy:
The industry must
The digital preservation needs of the motion picture industry as a whole have not changed. In this regard the needs of independent filmmakers and nonprofit audiovisual archives seem more urgent, given the changing dynamics of theatrical and non-theatrical digital distribution, the duration of copyright protection, the lack of a defined path to an archive and severe resource limitations.
It also seems unreasonable to burden nonprofit audiovisual archives with a dilemma they did not create, and unrealistic to expect them to organize themselves to tackle the dilemma without help. The studies by the Library of Congress mentioned earlier and the final report from the Blue Ribbon Task Force on Sustainable Digital Preservation and Access all contain calls to action for improved funding mechanisms and collaborative efforts. In effect these studies suggest sharing the burden of devising practical solutions that will enable all community members to maintain access to important cultural, historical and artistic works. Until the underlying operational and technology obsolescence problems are solved, however, a renewed and revised call to action in three key areas seems to be justified:
Documentarian and author Betsy McLane suggests that the only way to assure preservation is to have funding organizations insist that applicants include preservation in their budgets. This concept is being implemented in the scientific research community by the National Science Foundation (NSF), which now requires that all grant proposals for NSF projects include a data preservation plan if scientific data is generated as part of the project.
A primary sales pitch by digital storage system vendors is that "storage always gets cheaper," and while that has proven true with respect to storage media, price efficiency is significantly offset by rising energy and labor costs as well as more sophisticated technological support systems and data management policies that such systems require. Increases in data storage density show no signs of abating, but history suggests there is little reason to believe that the already huge (and growing) amount of important digital data will somehow manage itself.
While some claim that following simple data backup procedures can preserve one or even a handful of digital motion pictures, these procedures do not scale for larger collections held at the hundreds of audiovisual archives worldwide. For these archives, the wide variety of incoming data formats and media types, coupled with the archives' limited financial, technological and staff resources, make it impossible for them to do much more than shelve the material while they wait for the largest and best-funded institutions and organizations across the private and public sectors to solve the problem in a way that might yield trickle-down benefits.
Time is and will be the greatest enemy of future access to digital data. Filmmakers ignore the limited lifetime of unmanaged digital data at their own peril. For nonprofit audiovisual archives, continued deferment of a comprehensive digital preservation program will result in mission failure. Interim options offer some possibilities for temporarily extending the accessibility of digital content. The authors hope that there will eventually be a standardized, globally adopted solution that will address the technology obsolescence issue. Until that happens, and without immediate mitigating action, our moving image and recorded sound heritage is in danger of beginning to disappear in a few years. Facing this danger begins with answering key questions during the production of economically or culturally valuable digital work:
But the broader questions remain:
The time for doing studies and defining problems has passed. The issues are clear. The steps to answering these critical questions are also clear, and they start with you, the reader.
It is the hope of all who worked on The Digital Dilemma and The Digital Dilemma 2 that the next report will be titled The Digital Solution.
The Tech Review . April, 2012. ©2012. Association of Moving Image Archivists.