Canadian Museum of History’s Digital Collection Inventory: A Case Study Applying the Digital Preservation Toolkit

CHIN case studies with smaller institutions

Up until this point, CHIN had collaborated on digital preservation work with the 8th Hussars Museum in Sussex, New Brunswick, and the Medalta Museum of Medicine Hat, Alberta. The 8th Hussars is a small, volunteer-run organization whose only paid staff are interns in the summer months. Medalta is mid-sized. Both organizations are representative in their size and structure of the majority of Canadian museums, and neither has the resources to manage a system such as the Open Archival Information Systems (OAIS) model recommended by the professional digital archiving community. In each case, some of the toolkit resources were pared down to decrease the amount of work required. Also in each case, simplified solutions were adopted, which did not observe the OAIS model but still respected as many digital preservation best practices as possible, given the resources available.

These two case studies are useful resources for the majority of Canada’s museums. However, CHIN was still interested in seeing how the tools might be taken up by a larger institution. The Canadian Museum of History’s offer to participate as a case study was timely, and the fact that only the first phase, the digital assets inventory, would be sufficient for this study was testimony to the degree of thoroughness with which CMH approached the task.

Archival reorganization at the Canadian Museum of History

In 2013, the CMH Library, Archives and Documentation Services department underwent a reorganization, which prompted a review of archival practices. Previously, each archival collection had operated as its own section. As part of the reorganization, the disparate sections were brought together and all archival and library collections fell under the care of the Information Management (IM) section. The six collections in the IM section now include Audio-Visual, Photos, Textual, Corporate Records, Library and a brand new collection called New Media for assets including websites and interactive content.

The reorganization has opened the door to a more holistic information management approach. As archival deposits and accessions come from a number of different internal and external sources, which do not always fit into one archival collection, it is challenging to maintain the integrity of the intellectual and technical organization of the deposit or accession. The goal is to streamline processes, procedures and practices so that multi-format and multi-collection deposits may be better managed together, not just intellectually, but also technically.

The timing of the launch of CHIN’s Digital Preservation Inventory Template coincided with CMH’s archival reorganization and provided the museum with a good opportunity, and tool, to start the inventory process.

The first step for CMH in re-evaluating its practices and procedures and creating a digital preservation policy was grasping the scope of the collection. This included not only the assets themselves, but also the practices, procedures and tools employed to manage the assets. The challenge consisted of scoping a digital collection that dated back to the 1970s and included a multitude of formats, managed according to the information management standards of the day. To be all inclusive, and because many existing digital collections are digitized objects, CMH conducted a concurrent analogue collections inventory using the same template and processes outlined in this document.

Objectives

• To know the scope in quantity, format, digital size and location of digital assets under IM management
• To identify the areas of risks for damage, access, obsolescence

Phases

• Completed
  • Phase 1 – Provide an inventory that details the quantity, formats and digital size of CMH’s archival collection
  • Phase 1 – Identify areas of risks for damage, access, obsolescence
• To Come
  • Phase 2 – Conduct a corporation-wide inventory of digital content requiring special consideration outside of the CMH archives, and integrate preservation practices at a corporate/information producer level
  • Phase 3 – Develop and implement a preservation plan for the collections under IM management, along with recommended archival standards and formats

Formatting the Digital Preservation Inventory Template for Museums

The online Digital Preservation Inventory Template for Museums is formatted for the web and outlines the information to be gathered. However, CHIN provided the template as an Excel spreadsheet that is formatted as a form that can be filled out. The spreadsheet contains exactly the same information as the web version, but it does not conform to online accessibility standards.

• Obtain a copy of the Excel template by sending an email request to CHIN Client Services

To better capture the statistics of CMH’s collection, the template was altered and further formatted to add some additional inventory requirements.

Case example: Amongst other additions, a “Networked Storage” section was added to the Location and Environmental Conditions table since much of the digital storage at CMH is server-based. This section was added to identify the various server locations where the asset groups are stored. A “Server/Network Security and Access” section was also added to the Security table to identify the levels of access granted to the networked locations.

Collection asset groups

The inventory starts by splitting all the material into asset groups (in short, digital assets that share enough similarities to be inventoried and surveyed together). Grouping assets together not only makes the counting and surveying easier but also provides more decisive results for preservation and risk assessment purposes.

CMH designated seven asset groups, which were based on their existing collections found in Table 1. Additionally, CMH further sub-divided each asset group between processed and unprocessed material. In general, processed materials are those that have a reference number assigned and have been registered in a database or, in the case of older material, an Excel spreadsheet. Unprocessed material has no formal reference number and may be recorded in inventories but not in a database. Unprocessed materials include large accessions currently being ingested, materials that are due to be ingested but are backlogged and a few special instances of assets in the care of the archives that will never be formally accessioned.

Case example: Special unprocessed material may include leftover images from internal photographers that were not deposited with the “chosen” images but are kept on hard drives with retention schedules in case more images from a particular project are requested.

Asset groups required additional considerations:

• The Librarycollection was further split to represent the collection of digital CMH publications as opposed to commercially available digital publications. Unlike other collections, Library houses commercially available publications such as CD-ROMs which affected some survey responses such as “Years required to preserve.”
• Although the AudioandVisualcollections appear separate, they are in fact managed as one. Where necessary, results of the inventory and answers to the survey were answered to reflect both the Audio and Visual collections separately or the two collections together.
• Corporate Records are managed with some different information management industry practices since the vast majority of assets are destined for either deposit to the textual archives or destruction once they have met the end of their retention schedule.

Tools

A variety of tools were used in the inventory process. An overview of each tool, along with its uses, is provided for reference.

• Excel
  • Not only is the template formatted in an Excel spreadsheet, but Excel was used to manipulate data outputted from other tools and to tally results.
  • Most software that manages files allows for exported reports in formats that can be rendered in Excel, like .csv files.
  • Many collections had previous information in Excel spreadsheets.
• DROID ^{Footnote 3}
  • DROID was used to inventory some collections that were either stored on servers or hard drives. It was not used on collections stored on CD and other physical carriers as it would be too time consuming.
  • DROID can create .csv reports that list information from directories, including but not limited to file format, file extension, location and size. The report is a “listing” and required us to manipulate the data in Excel to get counts of file formats.
  • Although not used during this process, DROID also provides a link to the PRONOM ^{Footnote 4} registry of formats.
• MS-DOS
  • The “dir /s” command provides a listing of files in a given directory that includes the extension and file size.
  • It is much less resource intensive than DROID and is quick.
  • It creates a .txt file that requires further manipulation in Excel to extract the desired data.
• KE EMu
  • The CMH image collection is partially managed in KE EMu, so it was used to help count assets.
• Vubis
  • The CMH textual and AV archives are registered in Vubis, so it was used to help count assets.

Inventory: Overall approach

Although the various digital assets are managed with the same “preserve and make accessible” intent, archival practices can be different for each asset group. Various formats, collection structures, locations, databases, collection sizes, tools specific to formats, industry practices and legacy practices mean that one blanket inventory method could not be used for all of the asset groups. To ensure the inventory provided commensurate numbers, one staff member was assigned to assist with the counts, tally the numbers and enter the results. The same staff member conducted the survey questions alongside the archivists working with each of the asset groups so that the language used in the results was also consistent.

Inventory: Counting

The wide variety of file formats and storage types, whether server-based or on physical carriers, necessitated various counting techniques. Where possible, the techniques that could provide the most detailed inventory were used. However, certain collections could only be approximated.

Case example: Parts of the Audio-Visual asset group allowed for the most granular count by using DROID on the hard drives where some of the assets are stored. Conversely, the Textual asset group posed a challenge due to the quantity and variety of physical carriers. It would be unrealistic to expect a granular inventory of the file formats, file quantities and digital size of every diskette, ZIP drive, CD, etc. Instead, estimations could be done by listing the known file formats used during the time of the physical carrier along with the maximum physical carrier sizes. File quantities could not be counted. As a result, the older physical carriers in the Textual asset group were identified as an area of risk.

The complexity of the collection meant that a couple different combinations of tools were used. They were usually dictated by factors that allowed varying degrees of accuracy and detail: whether the collection was on a server or on physical carriers, whether existing metadata for the collection was in a database or Excel spreadsheet, etc.

The methods used can generally be broken down into the following:

• Using DROID for a granular and accurate count
  • DROID is a free and portable file identification tool created by The National Archives in the UK. Amongst other information, it identifies file formats and produces a link to the PRONOM technical registry which provides information about the file format.
  • DROID can be used to create file listings that include information such as file size, file type and file extensions. The file listing can be exported to a .csv file and the data manipulated to strip out any data that should not be counted. Once the rows containing only the information for the files to be counted are left, it is easy to sort by format, to count formats, to tally file sizes, etc.
  • DROID identifies file formats a number of different ways that are very useful for digital preservation, including “signature identification” which identifies signature patterns of file formats and versions.
  • This method provides the most granular inventory and was used for some smaller server-stored collections and hard drive–stored collections. Network bandwidth limitations prevented larger collections from being inventoried with the tool.
  • DROID is faster with low quantities of high file sizes (audio-visual) rather than with high quantities of small file size collections (images, textual).
  • It was used for parts of the following collections: AV, Photos, Textual, New Media, Library.
  • Case example: We used DROID on 16 hard drives containing audio-visual assets. DROID took approximately one minute to create a file listing for each 2 TB drive. From the file listing, we created an exact count of file formats, file quantities and digital size.
• Using the MS-DOS“dir” command
  • The MS-DOS “dir” command is a simple command that provides a list of contents found in a directory.
  • Here is an example of a command that will create a .txt file called “listing.txt” on the C drive/temp folder with a list of the contents of the images folder on the H drive:
    • H:\images > dir /s >c:\temp\listing.txt
    • [drive:][path]>dir /s >[location to save .txt output file]
  • It is useful where DROID would be too taxing on a network or server.
  • It requires good knowledge of Excel functions such as “Sort and Filter,” “Replace” and “Text to Columns” to distill text-formatted data into useful spreadsheet data.
  • It was used for parts of the following collections: AV, Photos, Textual.
  • Case example: We used the “dir /s” command to inventory some server spaces where DROID would be too resource-intensive, including the Audio-Visual and Textual asset groups.
• Using “Properties” for a general count
  • Often referred to as “right click – Properties.” This is a common way to find the size of a folder’s contents and the number of files located in the folder. It is not granular nor accurate since you cannot remove files you don’t wish to count (an Adobe Bridge sort file or supporting documentation such as a finding aid). However, it does provide accurate enough numbers to be used to make decisions and get a fairly good idea of file numbers and collection size.
  • It is useful for preliminary inventories to get a rough idea of file count and digital size of collections about to be inventoried by more powerful software such as DROID.
  • This method was initially used for some larger server-stored collections and hard drive–stored collections where limited network bandwidth or quantity prevented DROID from being used.
  • It was used for parts of the following collections: AV, Photos, Textual.
  • Case example: The high file counts and digital size of the Audio-Visual and Textual asset groups stored on the server were discovered using “right click – Properties.” This indicated that DROID would be inappropriate as it is resource-intensive across the server; therefore, the “dir /s” command would be a better tool.
• Using databases, finding aids and existing inventories
  • We found these tools the most useful method for counting carrier types and quantities. This information is often recorded upon ingest, so it already exists for many of CMH’s collections. Since different collections were managed by different systems at different times, information needed to be pulled from multiple resources then put together. Even where numbers were not recorded, it at least pointed us in the direction of physical carriers to be counted.
  • This method was used for some larger collections stored on physical carriers where information already existed.
  • Systems and software used were KE EMu, Vubis and Excel.
  • It was used for parts of the following collections: Textual, AV, Photos, Library
  • Case example: This was most useful in counting the physical carriers in the Textual collection. In the past, physical carriers such as diskettes and CDs were deposited and stored in folders with the paper assets. To check every folder and box would not be a good use of time. Instead, Vubis and finding aids provided the locations, and staff physically confirmed the numbers for the bigger groups of physical carriers.
• Taking a manual count for physical carriers and digital assets
  • Manual counts were used in two instances: for physical carriers and for assets with source files.
  • For some of the smaller collections of physical carriers, or where numbers pulled from existing datasets were outdated, we had to physically count the carriers. This was done primarily for the textual archives which had some older catalogue records that recorded the existence of digital carriers but not the quantity. Although very small, CMH’s collection of hard drives was also counted manually.
  • Manual counts were used for some collections where an “asset” is many files together.
  • Manual counts were also used where automated tools would count an asset’s source files but not the asset itself.
  • Case example: CMH’s very small websites collection has thousands of files, but they make up only a handful of websites. We used DROID for the source files, but the quantity of websites was counted manually.

Each asset group required a combination of the four methods to achieve the highest degree of accuracy possible. If an asset group could be counted with granular detail, it was. Although it was generally avoided, some collections stored on physical digital media formats such as MiniDisc or 3 1/2 diskette only have an estimate based on the maximum capacity of the physical media. In such cases, no file count or file formats are listed.

As a result of having to use different tools and techniques, notes were added to the inventory to give context to the numbers produced by these tools. Additionally, to provide commensurate numbers, the summary sheet of the inventory was divided between files that could be counted in detail and files that could not.

Simple manual counts and using “right click – Properties” should only be considered a first step, or triage step, that provides imprecise numbers yet still draws attention to collection risks. Sometimes there is no added value in counting every object. Instead a general number is better; more precise numbers will be generated when the risk is remedied through a project, such as migrating from physical carriers to server storage.

Case example: We have 1321 CDs in the combined Audio-Visual asset group. We know the maximum capacity of each CD is 700 MB, so the maximum digital size of the CD contents is approximately 1 TB. Inventorying the contents of each CD would be impractical. These numbers were added in the Audio-Visual asset group column of the inventory sheet with a note. Since these numbers would skew the total file size and digital size of the overall inventory totals, they were added to the “A count of each file is not possible” section of the summary along with other assets where only approximate numbers could be gathered. However, we now know that 1 terabyte would be needed to transfer the data to a server.

Inventory: Survey

Just as format variation presented a significant challenge while conducting the inventory, the application of industry practices by the diverse staff responsible for individual asset groups presented challenges in the survey section. The complexity of industry standards applied to different formats and collections, combined with the diversity of roles of individual staff members, means that survey questions can have varying levels of relevance and applicability, can be interpreted discordantly and can be answered with different language and terminology. The standards and practices of a Library collection versus an AV collection versus a New Media collection are great. Consequently, a survey question may not be applicable to certain collections. Although one staff member was appointed to coordinate the survey, it still required quite a bit of consideration for the variations in collection formats and the interpretations from responding staff.

Collection scope

Due to size limitations, only a condensed summary of the collection scope can be provided here. To explore the collection more, the inventory itself must be consulted.

Some statistics for “unprocessed material” were left out if the intention is never to accession the material. For instance, if the material had already been transferred to preservation formats, or if it was not selected for accession and was to be destroyed. Example: deposited CDs of JPEGs or RAW images are transferred to TIFF and migrated to the server, after which the CD is put on a retention schedule and placed in a cabinet in the event that originals are required in the near future.

Areas of risk

The following are identified as potential areas of risk:

• Photos
  • The “unprocessed” collections pose a risk of loss due to corruption and obsolescence of both physical assets and file formats.
  • The aging Kodak PCD collection should be re-evaluated as new tools would allow for better file format migrations than was previously done.
• Library
  • Commercial products (CD-ROMS, etc.) pose a risk as they often require obsolete software and operating systems to be accessed.
• Audio-Visual
  • Legacy duplicates, versions, obsolete formats, inconsistent numbering and organization in the past pose the greatest risk to the collection as they hamper management of the assets. (Numbering convention and collection/folder structure instituted in 2016.)
  • Large video collection stored on HDDs has no backup (will be transferred to server in 2016).
  • Physical carriers pose a risk of obsolescence, making the digital size of contents unknown.
• Textual
  • Many obsolete physical carriers dispersed through collection, contents unknown.
• New Media
  • Collection standards and practices are not formalized for the collection yet (it is a very young collection).
• Corporate Records
  • Has limited digital capacity.
• General
  • The absence of a digital asset management system is putting the digital assets at risk because it hinders preservation practices and access. (CMH will be acquiring a DAM in 2016.)
  • Multiple metadata schemas, both standardized (MARC 21, Dublin Core) and unstandardized (created in-house) hinder cross-platform sharing, slow workflows and make discovery and access complex. (New DAM has prompted CMH to map and streamline metadata.)
  • Dispersed collection with no central repository makes preservation, management and access difficult. (CMH will be transferring all external hard drive–stored assets to the server and consolidating all server-stored digital assets in 2016. Digital collections repositories for ingest, storage and access will be created for each collection.)
  • For diverse physical carriers, inconsistent/historical practices have the potential to carry forward. (CMH is reviewing older practices and procedures and adjusting where necessary. This includes recording past practices so they are known in the future.)
  • Not all collections have formal ingest, management, preservation or access standards and procedures. (CMH is reviewing practices and procedures and adjusting where necessary to fill gaps.)