Audio Video Solution Pack Release Notes

Large Media Files in Digital Repository Contexts

• Introduction
• Ingest
  • Browser limitations
  • UX projects
  • Server settings
• Management
  • Metadata extraction + storage
  • Datastream storage and management
  • Transcoding + thumbnail extraction
  • Integration with delivery systems
• Access
  • Clients
  • Servers, platforms and protocols

---

This report will walk through the life-cycle of large media objects within a digital repository ecosystem, starting from initial ingest and dealing with software limitations to management and preservation concerns to access and playback issues. While, on the surface, similar to issues around large scientific datasets, with large media files one likely encounters additional challenges including monolithic (and un-segmentable) datastreams, heterogenous, proprietary encoding schemes, intellectual property rights (including third-party and appearance rights), a nascent community without large scale support, etc. I will look at each section from both the server-side infrastructure and client-side user experience challenges. While many of the tools and approaches are infrastructure agnostic, I will dive into some of the concerns specifically around the Fedora Commons repository architecture, the Islandora (Drupal) and Blacklight/Hydra (Ruby on Rails) stack. Much of this information is drawn from my experiences implementing media interfaces for WGBH's Open Vault website.

Ingest Challenges for Large Media Files

Dealing with large files sizes on the Internet is a challenge in "normal" web development, only compounded by the many connected components used in a digital repository context. Web browers are generally limited to 2GB (sometimes as much as 4GB for newer versions), when the file size exceeds the limit, it may be silently truncated or failed to upload. A variety of work-arounds exist, using Flash (swfupload, uploadify) Java (JUpload, postlet) or some asynchronous file-base workflow (FTP and then process). The default file upload handler does not provide the user with feedback during the upload process, which is another feature available in the above work-around (as well as some HTML + JS upload scripts, and possibly standardized in HTML5).

On the server side, different web server modules and applications may add additional file restrictions,

Apache: LimitRequestBody Directive, disabled by default. Depending on the extent of post-processing needed, installing a webserver module like mod_porter to do the heavy lifting may be advisable (lighttpd has the X-SendFile header and nginx has something equivalent).

PHP: The PHP documentation has an entire section devoted to file upload problems.

Drupal: Many of the client-side workarounds have Drupal modules for integrating client-side plugins with the Drupal upload module. For dealing with video files, there are also bundled plugins like media_mover.

Fedora: When digital repository developers are thinking about large-scale data applications, they seem to consider and prioritize many smaller objects over a few large objects. Large datastream challenges are currently tracked in JIRA and on a wiki page (which has been relatively stagnant for over a year; hopefully the core refactoring to the Spring framework wil incidentally fix some of the problems.)

Historically, until Fedora 3.2, uploaded file content was stored entirely on the heap, limiting file sizes to free memory, however until Fedora 3.4, managed datastreams larger than 2GB were not checksumable. There is an ongoing effort to globally refactor file handling for Fedora 3.5.

For implementors using older version of Fedora, externally managed and redirect datastreams were the only reliable way to handle large datastream sizes, which worked, more-or-less, for predictable use cases.

Other restrictions: In the Duracloud pilot, we discovered that cloud storage providers may also implement file storage restrictions (e.g. AWS S3 has a maximum file size of 5GB, other providers may be less), which required "chunking" large, preservation quality files into smaller pieces and reconstituting them later, which is less than ideal.

Media Management Challenges

Management challenges of large media files is split into three sections: first, what needs to happen with submitted content before it is ready to be stored for the long term; second, processes and workflows around storage; finally, processes and workflows around making the content accessible and usable.

a) Newly ingested content may arrive with significant descriptive metadata, but fragmentary or inaccurate technical metadata (much like other datastreams, but with a larger set of important information including wrappers, codecs, track information, frame size, etc). As a secondary concern in digital curation systems, robust support for media formats is frequently lacking in the traditional tools (e.g. JHOVE).

Probably the most comprehensive, open source metadata extraction tool is exiftool, a perl utility that can extract technical metadata from images, audio, video and more.

ExifTool Version Number         : 7.49
File Name                       : test.mp4
Directory                       : .
File Size                       : 2.8 MB
File Modification Date/Time     : 2010:01:01 16:54:04-05:00
File Type                       : MP4
MIME Type                       : video/mp4
Version                         : 0
Create Date                     : 2010:01:01 21:53:38
Modify Date                     : 2010:01:01 21:54:04
Time Scale                      : 90000
Duration                        : 01:25
Preferred Rate                  : 1
Preferred Volume                : 100.00%
Preview Time                    : 0 s
Preview Duration                : 0 s
Poster Time                     : 0 s
Selection Time                  : 0 s
Selection Duration              : 0 s
Current Time                    : 0 s
Next Track ID                   : 3
Track Version                   : 0
Track Create Date               : 2010:01:01 21:53:38
Track Modify Date               : 2010:01:01 21:54:04
Track ID                        : 1
Track Duration                  : 01:25
Track Layer                     : 0
Track Volume                    : 0.00%
Graphics Mode                   : srcCopy
Op Color                        : 0 0 0
Compressor ID                   : avc1
Image Width                     : 320
Image Height                    : 240
X Resolution                    : 72
Y Resolution                    : 72
Compressor Name                 : JVT/AVC Coding
Bit Depth                       : 24
Video Frame Rate                : 15.0
Media Header Version            : 0
Media Create Date               : 2010:01:01 21:53:38
Media Modify Date               : 2010:01:01 21:54:04
Media Time Scale                : 48000
Media Duration                  : 01:23
Media Language Code             : eng
Balance                         : 0
Audio Format                    : mp4a
Audio Channels                  : 2
Audio Bits Per Sample           : 16
Audio Sample Rate               : 48000
Name                            : Stereo
Handler Type                    : Metadata
Encoder                         : HandBrake 0.9.4 2009112300
Image Size                      : 320x240

Note that modern versions of exiftool can also output basic RDF/XML, which may be easier to work with within a repository context:

<?xml version='1.0' encoding='UTF-8'?>
<rdf:RDF xmlns:rdf='http://www.w3.org/1999/02/22-rdf-syntax-ns#'>

<rdf:Description rdf:about='test.mp4'
  xmlns:et='http://ns.exiftool.ca/1.0/' et:toolkit='Image::ExifTool 7.49'
  xmlns:ExifTool='http://ns.exiftool.ca/ExifTool/1.0/'
  xmlns:File='http://ns.exiftool.ca/File/1.0/'
  xmlns:QuickTime='http://ns.exiftool.ca/QuickTime/QuickTime/1.0/'
  xmlns:Track1='http://ns.exiftool.ca/QuickTime/Track1/1.0/'
  xmlns:Track2='http://ns.exiftool.ca/QuickTime/Track2/1.0/'
  xmlns:Composite='http://ns.exiftool.ca/Composite/1.0/'>
 <ExifTool:ExifToolVersion>7.49</ExifTool:ExifToolVersion>
 <File:FileName>test.mp4</File:FileName>
 <File:Directory>.</File:Directory>
 <File:FileSize>2.8 MB</File:FileSize>
 <File:FileModifyDate>2010:01:01 16:54:04-05:00</File:FileModifyDate>
 <File:FileType>MP4</File:FileType>
 <File:MIMEType>video/mp4</File:MIMEType>
 <QuickTime:Version>0</QuickTime:Version>
 <QuickTime:CreateDate>2010:01:01 21:53:38</QuickTime:CreateDate>
 <QuickTime:ModifyDate>2010:01:01 21:54:04</QuickTime:ModifyDate>
 <QuickTime:TimeScale>90000</QuickTime:TimeScale>
 <QuickTime:Duration>01:25</QuickTime:Duration>
 <QuickTime:PreferredRate>1</QuickTime:PreferredRate>
 <QuickTime:PreferredVolume>100.00%</QuickTime:PreferredVolume>
 <QuickTime:PreviewTime>0 s</QuickTime:PreviewTime>
 <QuickTime:PreviewDuration>0 s</QuickTime:PreviewDuration>
 <QuickTime:PosterTime>0 s</QuickTime:PosterTime>
 <QuickTime:SelectionTime>0 s</QuickTime:SelectionTime>
 <QuickTime:SelectionDuration>0 s</QuickTime:SelectionDuration>
 <QuickTime:CurrentTime>0 s</QuickTime:CurrentTime>
 <QuickTime:NextTrackID>3</QuickTime:NextTrackID>
 <QuickTime:MediaHeaderVersion>0</QuickTime:MediaHeaderVersion>
 <QuickTime:MediaCreateDate>2010:01:01 21:53:38</QuickTime:MediaCreateDate>
 <QuickTime:MediaModifyDate>2010:01:01 21:54:04</QuickTime:MediaModifyDate>
 <QuickTime:MediaTimeScale>90000</QuickTime:MediaTimeScale>
 <QuickTime:MediaDuration>01:25</QuickTime:MediaDuration>
 <QuickTime:MediaLanguageCode>und</QuickTime:MediaLanguageCode>
 <QuickTime:HandlerType>Video Track</QuickTime:HandlerType>
 <QuickTime:GraphicsMode>srcCopy</QuickTime:GraphicsMode>
 <QuickTime:OpColor>0 0 0</QuickTime:OpColor>
 <QuickTime:CompressorID>avc1</QuickTime:CompressorID>
 <QuickTime:ImageWidth>320</QuickTime:ImageWidth>
 <QuickTime:ImageHeight>240</QuickTime:ImageHeight>
 <QuickTime:XResolution>72</QuickTime:XResolution>
 <QuickTime:YResolution>72</QuickTime:YResolution>
 <QuickTime:CompressorName>JVT/AVC Coding</QuickTime:CompressorName>
 <QuickTime:BitDepth>24</QuickTime:BitDepth>
 <QuickTime:VideoFrameRate>15.0</QuickTime:VideoFrameRate>
 <QuickTime:MediaHeaderVersion>0</QuickTime:MediaHeaderVersion>
 <QuickTime:MediaCreateDate>2010:01:01 21:53:38</QuickTime:MediaCreateDate>
 <QuickTime:MediaModifyDate>2010:01:01 21:54:04</QuickTime:MediaModifyDate>
 <QuickTime:MediaTimeScale>48000</QuickTime:MediaTimeScale>
 <QuickTime:MediaDuration>01:23</QuickTime:MediaDuration>
 <QuickTime:MediaLanguageCode>eng</QuickTime:MediaLanguageCode>
 <QuickTime:HandlerType>Audio Track</QuickTime:HandlerType>
 <QuickTime:Balance>0</QuickTime:Balance>
 <QuickTime:AudioFormat>mp4a</QuickTime:AudioFormat>
 <QuickTime:AudioChannels>2</QuickTime:AudioChannels>
 <QuickTime:AudioBitsPerSample>16</QuickTime:AudioBitsPerSample>
 <QuickTime:AudioSampleRate>48000</QuickTime:AudioSampleRate>
 <QuickTime:Name>Stereo</QuickTime:Name>
 <QuickTime:HandlerType>Metadata</QuickTime:HandlerType>
 <QuickTime:Encoder>HandBrake 0.9.4 2009112300</QuickTime:Encoder>
 <Track1:TrackVersion>0</Track1:TrackVersion>
 <Track1:TrackCreateDate>2010:01:01 21:53:38</Track1:TrackCreateDate>
 <Track1:TrackModifyDate>2010:01:01 21:54:04</Track1:TrackModifyDate>
 <Track1:TrackID>1</Track1:TrackID>
 <Track1:TrackDuration>01:25</Track1:TrackDuration>
 <Track1:TrackLayer>0</Track1:TrackLayer>
 <Track1:TrackVolume>0.00%</Track1:TrackVolume>
 <Track1:ImageWidth>320</Track1:ImageWidth>
 <Track1:ImageHeight>240</Track1:ImageHeight>
 <Track2:TrackVersion>0</Track2:TrackVersion>
 <Track2:TrackCreateDate>2010:01:01 21:53:38</Track2:TrackCreateDate>
 <Track2:TrackModifyDate>2010:01:01 21:54:04</Track2:TrackModifyDate>
 <Track2:TrackID>2</Track2:TrackID>
 <Track2:TrackDuration>01:23</Track2:TrackDuration>
 <Track2:TrackLayer>0</Track2:TrackLayer>
 <Track2:TrackVolume>100.00%</Track2:TrackVolume>
 <Composite:ImageSize>320x240</Composite:ImageSize>
</rdf:Description>
</rdf:RDF>

For preservation workflows that may want to use JHOVE (or similar applications) for metadata, the FITS project combines output from multiple metadata extraction tools (including JHOVE and exiftool) into a single interface.

<?xml version="1.0" encoding="UTF-8"?>
<fits xmlns="http://hul.harvard.edu/ois/xml/ns/fits/fits_output" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://hul.harvard.edu/ois/xml/ns/fits/fits_output http://hul.harvard.edu/ois/xml/xsd/fits/fits_output.xsd" version="0.3.1" timestamp="11/10/10 08:43">
  <identification status="SINGLE_RESULT">
    <identity format="ISO Media, MPEG v4 system, version 2" mimetype="video/mp4">
      <tool toolname="Exiftool" toolversion="7.74" />
      <version toolname="Exiftool" toolversion="7.74">0</version>
    </identity>
  </identification>
  <fileinfo>
    <lastmodified toolname="Exiftool" toolversion="7.74" status="SINGLE_RESULT">2010:01:01 16:54:04-05:00</lastmodified>
    <created toolname="Exiftool" toolversion="7.74" status="SINGLE_RESULT">2010:01:01 21:53:38</created>
    <filename toolname="OIS File Information" toolversion="0.1" status="SINGLE_RESULT">test.mp4</filename>
    <size toolname="OIS File Information" toolversion="0.1" status="SINGLE_RESULT">2963484</size>
    <md5checksum toolname="OIS File Information" toolversion="0.1" status="SINGLE_RESULT">df38217609323522909e25a37e444d26</md5checksum>
    <fslastmodified toolname="OIS File Information" toolversion="0.1" status="SINGLE_RESULT">1262382844000</fslastmodified>
  </fileinfo>
  <filestatus />
  <metadata>
    <video>
      <duration toolname="Exiftool" toolversion="7.74" status="SINGLE_RESULT">01:25</duration>
      <frameRate toolname="Exiftool" toolversion="7.74" status="SINGLE_RESULT">15.0</frameRate>
      <bitDepth toolname="Exiftool" toolversion="7.74" status="SINGLE_RESULT">24</bitDepth>
      <sampleRate toolname="Exiftool" toolversion="7.74" status="SINGLE_RESULT">48000</sampleRate>
      <channels toolname="Exiftool" toolversion="7.74" status="SINGLE_RESULT">2</channels>
      <imageWidth toolname="Exiftool" toolversion="7.74" status="SINGLE_RESULT">320</imageWidth>
      <imageHeight toolname="Exiftool" toolversion="7.74" status="SINGLE_RESULT">240</imageHeight>
      <xSamplingFrequency toolname="Exiftool" toolversion="7.74" status="SINGLE_RESULT">72</xSamplingFrequency>
      <ySamplingFrequency toolname="Exiftool" toolversion="7.74" status="SINGLE_RESULT">72</ySamplingFrequency>
      <creatingApplicationName toolname="Exiftool" toolversion="7.74" status="SINGLE_RESULT">JVT/AVC Coding</creatingApplicationName>
    </video>
  </metadata>
</fits>

From this data, there are several key pieces that are needed for preservation and access workflows:

• Image Height/Image Width
• Duration
• Compressor ID/Compressor Name/Encoder

Because the media datastreams are arguably more heterogenous than other relatively standardized formats, having a tool that merges so-called repository workflows and popular, well-supported community tools may make it easier to accept obscure or nascent formats as quickly as possible (e.g. the brand new WebM video format from Google)

While the tool-based metadata may be sufficient, converting the output into a standard format may be desirable, however there is little agreement on a universal standard for storing it.

• (VideoMD)[http://lcweb2.loc.gov/mets/Schemas/VMD.xsd] is part of a series of metadata specifications from the Library of Congress and does a good job capturing a base-level for information about media assets.

• (PBCore)[http://pbcore.org] is a nascent standard (currently in 2.0 development) from the U.S. Corporation for Public Broadcasting, intendent primarily for broadcast and media production workflows. While not just limited to technical metadata, the PBCore Instantiation element has elements for encoding wrappers, tracks, and codec.

• (EBUCore)[http://tech.ebu.ch/lang/en/MetadataEbuCore], similar to PBCore, is a metadata standard created by the European Broadcasting Union, with a more developed structure and controlled vocabulary.

• (MPEG-7)[http://mpeg.chiariglione.org/standards/mpeg-7/mpeg-7.htm]/(MPEG-21)[http://mpeg.chiariglione.org/standards/mpeg-21/mpeg-21.htm] These are two standards developed by the Moving Picture Experts Group, which are extremely technical, but have found acceptance within some communities. Perhaps due to the difficulty in authoring the MPEG metadata, there is a growing set of (community-developed tools)[http://www.multimedia-metadata.info/Software%20and%20Tools]

• SMPTE DMS-1

• (SMPTE Metadata Dictionary RP210)[http://www.smpte-ra.org/mdd/RP210v2-1merged-020507b.xls] (XLS)

b) Asset management and storage

Storing large media files isn't much different than regular large datastream storage, however there are a couple additional considerations. First, segmenting large media files may significantly increase access time

c) In order to support access, preservation-quality media files may need to be transcoded. While it can be an entirely automated process (and often is), media quality can be improved with a little manual intervention.

• Quality
• Watermarking
• Hinting/m3p8/etc

The art of the thumbnail... If you're lucky, the exiftool-extracted metadata could provide poster frames information (Preview Time/Poster Time).

Getting it into delivery systems (streaming servers, CDNs, and more..)

mod_rewrite hack.

Access Challenges

Fractured client market

• significant focus on web-friendly playback, 3rd party providers..
• few good alternatives for arbitrary playback, e.g. Quicktime (poorly neglected..)

Awkward delivery mechanisms