seq2ffv1/rawcooked - Multi reel image sequence support

[kieranjol]

This will attempt to support an input where there are folders:

• blabla_reel1
• blabla_reel2
  etc
  where each folder contains images with optional audio.

this will create an output that looks like this - where each reel has its own MKV:

├── logs
│   ├── 8bb43275-ea84-4c25-9422-db6d98e6bd7a_reel1.mkv_manifest.md5
│   ├── 8bb43275-ea84-4c25-9422-db6d98e6bd7a_reel2.mkv_manifest.md5
│   ├── 8bb43275-ea84-4c25-9422-db6d98e6bd7a_reel3.mkv_manifest.md5
│   ├── 8bb43275-ea84-4c25-9422-db6d98e6bd7a_reel3.mkv_rawcooked.log
│   ├── 8bb43275-ea84-4c25-9422-db6d98e6bd7a_sip_log.log
│   ├── 8bb43275-ea84-4c25-9422-db6d98e6bd7a_source_dfxml.xml_manifest.md5
│   ├── 8bb43275-ea84-4c25-9422-db6d98e6bd7a_source_mediainfo.xml_manifest.md5
│   ├── 8bb43275-ea84-4c25-9422-db6d98e6bd7a_source_mediatrace.xml_manifest.md5
│   ├── metadata_manifest.md5
│   └── multi_reel_manifest-md5.txt_manifest.md5
├── metadata
│   ├── 8bb43275-ea84-4c25-9422-db6d98e6bd7a_reel1.mkv_mediainfo.xml
│   ├── 8bb43275-ea84-4c25-9422-db6d98e6bd7a_reel1.mkv_mediatrace.xml
│   ├── 8bb43275-ea84-4c25-9422-db6d98e6bd7a_reel2.mkv_mediainfo.xml
│   ├── 8bb43275-ea84-4c25-9422-db6d98e6bd7a_reel2.mkv_mediatrace.xml
│   ├── 8bb43275-ea84-4c25-9422-db6d98e6bd7a_reel3.mkv_mediainfo.xml
│   ├── 8bb43275-ea84-4c25-9422-db6d98e6bd7a_reel3.mkv_mediatrace.xml
│   └── supplemental
│       ├── 8bb43275-ea84-4c25-9422-db6d98e6bd7a_source_dfxml.xml
│       ├── 8bb43275-ea84-4c25-9422-db6d98e6bd7a_source_mediainfo.xml
│       ├── 8bb43275-ea84-4c25-9422-db6d98e6bd7a_source_mediatrace.xml
│       └── multi_reel_manifest-md5.txt
└── objects
    ├── 8bb43275-ea84-4c25-9422-db6d98e6bd7a_reel1.mkv
    ├── 8bb43275-ea84-4c25-9422-db6d98e6bd7a_reel2.mkv
    └── 8bb43275-ea84-4c25-9422-db6d98e6bd7a_reel3.mkv

[kieranjol]

This is looking pretty good. The reversibility check is working for multi-reeler.

Single reelers are broken at the moment, but I'll fix those.

[kieranjol]

ok single reelers are fixed, so now it's just

• testing
• cleanup

[kieranjol]

To fix - only one rawcooked logfile appears in the logs.

[kieranjol]

This looks good to go.
The temporary manifests are now removed.
There is also a textfile containing information about the actual package and some version control on the package as well.

/oe10004
├── 7756fb2d-8d43-4f69-8164-2c1c701af1a8
│   ├── logs
│   │   ├── 7756fb2d-8d43-4f69-8164-2c1c701af1a8_reel1.mkv_rawcooked.log
│   │   ├── 7756fb2d-8d43-4f69-8164-2c1c701af1a8_reel2.mkv_rawcooked.log
│   │   ├── 7756fb2d-8d43-4f69-8164-2c1c701af1a8_reel3.mkv_rawcooked.log
│   │   └── 7756fb2d-8d43-4f69-8164-2c1c701af1a8_sip_log.log
│   ├── metadata
│   │   ├── 7756fb2d-8d43-4f69-8164-2c1c701af1a8_reel1.mkv_mediainfo.xml
│   │   ├── 7756fb2d-8d43-4f69-8164-2c1c701af1a8_reel1.mkv_mediatrace.xml
│   │   ├── 7756fb2d-8d43-4f69-8164-2c1c701af1a8_reel2.mkv_mediainfo.xml
│   │   ├── 7756fb2d-8d43-4f69-8164-2c1c701af1a8_reel2.mkv_mediatrace.xml
│   │   ├── 7756fb2d-8d43-4f69-8164-2c1c701af1a8_reel3.mkv_mediainfo.xml
│   │   ├── 7756fb2d-8d43-4f69-8164-2c1c701af1a8_reel3.mkv_mediatrace.xml
│   │   ├── film_scan_aip_documentation.txt
│   │   └── supplemental
│   │       ├── 7756fb2d-8d43-4f69-8164-2c1c701af1a8_source_dfxml.xml
│   │       ├── 7756fb2d-8d43-4f69-8164-2c1c701af1a8_source_mediainfo.xml
│   │       ├── 7756fb2d-8d43-4f69-8164-2c1c701af1a8_source_mediatrace.xml
│   │       └── multi_reel_manifest-md5.txt
│   └── objects
│       ├── 7756fb2d-8d43-4f69-8164-2c1c701af1a8_reel1.mkv
│       ├── 7756fb2d-8d43-4f69-8164-2c1c701af1a8_reel2.mkv
│       └── 7756fb2d-8d43-4f69-8164-2c1c701af1a8_reel3.mkv
└── 7756fb2d-8d43-4f69-8164-2c1c701af1a8_manifest.md5

[kieranjol]

I'm going to merge as I think that aside from the aip_documentation-, this is fairly uncontroversial. As for the AIPdocumentation, I'll paste here - but the AIPs for film scans alone will change with this pull request (temporary manifests are now removed) - so any kind of version control and documentation is better than the nothing that currently exists.

v1.1 Image Sequence AIP
This textfile - Generic description of the features of the Archival Information Package.
Changelog: v1.1 - temporary manifests removed from logs_directory.

parent_directory - either OE (oe##### - Object_Entry Spectrum Collections Management( Identifier or Accession Number (aaa####)
├── uuid_directory - this directory contains the AIP. The identifier is  for the entire representation/package.
│   ├── logs - logs that are generated via IFIscripts. This can include process logs for FFmpeg/rawcooked and other tools.
│   │   └── uuid_sip_log.log - the most important log - this is a rolling log for all processes that IFIscripts will execute.
│   ├── metadata - for AV objects - mediainfo and mediatrace XML files. For non-AV objects, Siegfried/PRONOM and EXIFTOOL JSON reports. Once the package is accessioned, there will be a Digital Forensics XML minus the hashes. This will also contain PBCore/Technical metadata CSV and IFI Filmographic CSV in the accessioned package.
│   │   └── supplemental - For normalised objects, checksum manifests, mediainfo, mediatrace and Digital Forensics XML regarding the source object.
│   └── objects - the actual focus of preservation. 
└── uuid_directory_manifest.md5 - MD5 manifest for entire package.
└── uuid_directory_manifest-sha512.txt - This will only be generated at the accessioning phase. SHA512 manifest for the entire package.

Example tree structure and explanation for a film scan normalised to FFV1/Matroska via seq2ffv1.py. 
Additional files appear when the accession/batchaccession process runs.

oe0001 | parent_directory
├── da990f5e-9846-4a8c-9792-3ee677d50591 |uuid_directory
│   ├── logs | directory for logs generated via IFIscripts
│   │   ├── da990f5e-9846-4a8c-9792-3ee677d50591_reel3.mkv_rawcooked.log | rawcooked_log via ffmpeg
│   │   └── da990f5e-9846-4a8c-9792-3ee677d50591_sip_log.log | central log for all IFIscripts processes
│   ├── metadata | directory for metadata
│   │   ├── aaa0001_pbcore.csv | ONLY APPEARS WITH ACCESSION.py/BATCHACCESSION.py - CSV containing metadata ready for ingest into IFI technical database. Loosely based on PBCore, with many custom IFI legacy fields.
│   │   ├── AF12345_filmographic.csv | ONLY APPEARS WITH ACCESSION.py/BATCHACCESSION.py - CSV containing descriptive metadata ready for ingest into IFI Filmographic/descriptive database. Loosely based on the original FIAF cataloguing guidelines.
│   │   ├── da990f5e-9846-4a8c-9792-3ee677d50591_dfxml.xml | ONLY APPEARS WITH ACCESSION.py/BATCHACCESSION.py - file system metadata in Digital Forensics XML format for all files within the package.
│   │   ├── da990f5e-9846-4a8c-9792-3ee677d50591_reel1.mkv_mediainfo.xml | technical metadata in XML format via mediainfo
│   │   ├── da990f5e-9846-4a8c-9792-3ee677d50591_reel1.mkv_mediatrace.xml | granular technical metadata in XML format via mediainfo
│   │   ├── da990f5e-9846-4a8c-9792-3ee677d50591_reel2.mkv_mediainfo.xml | technical metadata in XML format via mediainfo
│   │   ├── da990f5e-9846-4a8c-9792-3ee677d50591_reel2.mkv_mediatrace.xml granular technical metadata in XML format via mediainfo
│   │   ├── da990f5e-9846-4a8c-9792-3ee677d50591_reel3.mkv_mediainfo.xml | technical metadata in XML format via mediainfo
│   │   ├── da990f5e-9846-4a8c-9792-3ee677d50591_reel3.mkv_mediatrace.xml | granular technical metadata in XML format via mediainfo
│   │   └── supplemental | directory for metadata for the source object, prior to normalisation
│   │       ├── da990f5e-9846-4a8c-9792-3ee677d50591_source_dfxml.xml | file system metadata in Digital Forensics XML format for the source DPX sequence.
│   │       ├── da990f5e-9846-4a8c-9792-3ee677d50591_source_mediainfo.xml | technical metadata in XML format for the source DPX sequence.
│   │       ├── da990f5e-9846-4a8c-9792-3ee677d50591_source_mediatrace.xml | granulartechnical metadata in XML format for the source DPX sequence.
│   │       └── multi_reel_manifest-md5.txt | md5 checksum manifest for the source DPX sequence
│   └── objects | directory for the preservation
│       ├── da990f5e-9846-4a8c-9792-3ee677d50591_reel1.mkv | FFV1/Matroska normalisation of image sequence.
│       ├── da990f5e-9846-4a8c-9792-3ee677d50591_reel2.mkv | FFV1/Matroska normalisation of image sequence.
│       └── da990f5e-9846-4a8c-9792-3ee677d50591_reel3.mkv | FFV1/Matroska normalisation of image sequence.
├── da990f5e-9846-4a8c-9792-3ee677d50591_manifest.md5 | MD5 checksum manifest for the whole package.
└── da990f5e-9846-4a8c-9792-3ee677d50591_manifest-sha512.txt |  ONLY APPEARS WITH ACCESSION.py/BATCHACCESSION.py SHA512 checksum manifest for the whole package.