slides/wednesday_02.html

<!DOCTYPE html>
<html>
  <head>
    <title>Digital Summer School 2024: TUE04</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
    <link rel="stylesheet" href="../style.css">  </head>
  <body>
    <textarea id="source">


class: center, middle, titlepage
### WED02: *Fixity and storage.*

---
class: contentpage
### **Agenda**


1. Intro to Checksums
2. Python hashlib
3. Bagit
4. Tarwrapping
5. OAIS
6. PREMIS
7. Perceptual hashing


---
class: contentpage
### **1. Intro to Checksums**

Checksums are "fixed width" (i.e. always the same length) digital fingerprints.

MD5 for a DPX file!

```sh
md5sum media/film_scan/0088600.dpx 
e0c39a37430fbacb9b7fb634580a1858
```

---
class: contentpage
### **1. Intro to Checksums**

Different operating systems have different ways of generating checksums, in order of complexity.

macOS

```sh
md5 media/film_scan/0088600.dpx 
```

Linux

```sh
md5sum media/film_scan/0088600.dpx 
```

Windows

```sh
CertUtil -hashfile media/film_scan/0088600.dpx MD5
```

<tspan style="color: orange"><b>PRACTICAL:</b></tspan> Hash a file in the command line.

---
class: contentpage

### **1. Intro to Checksums**

An important concept of checksums is that they are predictable, with the same input they will produce the same output every time.

```sh
md5sum media/film_scan/0088600.dpx 
e0c39a37430fbacb9b7fb634580a1858
```


```sh
md5sum media/film_scan/0088600.dpx 
e0c39a37430fbacb9b7fb634580a1858
```

```sh
md5sum media/film_scan/0088600.dpx 
e0c39a37430fbacb9b7fb634580a1858
```

```sh
md5sum media/film_scan/0088600.dpx 
e0c39a37430fbacb9b7fb634580a1858
```


---
class: contentpage

### **1. Intro to Checksums**

However, any slight change in the input will, using the most popular checksumming variants, produce a completely different output. This is to say we have no indication of how much it has changed, just that it has changed.


```sh
md5sum media/film_scan/0088600.dpx 
e0c39a37430fbacb9b7fb634580a1858
```


```sh
md5sum media/film_scan/0088601.dpx 
ff329821d9330b9688148da8abc4147e
```


```sh
md5sum media/film_scan/0088600.dpx 
e0c39a37430fbacb9b7fb634580a1858
```


```sh
md5sum media/film_scan/0088601.dpx 
ff329821d9330b9688148da8abc4147e
```


---
class: contentpage

### **1. Intro to Checksums**

Checksums are not reversible, you cannot recreate the source from the checksum, but you can verify if the source is unchanged.


---
class: contentpage

### **1. Intro to Checksums**


Note on terminology, generally "hashes" are used in the context of digital security/authentication and "checksums" for checking file fixity, but both use the same concept and are generally used interchangeably.

"File fixity" meaning that the file is fixed, unchanged, which is obviously a key concern for digital archives.


---
class: contentpage

### **1. Intro to Checksums**

One key difference between the use of these technologies for security and for digital archiving is that digital archiving is mostly concerned with accidental breakage (file truncation) while net security is dealing with intentional malicious behaviour (hackers). This means archives generally are using what would be considered "out of date" hashing algorithms.

MD5, which we have seen so far, is pretty ubiquitous in the digital preservation community. It has been around since 1991 and is considered "good enough"


What we mean by "good enough" - our main concern here are "collisions", where two different sources generate the same hash. What are the odds!

---
class: contentpage

### **1. Intro to Checksums**


There are `340,282,366,920,938,463,463,374,607,431,768,211,456` possible MD5 hashes, sounds pretty good! 

But once we factor in "The Birthday Paradox", the chance of a single collision is 50% every `18,446,744,073,709,551,616` hashes.

Birthday Paradox tangent: https://pudding.cool/2018/04/birthday-paradox/

MD5 is "good enough" for archives, in that it there are currently no AV archives holding this many files (not even close).

---
class: contentpage

### **1. Intro to Checksums**

A downside of using MD5 is that it can be quite slow to hash files, which can add up if you are generating checksums across an entire system.

Other algorithms like xxhash are much quicker, but will not work so well with existing tools, or might be more likely to allow "collisions".

https://www.dpconline.org/docs/technology-watch-reports/2399-twgn-checksums-addis/file


---
class: contentpage

### **2. Python hashlib**

As we saw before, we can generate hashes directly from the command line, but there are reasons we might want to do this in Python. For example, as part of a more complex media processing script, or if we want to hash a large directory of files and find where there are matches in a database.

We could use subprocess, but Python already comes with a handy inbuilt library called `hashlib`.

Example of hashing a file.

```python
import hashlib
import pathlib

filepath = pathlib.Path.cwd() / 'media' / 'film_scan' / '0088600.dpx'

with open(filepath, 'rb') as file:  
    file_content = file.read()  
    result = hashlib.md5(file_content).hexdigest()  
    print(filepath.name, result)
```


<tspan style="color: orange"><b>PRACTICAL:</b></tspan> Hash a file using Python.


---
class: contentpage

### **2. Python hashlib**

Example of processing a whole directory.

```python
import hashlib
import pathlib

dpx_frames = pathlib.Path.cwd() / 'media' / 'film_scan'
for filepath in dpx_frames.iterdir():
    with open(filepath, 'rb') as file:  
        result = hashlib.md5(file.read()).hexdigest()  
        print(filepath.name, result)


```

<tspan style="color: orange"><b>PRACTICAL:</b></tspan> Hash a whole directory using Python.


---
class: contentpage

### **3. Bagit**


BagIt is an open-source project developed by the Library of Congress to provide structures and tools, typically used to verify that collections of files have remained unchanged.

You could absolutely write your own Python scripts to create, store and validate checksums, BagIt is just a convenience method for fundamental digital archiving processes. 

You can run BagIt from the command line, 

```sh
bagit.py /directory/to/bag
```

but there is also an excellent Python library.

```python
import bagit
```

As we don't have the 'bagit' library installed, we will get a `ModuleNotFoundError: No module named 'bagit'` error in Python.


---
class: tangentpage

### **Tangent 1: installing external libraries.**

Now, unlike the other libraries we have looked at today, BagIt is not pre-installed with Python (start a petition!), so we have to install it ourselves.


For this you will need `pip` which is the Python management systems for libraries.

We can check out libraries using `pip freeze` to verify that this is true!

The easiest way to install is with pip, which is the recommended Python library manager.

```sh
pip install bagit
```

Note that we can also install a specific version if we so choose. This can be useful if there are changes to the library, and we want to be confident everyone is running the same thing.

```sh
pip install bagit==1.8.1
```

`import bagit` should now work.


---
class: tangentpage

### **Tangent 1: installing external libraries.**

Just be aware that we are now running other people's code on our machines, there is no 100% guarantee that they have not programmed the library to behave in undesirable ways.

From a security perspective, worth considering installing libraries which either have a lot of "stars" on GitHub, or are from people or organisations we trust (like Library of Congress).


---
class: contentpage

### **3. Bagit**


Walkthrough of BagIt for Python!

Copy the `film_scan` folder to a `bag_test` folder.

```python
import bagit
import pathlib

bagit.make_bag(pathlib.Path.cwd() / 'bag_test', checksums=['md5'])
```


---
class: contentpage

### **3. Bagit**


<tspan style="color: orange"><b>PRACTICAL:</b></tspan> Tour of what a "bag" looks like.


---
class: contentpage

### **3. Bagit**


Danger of "recursive bags" if process is run multiple times. 


<tspan style="color: orange"><b>QUESTION:</b></tspan> How would we use an "if" and "exists" to not create recursive bags?.


---
class: contentpage

### **3. Bagit**


We can also load our bag and validate it is correct, meaning intact from when we bagged it.

```python
bag = bagit.Bag(str(pathlib.Path.cwd() / 'bag_test'))
if bag.is_valid():
    print('yes, we are good')
else:
    print('something went wrong')
```

Adjust something to make this fail!


---
class: contentpage

### **4. Tarwrapping**

Great, so we now have our "bag", but what do we do with it?


Generally speaking, many archives will then "wrap the bag". 
TAR is a very old option (back to 1979) and widely supported means of wrapping files for storage or transport.


As you have now heard many times, we can either create TARs from the command line on a Unix-machine (Linux/macOS) or using Python (OS-agnostic)


```sh
tar cfv archive.tar bag_test/
```


---
class: contentpage

### **4. Tarwrapping**

Resons for TAR wrapping directories:
- It is much easier to move files once they are packaged (and quicker).
- It is easier to keep track of changes at the top level by hashing the TAR.

Downsides, which you should plan for:
- Once files are in a TAR you may lose visibility on what the contents are (save your manifest externally!).
- If the TAR becomes corrupt or is modified it may not be obvious which files are affected.


---
class: contentpage


### **4. Tarwrapping**


Python has an inbuilt `tarfile` library!

```python
import pathlib
import tarfile

with tarfile.open(pathlib.Path.cwd() / 'archive.tar', "w") as tar_file:
    bag_files = pathlib.Path.cwd() / 'bag_test'
    for file in bag_files.iterdir():
        tar_file.add(file)
```

<tspan style="color: orange"><b>PRACTICAL:</b></tspan> Tar the media file (only if you have enough space)!.


---
class: contentpage

### **5. OAIS**


Now is probably a good moment to raise the concept of OAIS, or Open Archival Information System.

This is a standard (although, importantly, conceptual standard) around how archives should store their digital material. 

I would recommend the DPC introduction to OAIS if you are interested: https://www.dpconline.org/docs/technology-watch-reports/1359-dpctw14-02/file.

OAIS describes three stages in the archival object's lifecycle:
- The Submission Information Package (aka SIP), the package entering archival control.
- The Archival Information Package (aka AIP), the package in archival control.
- The Dissemination Information Package (aka DIP), the package leaving archival control.


Because this is a conceptual or reference model there is some flexibility to interpret what an implementation should look like, but generally understood meaning that files should be stored in a state of assured fixity, with sufficient metadata to assess format conformance and playability.


---
class: contentpage

### **6. PREMIS**

PREMIS is another concept originating from Library of Congress, which defines an expected model for expressing information around preservation activity. This includes defining the entities of those involved in preservation activity: Objects, Events, Agents and Rights.

These entities also have expected properties, instigating the collection of the sort of metadata required to properly contextualise digital preservation activity.

If you were so inspired you could literally express this using their XML or OWL schemas, otherwise it could be a good reference to understand what data you should be consider collecting: who created this file, when, how? Was the filename changed? What are the access rights?


https://www.loc.gov/standards/premis/v3/index.html


---
class: contentpage

### **7. Perceptual hashing, similarity matching**

Before we completely finish up with checksums and hashing, a "fun" variant is "perceptual hashing".


---
class: contentpage

### **7. Perceptual hashing, similarity matching**


Perceptual hashing has a different goal and is worth introducing as a concept. Our previous examples of hashing would produce completely different results if there were any changes major or minor, whereas perceptual hashes should differ proportional to the amount of change.


While checksum hashing for file validation is ubiquitous, hashing for content similarity is more unusual, and only a few institutions are currently doing so, to my knowledge.

There are some Python libraries for image perceptual hashing specifically, such as `imagehash`.


---
class: contentpage

### **7. Perceptual hashing, similarity matching**

Now we can run through our JPGs, find a random file, and then return the closest and least close files from the scan.

```python
import imagehash, pathlib, random, subprocess, tqdm
from PIL import Image

frames = pathlib.Path.cwd() / 'media' / 'jpg'
jpeg_files = [x for x in frames.rglob('*') if x.suffix == '.jpg']
random_jpg = random.choice(jpeg_files)
random_jpg_phash = imagehash.average_hash(Image.open(str(random_jpg)))

print(random_jpg, random_jpg_phash)

result = []
for jpg in tqdm.tqdm(jpeg_files):
    compare_hash = imagehash.average_hash(Image.open(str(jpg)))
    result.append([random_jpg_phash - compare_hash, jpg])

result = sorted(result, key=lambda x: x[0])

print('most closest file', result[1])
print('most distant file', result[-1])

```

---
class: contentpage

### **7. Perceptual hashing, similarity matching**

Where would this be useful for an archive?

If someone gives you digital material, it is useful to be able to check if it is already in your collection. 

Hashing processes like MD5 will tell you if you already have *exactly* the same file, but this will tell you if you have something which is just very similar, say a JPEG and a PNG of the same image, or a compressed vs uncompressed image.

I am also personally very interested in its use in determining where visual work has been sampled or featured in other works, to identify where clips or sections of video are present in other moving image files.


    </textarea>
    <script src="https://remarkjs.com/downloads/remark-latest.min.js" type="text/javascript"></script>
    <script type="text/javascript">var slideshow = remark.create({ratio: "16:9"});</script>
  </body>
</html>