py2store cookbook

Dealing with various formats, distinguishable from the keys (e.g. extensions)

Problem

You have a store that stores different kinds of things in different formats, and you are able to specify how you want to deserialize or transform the data you retrieve according to the key. You want to make a store that will take care of that logic for you.

Solution

Either tell the __getitem__ how to handle each case, or add a postget layer after the value is retrieved, specifying how to transform it before forwarding to the caller.

Say you had a LocalBinaryStore and you wanted to get your data according to the extension of the filename: As a string if the extension is .txt, as a dict if it's .json, unpickle it if it's .p, and just get the raw binary otherwise. You could do it like this:

import pickle
import json
import os
from py2store import LocalBinaryStore

class MiscStore(LocalBinaryStore):
    _postget_func_key = lambda self, k: os.path.splitext(k)[1]
    _postget_key_func = {
        '.txt': lambda v: v.decode(),
        '.p': pickle.loads,
        '.json': json.loads,
    }
    def __getitem__(self, k):
        func_key = self._postget_func_key(k)
        trans_func = self._postget_key_func.get(func_key, lambda x: x)
        return trans_func(super().__getitem__(k))

In the solution above, if you needed to make the transformation logic available when making the MiscStore object, you could include the _postget_func_key and _postget_key_func in the init argument. Consider also, the equivalent alternative solution below, that uses a class decorator to make the MiscStore, and reveals more clearly the where the store's (key-base) value transformation can be modified.

dflt_postget_key_func = {
        '.txt': lambda v: v.decode(),
        '.p': pickle.loads,
        '.json': json.loads,
    }

extract_extension = lambda k: os.path.splitext(k)[1]

def mk_postget_func(func_for_key_func=dflt_postget_key_func, 
                    func_key_from_key=extract_extension, 
                    default=lambda x: x):
    def postget(k, v):
        trans_func = func_for_key_func.get(func_key_from_key(k), default)
        return trans_func(v)
    return postget

MiscStore = wrap_kvs(LocalBinaryStore, 'MiscStore', postget=mk_postget_func())

If you had three files in a folder mydir, named a.txt, b.p and c.json, containing respectively the text hello world!, the pickled list [1, 2, 3], and the json {"i": "am", "json": 42}, you'd get

>>> s = MyStore(mydir)
>>> list(s)
['a.txt', 'b.p', 'c.json']
>>> s['a.text']
'hello world!'
>>> s['b.p']
[1, 2, 3]
>>> s['c.json']
{'i': 'am', 'json': 42}