tar index as json file

[pgierz]

This adds an index file caching for tar filesystems, useful when you have listed the tar already and don't want to re-examine it.

At the moment I use JSON for this, but I'm not sure if that is the best idea or if something else would be better.

[pgierz]

Still a draft, I need to tweak a few things, I'll mark it as ready soon.

[pgierz]

It might be sensible to also check against a "Truthy" value here and default to a filename that is similar to the name of the actual tar file. At the moment, I assumed that the user was giving a file-like string.

[pgierz]

Alright, this seems to work the way I would imagine it to. I'd be very happy to get some feedback if this is useful or not 🙂

[martindurant]

Thanks for having a go at this! Please ping me next week for a thorough review.

Before proceeding, I wonder if there are any existing TAR index formats and/or index file naming conventions that we should attempt to follow. I agree that JSON is a sensible alternative if not.

You might also be interested in https://github.com/pauldmccarthy/indexed_gzip , which allows for starting to read at checkpoints throughout the GZIP stream rather than always reading from the beginning as normal. If the two ideas were combined, you could random-access to the beginning of any file in the archive.

[pgierz]

Before proceeding, I wonder if there are any existing TAR index formats and/or index file naming conventions that we should attempt to follow. I agree that JSON is a sensible alternative if not.

I wasn't able to find anything, and perhaps the most interesting answer was here, which "long story short" seems to indicate that indexed tar with standard GNU tools is not supported, and therefore, there is also no standard for naming such an index.

You might also be interested in https://github.com/pauldmccarthy/indexed_gzip , which allows for starting to read at checkpoints throughout the GZIP stream rather than always reading from the beginning as normal. If the two ideas were combined, you could random-access to the beginning of any file in the archive.

Perhaps also this? https://github.com/colon3ltocard/pyindexedtar

I would have a try at adding some of the ideas there, and let you know later in the week.

[martindurant]

No additional index file, the archive should contain the index and be 'all inclusive'

(from pyindexedtar)

I don't think this is a reasonable requirement. We normally are operating in the regime where the original data can't be changed, so the index needs to go elsewhere, possibly in a totally different storage location from the original. This is the kind of thing that kerchunk does, and indeed the idea of making a filesystem out of buffers within archived files has been a longstanding aim there - similar but different from the case here of the offsets to files.

[martindurant]

I think the implementation is fine aside a couple of comments.

Would appreciate a test (that the index file gets created and looks correct) and text to describe what happens.

Extension to tar.gz or other types can follow later, if there is interest.

[martindurant]

I think it's reasonable to only allow str here. That would allow for the index to be stored anywhere (in another storage backend), whereas Path is local specific. If you use fsspec.open(), then you could include anything that it can handle.

[pgierz]

I like that idea. Does fsspec have some way to check if a file exists, akin to pathlib.Path(...).exists()? I could do try/except and catch for a FileNotFoundError, unless there is already a way to check that.

[martindurant]

It would be two-step, create a filesystem instance (fsspec.url_to_fs), and use .exists on that - so probably fine to use try/except here. We have discussed elsewhere whether we should have top-level operations like exists(url) that dispatch to the implementations like fsspec.open, but nothing has been done yet (cf fsspec.generic)

[martindurant]

fsspec.open(self.index_store)

[martindurant]

fsspec.open(self.index_store, "wt")