AiiDA v2.0 and the new file repository

[sphuber]

Those of you who have been using AiiDA for serious high-throughput studies likely have noticed that AiiDA profiles with several 100k or even millions of nodes can strain your hard drive - not only in terms of disk space but also in terms of inodes as AiiDA creates a separate folder for each node, and then places any files associated with that node inside this folder (or its subfolders). Furthermore, there is currently no de-duplication of files (storing the same data twice will take twice the amount of disk space) and no compression.

This is especially serious when using slow disks (e.g. magnetic or network filesystems) and when several large projects reside on the same machine. In order to tackle this, we've started over a year ago to prepare the stage for a transition to a more efficient file repository based around the disk-objectstore package (see AiiDA enhancement proposals 1 2). During regular usage, you can expect substantial reduction in the amount of disk space used and orders of magnitude reductions in the number of inodes used. Operations that involve reading or writing large numbers of files (e.g. export / import) or even all files (e.g. backup using rsync) are much faster than before, in some cases orders of magnitude.

While the current AiiDA API for accessing files already includes some level of abstraction, the switch from a file-based repository to an object-store interface required some minor breaking changes to the interface that are detailed on the wiki. Since this is a breaking change, the version of AiiDA with the new file repository will be AiiDA v2.0 - we want to stress, however, that the transition is far less disruptive than the transition from AiiDA 0.X to AiiDA 1.0.

The largest incompatible change is that files in the file repository must be accessed through the API provided by AiiDA and can no longer be addressed directly through an absolute path on the file system. The old API that allowed this has already been deprecated in v1.0 (i.e. you should be seeing warnings if you are using it) and most aspects of the v2.0 API were already put in place. However, looking almost two years into the future is not easy and so some minor changes still had to be made to the file repository interface.

We provide instructions on how to migrate your plugin, including instructions on writing code that is compatible with v1.0 as well as v2.0 which is quite straightforward. For an example of what this migration looked like for the aiida-quantumespresso plugin, see this PR.

Now that the implementation of the new file repository is almost ready, we are starting to map out the release schedule of v2.0 (see also this discussion on the branch/release approach) and want to gauge the situation with the plugins. The first step for us is to check whether there still are any plugins that rely on directly accessing repository files on the disk. If so, let us know and can we help migrate them to the official API.

This should constitute the biggest part of the migration. We plan to remove some other deprecated functionality in v2.0 (if you're using any of these, you should already be seeing deprecation warnings in AiiDA 1.x, or search for "deprecated" in the aiida-core codebase ), and so we may want to organize a short hackathon once the AiiDA 2.0 release is imminent to get everyone across the finish line. Please let us know in case you'd like to be involved. Comments, questions and discussion about v2.0 are also very welcome.

edited by @ltalirz

[ltalirz]

mentioning @aiidateam/plugin-developers for input

[astamminger]

Hi everyone,

a short question from my side. Based on the wiki page @sphuber linked in his comment, I can only use the file handle AiiDA provides me if I want to access a file stored to the repository. So far so good, but what about post-processing software which does not allow to pass a file handle / stream directly but rather requires a path to the actual file? AFAIK pymatgen's vasp inputs / outputs modules are two such examples and this is actually the reason why I did not access files in the recommended way in aiida-cusp.

Cheers!

[sphuber]

Great question. The most immediate solution that springs to mind is to write the file from AiiDA's repository to a temporary file on disk and pass that to the third-party library. This is of course not the most efficient since you have to copy the content to disk first but there will be no other way, I am afraid. To be able to prevent the copy operation, the consuming interface will have to accept a stream.

[zhubonan]

This is a valid use case. I use the solution that @sphuber mentioned - you just need to write to a temporary folder and then use third-party tools that do not support file stream input. Although this is not going to be very efficient, especially if you have to read lots of files.

Perhaps with the new object store, it would still be possible to have a functionality to "inflate" a partial set of the repository for easy (read-only) access? This is can be an optional feature but can be useful for many cases when other tools are needed. We should avoid having the data completely "locked-in".

I also want to say that the ability of AiiDA to handle a large amount of data and high-throughput workload is also very important and much needed (for my own research at least). Glad to see the new files-store is almost ready! 👍

[ltalirz]

I think documenting how to interact with objects from the AiiDA repository "as if they were files" (which may do temporary copies under the hood) is going to be very important (e.g. also #1121).
Knowing the specific use cases helps us making sure that the interface is fit for purpose.

In the specific case of pymatgen it may be even quite straightforward to just add support for streams in their API (?).

[astamminger]

Alright, writing an intermediate file was also the first thing that came to my mind. I was just hoping I'm missing a neat little trick hidden somewhere :D The only problem with this approach, which I'm a really concerned about, is not only when accessing a lot of files but also when it comes to files of several GBs of size, i.e. files generated by AIMD simulations.

@ltalirz that was the solution I was thinking about in the first place. But then I did not find the time to look into this in detail and started to abuse the protected attributes :D

[sphuber]

It's a valid concern. I can assure you that @giovannipizzi , who wrote the disk-objectstore library that we use underneath for storing files on disk, spent a lot of effort to profile and optimize its performance. Both for use cases of writing/reading big/many files. I am pretty confident that, under the constraints of having to support high-throughput computation loads that generate a lot of files, the performance is really good. Retrieving a big file will not be a problem as long as the consumer can directly use the stream of bytes. Unfortunately, the design makes it impossible to access the file directly where it is stored on the file system since it is no longer an isolated file. It is part of a much bigger file, together with the other files stored in the repository. This solution was necessary to address the problem of AiiDA using too many inodes.

All this to say, there currently are no neat tricks that can allow one to access the files in the repository as if they were on the filesystem. We could add something like this, but this will necessarily involve copying the bytes to temporary files on the file system. I don't think there will be any way around this. But happy to be proved wrong in the case I am overlooking something.

[blokhin]

Hi @sphuber let's say, I want to read the file accessible in the old API under /data/aiida/repository/node/68/83/76ad-6495-4206-9e4b-2ba58fd94967/path/OUTPUT how do I do that using the new API? Could you show me the code?

[sphuber]

Sure @blokhin, if the file belongs to the node node, you can read it as:

node.get_object_content('OUTPUT')

or if you prefer to get a handle, you can use:

with node.open('OUTPUT') as handle:
    data = handle.read()

[rikigigi]

That's great. Few days ago I did a backup of my repository by doing a cp -av over an usb3 external hard disk, and it took ~160min for 150GB of data at the average transfer rate of 16MB/s, much less than the typical transfer velocity of the hard drive.
Is there any utility to migrate existing v1 file repository and database to aiida v2? Or it is too dangerous?

[sphuber]

Existing repositories will be fully and automatically migrated to the new implementation, just as the database is always automatically migrated for new versions of AiiDA. We are now extensively testing to make sure this is as safe as possible. As always, it is important to make a backup before migrating to a new version, just in case.