UX improvements for working with metadata

[richfitz]

This deals with some of the metadata reading issues seen by Lydia when working with the malaria orderly2 project:

• add some interactive progress bars when building metadata
• don't read individual metadata files if we can use the data in the index (this is true most of the time)
• try and write out the index atomically (write then move, avoids corruption on network filesystems in parallel)
• in the unhappy situation where the index is corrupt, just throw it away

The progress bars are discussed here: https://cli.r-lib.org/articles/progress.html - they won't be shown unless rebuilding takes more than a couple of seconds, which is nice. They are however really annoying in tests so I've disabled them generally in the test setup and put in a pretty grim interaction test. I've tried this out Lydia's network sources, which take about 10-20s to read, and the behaviour is pretty good.

In a following PR I'll try out some way of writing out some mechanism faster than listing all files in the directory, which is still quite slow over the network

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (c065510) 100.00% compared to head (dc1d6cf) 100.00%.

Additional details and impacted files

@@            Coverage Diff            @@
##              main      #119   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           40        40           
  Lines         3513      3548   +35     
=========================================
+ Hits          3513      3548   +35     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[plietar]

What does %||% do ? Does it override progress only when null? Is that a standard thing?

[richfitz]

it's not standard, but very common. The implementation is in util.R but also in just about every package we write (and many other people too)!