How to save partial meta file as part of the pipeline

[turbanisch]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I would like to save a smaller version of the table created by tar_meta() to keep track of the hashes and see whether any object has changed. I am tracking the code with Git but not the derived objects. Currently, I am thinking of only keeping columns 1 to 5 and discard things like seed, errors and messages to get a more concise overview when viewing the diff.

I am assuming there is no way to modify how targets itself is writing the meta file, so I tried running tar_meta() as part of the pipeline. However, I need to pass all targets as inputs in order to have it rewrite the meta file whenever something changes but tidy-select expressions like everything() do not seem to work (plus, I would like to keep rows for functions etc. as well).

write_meta <- function(names, path) {
  tar_meta(names = names, fields = c(name, type, data, command, depend)) |> 
    arrange(desc(type)) |> 
    write_delim(path, delim = "|")
  path
}

and in _targets.R:

tar_target(meta_small, write_meta(everything(), "_targets/meta/meta-small"), format = "file")

I also tried loading the meta text file, subsetting and saving it again but this does not seem to be a smart idea since target will write the meta file only when the pipeline is completed (I assume).

Do you have any suggestions for me (or possibly even better ideas for my workflow)? The only thing I have in mind would be to create a shortcut/snippet that runs tar_make() followed by a custom write_meta() command.

[wlandau]

I think it would be pragmatic to call write_meta() outside the pipeline, i.e. always run it right after you run tar_make(). Alternatively, you could create a Git hook that forces the small metadata file to have a time stamp later than or equal to that of _targets/meta/process. This should block the commit unless your write_meta() call is up to date with the last tar_make().

[turbanisch]

Great to have your input, I really appreciate it! I now call tar_make() and tar_meta() with a single shortcut defined with the shrtcts package and the output is easy to digest at a single glance even when screen real estate is limited:

Next I will try to trigger unit tests with the same shortcut – it seems reasonable to keep those outside of the main pipeline as well. Thanks again for considering my niche use case! 🙏