diff --git a/404.html b/404.html
index cdad61fe..57780eb8 100644
--- a/404.html
+++ b/404.html
@@ -20,7 +20,7 @@
 
     <a class="navbar-brand me-2" href="https://mrc-ide.github.io/orderly2/index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/LICENSE-text.html b/LICENSE-text.html
index 63598d1d..5d2cd334 100644
--- a/LICENSE-text.html
+++ b/LICENSE-text.html
@@ -7,7 +7,7 @@
 
     <a class="navbar-brand me-2" href="index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/articles/collaboration.html b/articles/collaboration.html
index c59ff6b1..96e235ea 100644
--- a/articles/collaboration.html
+++ b/articles/collaboration.html
@@ -20,7 +20,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -193,7 +193,7 @@ <h2 id="an-example">An example<a class="anchor" aria-label="anchor" href="#an-ex
 <p>Alice needs to run <code>orderly::orderly_init()</code> first:</p>
 <div class="sourceCode" id="cb3"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_init.html">orderly_init</a></span><span class="op">(</span><span class="st">"."</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/Rtmp0EydQ2/file1a81213bceca/alice'</span></span>
+<span><span class="co">## <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpXFj1Og/file1a6b795204b0/alice'</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Wrote '.gitignore'</span></span>
 <span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_list_src.html">orderly_list_src</a></span><span class="op">(</span><span class="op">)</span></span>
 <span><span class="co">## [1] "data"</span></span></code></pre></div>
@@ -206,11 +206,11 @@ <h2 id="an-example">An example<a class="anchor" aria-label="anchor" href="#an-ex
 <p>Once done, she can run any analysis:</p>
 <div class="sourceCode" id="cb5"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="va">id</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"data"</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240903-120329-47d2bd65` at 2024-09-03 12:03:29.286355</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240905-080221-6b00c525` at 2024-09-05 08:02:21.423466</span></span>
 <span><span class="co">## &gt; orderly2::orderly_artefact("data.rds", description = "Final data")</span></span>
 <span><span class="co">## &gt; saveRDS(mtcars, "data.rds")</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">data.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240903-120329-47d2bd65 at 2024-09-03 12:03:29.345561 (0.05920625 secs)</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240905-080221-6b00c525 at 2024-09-05 08:02:21.478472 (0.05500579 secs)</span></span></code></pre></div>
 <p>Perhaps it takes several goes for Alice to be happy with the
 analysis, but at some point she has something ready to share. She can
 then “push” the final packet up onto their server:</p>
@@ -220,7 +220,7 @@ <h2 id="an-example">An example<a class="anchor" aria-label="anchor" href="#an-ex
 initialised, and the server added:</p>
 <div class="sourceCode" id="cb7"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_init.html">orderly_init</a></span><span class="op">(</span><span class="st">"."</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/Rtmp0EydQ2/file1a81213bceca/bob'</span></span>
+<span><span class="co">## <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpXFj1Og/file1a6b795204b0/bob'</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Wrote '.gitignore'</span></span></code></pre></div>
 <div class="sourceCode" id="cb8"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_location_add.html">orderly_location_add</a></span><span class="op">(</span></span>
@@ -234,18 +234,18 @@ <h2 id="an-example">An example<a class="anchor" aria-label="anchor" href="#an-ex
 <span>  name <span class="op">=</span> <span class="st">"data"</span>,</span>
 <span>  options <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/list.html" class="external-link">list</a></span><span class="op">(</span>allow_remote <span class="op">=</span> <span class="cn">TRUE</span>, pull_metadata <span class="op">=</span> <span class="cn">TRUE</span><span class="op">)</span><span class="op">)</span></span>
 <span><span class="co">##                         id name parameters</span></span>
-<span><span class="co">## 1 20240903-120329-47d2bd65 data</span></span></code></pre></div>
+<span><span class="co">## 1 20240905-080221-6b00c525 data</span></span></code></pre></div>
 <p>Having seen there is a new “data” packet here, he can pull this down
 locally (TODO: mrc-4414 makes this nicer):</p>
 <div class="sourceCode" id="cb10"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_location_pull_packet.html">orderly_location_pull_packet</a></span><span class="op">(</span><span class="va">id</span><span class="op">)</span></span>
 <span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Looking for suitable files already on disk</span></span>
 <span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Need to fetch 2 files (1.3 kB) from 1 location</span></span>
-<span><span class="co">## ⠙ Fetching file 1/2 (95 B) from 'server' | ETA:  0s [3ms]</span></span>
-<span><span class="co">## <span style="color: #00BB00;">✔</span> Fetched 2 files (1.3 kB) from 'server' in 20ms.</span></span>
+<span><span class="co">## ⠙ Fetching file 1/2 (95 B) from 'server' | ETA:  0s [2ms]</span></span>
+<span><span class="co">## <span style="color: #00BB00;">✔</span> Fetched 2 files (1.3 kB) from 'server' in 19ms.</span></span>
 <span><span class="co">## </span></span></code></pre></div>
 <p>Now Bob is in a position to develop against the same packet that
-Alice ran (20240903-120329-47d2bd65)</p>
+Alice ran (20240905-080221-6b00c525)</p>
 </div>
 <div class="section level2">
 <h2 id="possible-working-patterns">Possible working patterns<a class="anchor" aria-label="anchor" href="#possible-working-patterns"></a>
@@ -402,7 +402,7 @@ <h2 id="sharing-packets-with-collaborators-using-a-shared-file-system">Sharing p
 <span>  use_file_store <span class="op">=</span> <span class="cn">TRUE</span>, </span>
 <span>  require_complete_tree <span class="op">=</span> <span class="cn">TRUE</span></span>
 <span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/Rtmp0EydQ2/file1a81213bceca/sharepoint'</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpXFj1Og/file1a6b795204b0/sharepoint'</span></span></code></pre></div>
 <p>Create an orderly store with a file store and a complete tree. See
 <code><a href="../reference/orderly_init.html">orderly2::orderly_init()</a></code> for more details.</p>
 </li>
diff --git a/articles/dependencies.html b/articles/dependencies.html
index 5ced92d1..1e71eb2f 100644
--- a/articles/dependencies.html
+++ b/articles/dependencies.html
@@ -20,7 +20,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -138,16 +138,16 @@ <h3 id="basic-use">Basic use<a class="anchor" aria-label="anchor" href="#basic-u
 <code><a href="../articles/introduction.html">vignette("introduction")</a></code>, to get us started).</p>
 <div class="sourceCode" id="cb3"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="va">id1</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"data"</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240903-120332-bc493756` at 2024-09-03 12:03:32.737999</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240905-080224-b9674449` at 2024-09-05 08:02:24.726524</span></span>
 <span><span class="co">## &gt; d &lt;- read.csv("data.csv")</span></span>
 <span><span class="co">## &gt; d$z &lt;- resid(lm(y ~ x, d))</span></span>
 <span><span class="co">## &gt; saveRDS(d, "data.rds")</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">data.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240903-120332-bc493756 at 2024-09-03 12:03:32.779139 (0.04114032 secs)</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240905-080224-b9674449 at 2024-09-05 08:02:24.761707 (0.03518271 secs)</span></span>
 <span><span class="va">id2</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"analysis"</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">analysis</span>' `20240903-120332-ce9ea708` at 2024-09-03 12:03:32.809472</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">analysis</span>' `20240905-080224-c961843d` at 2024-09-05 08:02:24.788789</span></span>
 <span><span class="co">## &gt; orderly2::orderly_dependency("data", "latest()", "data.rds")</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Depending on <span style="color: #0000BB;">data</span> @ `20240903-120332-bc493756` (via latest(name == "data"))</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Depending on <span style="color: #0000BB;">data</span> @ `20240905-080224-b9674449` (via latest(name == "data"))</span></span>
 <span><span class="co">## &gt; d &lt;- readRDS("data.rds")</span></span>
 <span><span class="co">## &gt; png("analysis.png")</span></span>
 <span><span class="co">## &gt; plot(y ~ x, d)</span></span>
@@ -155,14 +155,14 @@ <h3 id="basic-use">Basic use<a class="anchor" aria-label="anchor" href="#basic-u
 <span><span class="co">## agg_png </span></span>
 <span><span class="co">##       2</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">analysis.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240903-120332-ce9ea708 at 2024-09-03 12:03:32.932545 (0.1230724 secs)</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240905-080224-c961843d at 2024-09-05 08:02:24.896412 (0.1076229 secs)</span></span></code></pre></div>
 <p>When we look at the metadata for the packet created from the
 <code>analysis</code> report, we can see it has used
-<code>20240903-120332-bc493756</code> as its dependency:</p>
+<code>20240905-080224-b9674449</code> as its dependency:</p>
 <div class="sourceCode" id="cb4"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_metadata.html">orderly_metadata</a></span><span class="op">(</span><span class="va">id2</span><span class="op">)</span><span class="op">$</span><span class="va">depends</span></span>
 <span><span class="co">##                     packet                  query        files</span></span>
-<span><span class="co">## 1 20240903-120332-bc493756 latest(name == "data") data.rds....</span></span></code></pre></div>
+<span><span class="co">## 1 20240905-080224-b9674449 latest(name == "data") data.rds....</span></span></code></pre></div>
 <p>(indeed it had to, there is only one copy of the <code>data</code>
 packet to pick from).</p>
 </div>
@@ -187,35 +187,35 @@ <h3 id="filtering-candidates-by-parameters">Filtering candidates by parameters<a
 <p>We can run this for several values of <code>cyl</code>:</p>
 <div class="sourceCode" id="cb7"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"data"</span>, <span class="fu"><a href="https://rdrr.io/r/base/list.html" class="external-link">list</a></span><span class="op">(</span>cyl <span class="op">=</span> <span class="fl">4</span><span class="op">)</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240903-120333-4dff39a4` at 2024-09-03 12:03:33.307221</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240905-080225-40f9c854` at 2024-09-05 08:02:25.256076</span></span>
 <span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Parameters:</span></span>
 <span><span class="co">## • cyl: 4</span></span>
 <span><span class="co">## &gt; orderly2::orderly_parameters(cyl = NULL)</span></span>
 <span><span class="co">## &gt; d &lt;- mtcars[mtcars$cyl == cyl, ]</span></span>
 <span><span class="co">## &gt; saveRDS(d, "data.rds")</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">data.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240903-120333-4dff39a4 at 2024-09-03 12:03:33.339484 (0.03226328 secs)</span></span>
-<span><span class="co">## [1] "20240903-120333-4dff39a4"</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240905-080225-40f9c854 at 2024-09-05 08:02:25.286214 (0.03013873 secs)</span></span>
+<span><span class="co">## [1] "20240905-080225-40f9c854"</span></span>
 <span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"data"</span>, <span class="fu"><a href="https://rdrr.io/r/base/list.html" class="external-link">list</a></span><span class="op">(</span>cyl <span class="op">=</span> <span class="fl">6</span><span class="op">)</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240903-120333-5bc6889c` at 2024-09-03 12:03:33.360801</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240905-080225-4da56319` at 2024-09-05 08:02:25.30542</span></span>
 <span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Parameters:</span></span>
 <span><span class="co">## • cyl: 6</span></span>
 <span><span class="co">## &gt; orderly2::orderly_parameters(cyl = NULL)</span></span>
 <span><span class="co">## &gt; d &lt;- mtcars[mtcars$cyl == cyl, ]</span></span>
 <span><span class="co">## &gt; saveRDS(d, "data.rds")</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">data.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240903-120333-5bc6889c at 2024-09-03 12:03:33.389766 (0.02896452 secs)</span></span>
-<span><span class="co">## [1] "20240903-120333-5bc6889c"</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240905-080225-4da56319 at 2024-09-05 08:02:25.332333 (0.02691317 secs)</span></span>
+<span><span class="co">## [1] "20240905-080225-4da56319"</span></span>
 <span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"data"</span>, <span class="fu"><a href="https://rdrr.io/r/base/list.html" class="external-link">list</a></span><span class="op">(</span>cyl <span class="op">=</span> <span class="fl">8</span><span class="op">)</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240903-120333-693f179c` at 2024-09-03 12:03:33.413622</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240905-080225-59e680b0` at 2024-09-05 08:02:25.353485</span></span>
 <span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Parameters:</span></span>
 <span><span class="co">## • cyl: 8</span></span>
 <span><span class="co">## &gt; orderly2::orderly_parameters(cyl = NULL)</span></span>
 <span><span class="co">## &gt; d &lt;- mtcars[mtcars$cyl == cyl, ]</span></span>
 <span><span class="co">## &gt; saveRDS(d, "data.rds")</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">data.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240903-120333-693f179c at 2024-09-03 12:03:33.440027 (0.02640557 secs)</span></span>
-<span><span class="co">## [1] "20240903-120333-693f179c"</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240905-080225-59e680b0 at 2024-09-05 08:02:25.378441 (0.02495623 secs)</span></span>
+<span><span class="co">## [1] "20240905-080225-59e680b0"</span></span></code></pre></div>
 <p>Our follow-on analysis contains:</p>
 <div class="sourceCode" id="cb8"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_parameters.html">orderly_parameters</a></span><span class="op">(</span>cyl <span class="op">=</span> <span class="cn">NULL</span><span class="op">)</span></span>
@@ -233,7 +233,7 @@ <h3 id="filtering-candidates-by-parameters">Filtering candidates by parameters<a
 currently running report (<code>this:cyl</code>).</p>
 <div class="sourceCode" id="cb9"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"analysis"</span>, <span class="fu"><a href="https://rdrr.io/r/base/list.html" class="external-link">list</a></span><span class="op">(</span>cyl <span class="op">=</span> <span class="fl">4</span><span class="op">)</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">analysis</span>' `20240903-120333-97e3f27b` at 2024-09-03 12:03:33.596035</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">analysis</span>' `20240905-080225-8719a52e` at 2024-09-05 08:02:25.530097</span></span>
 <span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Parameters:</span></span>
 <span><span class="co">## • cyl: 4</span></span>
 <span><span class="co">## &gt; orderly2::orderly_parameters(cyl = NULL)</span></span>
@@ -241,7 +241,7 @@ <h3 id="filtering-candidates-by-parameters">Filtering candidates by parameters<a
 <span><span class="co">## +   "data",</span></span>
 <span><span class="co">## +   "latest(parameter:cyl == this:cyl)",</span></span>
 <span><span class="co">## +   "data.rds")</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Depending on <span style="color: #0000BB;">data</span> @ `20240903-120333-4dff39a4` (via latest(parameter:cyl == this:cyl &amp;&amp; name == "data"))</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Depending on <span style="color: #0000BB;">data</span> @ `20240905-080225-40f9c854` (via latest(parameter:cyl == this:cyl &amp;&amp; name == "data"))</span></span>
 <span><span class="co">## &gt; d &lt;- readRDS("data.rds")</span></span>
 <span><span class="co">## &gt; png("analysis.png")</span></span>
 <span><span class="co">## &gt; plot(mpg ~ disp, d)</span></span>
@@ -249,8 +249,8 @@ <h3 id="filtering-candidates-by-parameters">Filtering candidates by parameters<a
 <span><span class="co">## agg_png </span></span>
 <span><span class="co">##       2</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">analysis.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240903-120333-97e3f27b at 2024-09-03 12:03:33.649306 (0.05327106 secs)</span></span>
-<span><span class="co">## [1] "20240903-120333-97e3f27b"</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240905-080225-8719a52e at 2024-09-05 08:02:25.579071 (0.04897404 secs)</span></span>
+<span><span class="co">## [1] "20240905-080225-8719a52e"</span></span></code></pre></div>
 </div>
 <div class="section level3">
 <h3 id="interpreting-errors">Interpreting errors<a class="anchor" aria-label="anchor" href="#interpreting-errors"></a>
@@ -258,7 +258,7 @@ <h3 id="interpreting-errors">Interpreting errors<a class="anchor" aria-label="an
 <p>If your query fails to resolve a candidate it will error:</p>
 <div class="sourceCode" id="cb10"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"analysis"</span>, <span class="fu"><a href="https://rdrr.io/r/base/list.html" class="external-link">list</a></span><span class="op">(</span>cyl <span class="op">=</span> <span class="fl">9000</span><span class="op">)</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">analysis</span>' `20240903-120333-bb5814a0` at 2024-09-03 12:03:33.734306</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">analysis</span>' `20240905-080225-a81e462f` at 2024-09-05 08:02:25.658968</span></span>
 <span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Parameters:</span></span>
 <span><span class="co">## • cyl: 9000</span></span>
 <span><span class="co">## &gt; orderly2::orderly_parameters(cyl = NULL)</span></span>
@@ -267,7 +267,7 @@ <h3 id="interpreting-errors">Interpreting errors<a class="anchor" aria-label="an
 <span><span class="co">## +   "latest(parameter:cyl == this:cyl)",</span></span>
 <span><span class="co">## +   "data.rds")</span></span>
 <span><span class="co">## <span style="color: #BB0000;">✖</span> Error running <span style="color: #0000BB;">analysis.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240903-120333-bb5814a0 at 2024-09-03 12:03:33.798243 (0.06393719 secs)</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240905-080225-a81e462f at 2024-09-05 08:02:25.717692 (0.05872464 secs)</span></span>
 <span><span class="co">## <span style="color: #BBBB00; font-weight: bold;">Error</span><span style="font-weight: bold;"> in `orderly2::orderly_run()`:</span></span></span>
 <span><span class="co">## <span style="color: #BBBB00;">!</span> Failed to run report</span></span>
 <span><span class="co">## <span style="font-weight: bold;">Caused by error in `outpack_packet_use_dependency()`:</span></span></span>
@@ -324,8 +324,8 @@ <h3 id="interpreting-errors">Interpreting errors<a class="anchor" aria-label="an
 <span><span class="co">## [1] 3</span></span>
 <span><span class="co">## </span></span>
 <span><span class="co">## $found</span></span>
-<span><span class="co">## [1] "20240903-120333-4dff39a4" "20240903-120333-5bc6889c"</span></span>
-<span><span class="co">## [3] "20240903-120333-693f179c"</span></span></code></pre></div>
+<span><span class="co">## [1] "20240905-080225-40f9c854" "20240905-080225-4da56319"</span></span>
+<span><span class="co">## [3] "20240905-080225-59e680b0"</span></span></code></pre></div>
 <p>(this would have worked with
 <code>rlang::last_error()$explanation$parts$A</code> too).</p>
 <p>You can also use <code><a href="../reference/orderly_metadata_extract.html">orderly2::orderly_metadata_extract</a></code> to
@@ -335,9 +335,9 @@ <h3 id="interpreting-errors">Interpreting errors<a class="anchor" aria-label="an
 <span>  name <span class="op">=</span> <span class="st">"data"</span>,</span>
 <span>  extract <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html" class="external-link">c</a></span><span class="op">(</span>cyl <span class="op">=</span> <span class="st">"parameters.cyl is number"</span><span class="op">)</span><span class="op">)</span></span>
 <span><span class="co">##                         id cyl</span></span>
-<span><span class="co">## 1 20240903-120333-4dff39a4   4</span></span>
-<span><span class="co">## 2 20240903-120333-5bc6889c   6</span></span>
-<span><span class="co">## 3 20240903-120333-693f179c   8</span></span></code></pre></div>
+<span><span class="co">## 1 20240905-080225-40f9c854   4</span></span>
+<span><span class="co">## 2 20240905-080225-4da56319   6</span></span>
+<span><span class="co">## 3 20240905-080225-59e680b0   8</span></span></code></pre></div>
 </div>
 <div class="section level3">
 <h3 id="filtering-candidates-in-other-ways">Filtering candidates in other ways<a class="anchor" aria-label="anchor" href="#filtering-candidates-in-other-ways"></a>
@@ -351,7 +351,7 @@ <h3 id="filtering-candidates-in-other-ways">Filtering candidates in other ways<a
 <code>latest(parameter:cyl == environment:cyl)</code> to match against
 whatever value <code>cyl</code> took in the evaluating environment.</p>
 <p>Instead of a query, you can provide a single id (e.g,
-<code>20240903-120333-97e3f27b</code>), which would mean that even as
+<code>20240905-080225-8719a52e</code>), which would mean that even as
 new copies of the <code>data</code> packet are created, this dependency
 will always resolve to the same value.</p>
 <p>You can chain together logical operations with
diff --git a/articles/index.html b/articles/index.html
index aaa1b55f..58b0221c 100644
--- a/articles/index.html
+++ b/articles/index.html
@@ -7,7 +7,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/articles/introduction.html b/articles/introduction.html
index f6e235e3..15db74de 100644
--- a/articles/introduction.html
+++ b/articles/introduction.html
@@ -20,7 +20,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -107,7 +107,7 @@ <h2 id="creating-an-empty-orderly-repository">Creating an empty orderly reposito
 <div class="sourceCode" id="cb2"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="va">path</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/tempfile.html" class="external-link">tempfile</a></span><span class="op">(</span><span class="op">)</span> <span class="co"># we'll use a temporary directory here - see note below</span></span>
 <span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_init.html">orderly_init</a></span><span class="op">(</span><span class="va">path</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpkGakDB/file1b0bcebb068'</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpqAQMea/file1af65b7c91e8'</span></span></code></pre></div>
 <p>which creates a few files:</p>
 <pre><code><span><span class="co">## <span style="color: #0000BB; font-weight: bold;">.</span></span></span>
 <span><span class="co">## ├── <span style="color: #0000BB; font-weight: bold;">.outpack</span></span></span>
@@ -169,14 +169,14 @@ <h2 id="creating-your-first-orderly-report">Creating your first orderly report<a
 <code><a href="../reference/orderly_run.html">orderly2::orderly_run()</a></code>:</p>
 <div class="sourceCode" id="cb8"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="va">id</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"incoming_data"</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">incoming_data</span>' `20240903-120337-0e6713a8` at 2024-09-03 12:03:37.060674</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">incoming_data</span>' `20240905-080228-ca5e5af6` at 2024-09-05 08:02:28.794513</span></span>
 <span><span class="co">## &gt; d &lt;- read.csv("data.csv")</span></span>
 <span><span class="co">## &gt; d$z &lt;- resid(lm(y ~ x, d))</span></span>
 <span><span class="co">## &gt; saveRDS(d, "data.rds")</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">incoming_data.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240903-120337-0e6713a8 at 2024-09-03 12:03:37.125051 (0.06437659 secs)</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240905-080228-ca5e5af6 at 2024-09-05 08:02:28.855893 (0.06137991 secs)</span></span>
 <span><span class="va">id</span></span>
-<span><span class="co">## [1] "20240903-120337-0e6713a8"</span></span></code></pre></div>
+<span><span class="co">## [1] "20240905-080228-ca5e5af6"</span></span></code></pre></div>
 <p>The <code>id</code> that is created is a new identifier for the
 packet that will be both unique among all packets (within reason) and
 chronologically sortable. A packet that has an id that sorts after
@@ -185,7 +185,7 @@ <h2 id="creating-your-first-orderly-report">Creating your first orderly report<a
 <pre><code><span><span class="co">## <span style="color: #0000BB; font-weight: bold;">.</span></span></span>
 <span><span class="co">## ├── <span style="color: #0000BB; font-weight: bold;">archive</span></span></span>
 <span><span class="co">## │   └── <span style="color: #0000BB; font-weight: bold;">incoming_data</span></span></span>
-<span><span class="co">## │       └── <span style="color: #0000BB; font-weight: bold;">20240903-120337-0e6713a8</span></span></span>
+<span><span class="co">## │       └── <span style="color: #0000BB; font-weight: bold;">20240905-080228-ca5e5af6</span></span></span>
 <span><span class="co">## │           ├── data.csv</span></span>
 <span><span class="co">## │           ├── data.rds</span></span>
 <span><span class="co">## │           └── <span style="color: #00BB00;">incoming_data.R</span></span></span>
@@ -198,7 +198,7 @@ <h2 id="creating-your-first-orderly-report">Creating your first orderly report<a
 <span><span class="co">##         └── <span style="color: #00BB00;">incoming_data.R</span></span></span></code></pre>
 <p>A few things have changed here:</p>
 <ul>
-<li>we have a directory archive/incoming_data/20240903-120337-0e6713a8;
+<li>we have a directory archive/incoming_data/20240905-080228-ca5e5af6;
 this directory contains
 <ul>
 <li>the file that was created when we ran the report
@@ -270,10 +270,10 @@ <h2 id="depending-on-packets-from-another-report">Depending on packets from anot
 <code><a href="../reference/orderly_run.html">orderly2::orderly_run()</a></code>:</p>
 <div class="sourceCode" id="cb13"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="va">id</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"analysis"</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">analysis</span>' `20240903-120337-93402a04` at 2024-09-03 12:03:37.57766</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">analysis</span>' `20240905-080229-4bf6295e` at 2024-09-05 08:02:29.299062</span></span>
 <span><span class="co">## &gt; orderly2::orderly_dependency("incoming_data", "latest()",</span></span>
 <span><span class="co">## +                              c("incoming.rds" = "data.rds"))</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Depending on <span style="color: #0000BB;">incoming_data</span> @ `20240903-120337-0e6713a8` (via latest(name == "incoming_data"))</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Depending on <span style="color: #0000BB;">incoming_data</span> @ `20240905-080228-ca5e5af6` (via latest(name == "incoming_data"))</span></span>
 <span><span class="co">## &gt; d &lt;- readRDS("incoming.rds")</span></span>
 <span><span class="co">## &gt; png("analysis.png")</span></span>
 <span><span class="co">## &gt; plot(y ~ x, d)</span></span>
@@ -281,7 +281,7 @@ <h2 id="depending-on-packets-from-another-report">Depending on packets from anot
 <span><span class="co">## agg_png </span></span>
 <span><span class="co">##       2</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">analysis.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240903-120337-93402a04 at 2024-09-03 12:03:37.660659 (0.08299875 secs)</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240905-080229-4bf6295e at 2024-09-05 08:02:29.375767 (0.07670498 secs)</span></span></code></pre></div>
 <p>For more information on dependencies, see
 <code><a href="../articles/dependencies.html">vignette("dependencies")</a></code>.</p>
 </div>
@@ -355,7 +355,7 @@ <h2 id="available-in-report-orderly-commands">Available in-report orderly comman
 available at that point (see below).</p>
 <div class="sourceCode" id="cb15"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="va">id</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"incoming_data"</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">incoming_data</span>' `20240903-120337-deea8c28` at 2024-09-03 12:03:37.872477</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">incoming_data</span>' `20240905-080229-95242471` at 2024-09-05 08:02:29.584198</span></span>
 <span><span class="co">## &gt; orderly2::orderly_strict_mode()</span></span>
 <span><span class="co">## &gt; orderly2::orderly_resource("data.csv")</span></span>
 <span><span class="co">## &gt; orderly2::orderly_artefact("Processed data", "data.rds")</span></span>
@@ -372,7 +372,7 @@ <h2 id="available-in-report-orderly-commands">Available in-report orderly comman
 <span><span class="co">## future versions of orderly, we will change the order of the arguments to</span></span>
 <span><span class="co">## 'orderly_artefact()' so that 'files' comes first. If you name your calls to</span></span>
 <span><span class="co">## 'description' then you will be compatible when we make this change.</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240903-120337-deea8c28 at 2024-09-03 12:03:37.919764 (0.04728723 secs)</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240905-080229-95242471 at 2024-09-05 08:02:29.626847 (0.04264855 secs)</span></span></code></pre></div>
 </div>
 <div class="section level2">
 <h2 id="parameterised-reports">Parameterised reports<a class="anchor" aria-label="anchor" href="#parameterised-reports"></a>
@@ -414,7 +414,7 @@ <h2 id="parameterised-reports">Parameterised reports<a class="anchor" aria-label
 set.</p>
 <div class="sourceCode" id="cb19"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="va">id</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"random"</span>, <span class="fu"><a href="https://rdrr.io/r/base/list.html" class="external-link">list</a></span><span class="op">(</span>n_samples <span class="op">=</span> <span class="fl">15</span><span class="op">)</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">random</span>' `20240903-120338-2315eb1e` at 2024-09-03 12:03:38.139638</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">random</span>' `20240905-080229-d73f20b5` at 2024-09-05 08:02:29.843275</span></span>
 <span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Parameters:</span></span>
 <span><span class="co">## • n_samples: 15</span></span>
 <span><span class="co">## &gt; orderly2::orderly_parameters(n_samples = 10)</span></span>
@@ -422,7 +422,7 @@ <h2 id="parameterised-reports">Parameterised reports<a class="anchor" aria-label
 <span><span class="co">## &gt; d &lt;- data.frame(x = x, y = x + rnorm(n_samples))</span></span>
 <span><span class="co">## &gt; saveRDS(d, "data.rds")</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">random.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240903-120338-2315eb1e at 2024-09-03 12:03:38.167204 (0.02756643 secs)</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240905-080229-d73f20b5 at 2024-09-05 08:02:29.868695 (0.02541995 secs)</span></span></code></pre></div>
 <p>Our resulting file has 15 rows, as the parameter we passed in
 affected the report:</p>
 <div class="sourceCode" id="cb20"><pre class="downlit sourceCode r">
@@ -449,11 +449,11 @@ <h2 id="parameterised-reports">Parameterised reports<a class="anchor" aria-label
 example we can find the most recent version of a packet by running:</p>
 <div class="sourceCode" id="cb21"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_search.html">orderly_search</a></span><span class="op">(</span><span class="st">'latest(name == "random")'</span><span class="op">)</span></span>
-<span><span class="co">## [1] "20240903-120338-2315eb1e"</span></span></code></pre></div>
+<span><span class="co">## [1] "20240905-080229-d73f20b5"</span></span></code></pre></div>
 <p>But we can also pass in parameter queries here:</p>
 <div class="sourceCode" id="cb22"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_search.html">orderly_search</a></span><span class="op">(</span><span class="st">'latest(name == "random" &amp;&amp; parameter:n_samples &gt; 10)'</span><span class="op">)</span></span>
-<span><span class="co">## [1] "20240903-120338-2315eb1e"</span></span></code></pre></div>
+<span><span class="co">## [1] "20240905-080229-d73f20b5"</span></span></code></pre></div>
 <p>These can be used within <code><a href="../reference/orderly_dependency.html">orderly2::orderly_dependency()</a></code>
 (the <code>name == "random"</code> part is implied by the first
 <code>name</code> argument), for example</p>
@@ -485,21 +485,21 @@ <h2 id="shared-resources">Shared resources<a class="anchor" aria-label="anchor"
 <pre><code><span><span class="co">## <span style="color: #0000BB; font-weight: bold;">.</span></span></span>
 <span><span class="co">## ├── <span style="color: #0000BB; font-weight: bold;">archive</span></span></span>
 <span><span class="co">## │   ├── <span style="color: #0000BB; font-weight: bold;">analysis</span></span></span>
-<span><span class="co">## │   │   └── <span style="color: #0000BB; font-weight: bold;">20240903-120337-93402a04</span></span></span>
+<span><span class="co">## │   │   └── <span style="color: #0000BB; font-weight: bold;">20240905-080229-4bf6295e</span></span></span>
 <span><span class="co">## │   │       ├── <span style="color: #00BB00;">analysis.R</span></span></span>
 <span><span class="co">## │   │       ├── <span style="color: #BB00BB; font-weight: bold;">analysis.png</span></span></span>
 <span><span class="co">## │   │       └── incoming.rds</span></span>
 <span><span class="co">## │   ├── <span style="color: #0000BB; font-weight: bold;">incoming_data</span></span></span>
-<span><span class="co">## │   │   ├── <span style="color: #0000BB; font-weight: bold;">20240903-120337-0e6713a8</span></span></span>
+<span><span class="co">## │   │   ├── <span style="color: #0000BB; font-weight: bold;">20240905-080228-ca5e5af6</span></span></span>
 <span><span class="co">## │   │   │   ├── data.csv</span></span>
 <span><span class="co">## │   │   │   ├── data.rds</span></span>
 <span><span class="co">## │   │   │   └── <span style="color: #00BB00;">incoming_data.R</span></span></span>
-<span><span class="co">## │   │   └── <span style="color: #0000BB; font-weight: bold;">20240903-120337-deea8c28</span></span></span>
+<span><span class="co">## │   │   └── <span style="color: #0000BB; font-weight: bold;">20240905-080229-95242471</span></span></span>
 <span><span class="co">## │   │       ├── data.csv</span></span>
 <span><span class="co">## │   │       ├── data.rds</span></span>
 <span><span class="co">## │   │       └── <span style="color: #00BB00;">incoming_data.R</span></span></span>
 <span><span class="co">## │   └── <span style="color: #0000BB; font-weight: bold;">random</span></span></span>
-<span><span class="co">## │       └── <span style="color: #0000BB; font-weight: bold;">20240903-120338-2315eb1e</span></span></span>
+<span><span class="co">## │       └── <span style="color: #0000BB; font-weight: bold;">20240905-080229-d73f20b5</span></span></span>
 <span><span class="co">## │           ├── data.rds</span></span>
 <span><span class="co">## │           └── <span style="color: #00BB00;">random.R</span></span></span>
 <span><span class="co">## ├── <span style="color: #0000BB; font-weight: bold;">draft</span></span></span>
@@ -530,7 +530,7 @@ <h2 id="shared-resources">Shared resources<a class="anchor" aria-label="anchor"
 <p>We can run this:</p>
 <div class="sourceCode" id="cb27"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="va">id</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"use_shared"</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">use_shared</span>' `20240903-120338-a291ae09` at 2024-09-03 12:03:38.63762</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">use_shared</span>' `20240905-080230-53670258` at 2024-09-05 08:02:30.328211</span></span>
 <span><span class="co">## &gt; orderly2::orderly_shared_resource("data.csv")</span></span>
 <span><span class="co">## &gt; orderly2::orderly_artefact("analysis", "analysis.png")</span></span>
 <span><span class="co">## Warning: Please use a named argument for the description in 'orderly_artefact()'</span></span>
@@ -549,11 +549,11 @@ <h2 id="shared-resources">Shared resources<a class="anchor" aria-label="anchor"
 <span><span class="co">## future versions of orderly, we will change the order of the arguments to</span></span>
 <span><span class="co">## 'orderly_artefact()' so that 'files' comes first. If you name your calls to</span></span>
 <span><span class="co">## 'description' then you will be compatible when we make this change.</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240903-120338-a291ae09 at 2024-09-03 12:03:38.685079 (0.04745865 secs)</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240905-080230-53670258 at 2024-09-05 08:02:30.372285 (0.04407382 secs)</span></span></code></pre></div>
 <p>In the resulting archive, the file that was used from the shared
 directory is present:</p>
 <pre><code><span><span class="co">## <span style="color: #0000BB; font-weight: bold;">archive/use_shared</span></span></span>
-<span><span class="co">## └── <span style="color: #0000BB; font-weight: bold;">20240903-120338-a291ae09</span></span></span>
+<span><span class="co">## └── <span style="color: #0000BB; font-weight: bold;">20240905-080230-53670258</span></span></span>
 <span><span class="co">##     ├── <span style="color: #BB00BB; font-weight: bold;">analysis.png</span></span></span>
 <span><span class="co">##     ├── data.csv</span></span>
 <span><span class="co">##     └── <span style="color: #00BB00;">use_shared.R</span></span></span></code></pre>
@@ -647,49 +647,49 @@ <h2 id="deleting-things-from-the-archive">Deleting things from the archive<a cla
 <span>  name <span class="op">=</span> <span class="st">"incoming_data"</span>,</span>
 <span>  extract <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html" class="external-link">c</a></span><span class="op">(</span>time <span class="op">=</span> <span class="st">"time.start"</span><span class="op">)</span><span class="op">)</span></span>
 <span><span class="co">##                         id                time</span></span>
-<span><span class="co">## 1 20240903-120337-0e6713a8 2024-09-03 12:03:37</span></span>
-<span><span class="co">## 2 20240903-120337-deea8c28 2024-09-03 12:03:37</span></span></code></pre></div>
+<span><span class="co">## 1 20240905-080228-ca5e5af6 2024-09-05 08:02:28</span></span>
+<span><span class="co">## 2 20240905-080229-95242471 2024-09-05 08:02:29</span></span></code></pre></div>
 <p>When we run the <code>analysis</code> task, it will pull in the most
-recent version (<code>20240903-120337-deea8c28</code>). However, if you
+recent version (<code>20240905-080229-95242471</code>). However, if you
 had deleted this manually (e.g., to save space or accidentally) or
 corrupted it (e.g., by opening some output in Excel and letting it save
 changes) it will not be able to be included, and running
 <code>analysis</code> will fail:</p>
 <div class="sourceCode" id="cb34"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"analysis"</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">analysis</span>' `20240903-120339-46b913ff` at 2024-09-03 12:03:39.278753</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">analysis</span>' `20240905-080230-f22cf381` at 2024-09-05 08:02:30.948278</span></span>
 <span><span class="co">## &gt; orderly2::orderly_dependency("incoming_data", "latest()",</span></span>
 <span><span class="co">## +                              c("incoming.rds" = "data.rds"))</span></span>
 <span><span class="co">## <span style="color: #BB0000;">✖</span> Error running <span style="color: #0000BB;">analysis.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240903-120339-46b913ff at 2024-09-03 12:03:39.356496 (0.07774282 secs)</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240905-080230-f22cf381 at 2024-09-05 08:02:31.020346 (0.07206869 secs)</span></span>
 <span><span class="co">## <span style="color: #BBBB00; font-weight: bold;">Error</span><span style="font-weight: bold;"> in `orderly2::orderly_run()`:</span></span></span>
 <span><span class="co">## <span style="color: #BBBB00;">!</span> Failed to run report</span></span>
 <span><span class="co">## <span style="font-weight: bold;">Caused by error in `orderly_copy_files()`:</span></span></span>
-<span><span class="co">## <span style="color: #BBBB00;">!</span> Unable to copy files, due to deleted packet 20240903-120337-deea8c28</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Consider 'orderly2::orderly_validate_archive("20240903-120337-deea8c28",</span></span>
+<span><span class="co">## <span style="color: #BBBB00;">!</span> Unable to copy files, due to deleted packet 20240905-080229-95242471</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Consider 'orderly2::orderly_validate_archive("20240905-080229-95242471",</span></span>
 <span><span class="co">##   action = "orphan")' to remove this packet from consideration</span></span>
 <span><span class="co">## <span style="font-weight: bold;">Caused by error:</span></span></span>
 <span><span class="co">## <span style="color: #BBBB00;">!</span> File not found in archive</span></span>
 <span><span class="co">## <span style="color: #BB0000;">✖</span> data.rds</span></span></code></pre></div>
 <p>The error here tries to be fairly informative, telling us that we
 failed because when copying files from
-<code>20240903-120337-deea8c28</code> we found that the packet was
+<code>20240905-080229-95242471</code> we found that the packet was
 corrupt, because the file <code>data.rds</code> was not found in the
 archive. It also suggests a fix; we can tell <code>orderly2</code> that
-<code>20240903-120337-deea8c28</code> is “orphaned” and should not be
+<code>20240905-080229-95242471</code> is “orphaned” and should not be
 considered for inclusion when we look for dependencies.</p>
 <p>We can carry out the suggestion and just validate this packet by
 running</p>
 <div class="sourceCode" id="cb35"><pre class="downlit sourceCode r">
-<code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_validate_archive.html">orderly_validate_archive</a></span><span class="op">(</span><span class="st">"20240903-120337-deea8c28"</span>, action <span class="op">=</span> <span class="st">"orphan"</span><span class="op">)</span></span></code></pre></div>
+<code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_validate_archive.html">orderly_validate_archive</a></span><span class="op">(</span><span class="st">"20240905-080229-95242471"</span>, action <span class="op">=</span> <span class="st">"orphan"</span><span class="op">)</span></span></code></pre></div>
 <p>or we can validate <em>all</em> the packets we have:</p>
 <div class="sourceCode" id="cb36"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_validate_archive.html">orderly_validate_archive</a></span><span class="op">(</span>action <span class="op">=</span> <span class="st">"orphan"</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BB00;">✔</span> 20240903-120337-0e6713a8 (incoming_data) is valid</span></span>
-<span><span class="co">## <span style="color: #00BB00;">✔</span> 20240903-120337-93402a04 (analysis) is valid</span></span>
-<span><span class="co">## <span style="color: #BB0000;">✖</span> 20240903-120337-deea8c28 (incoming_data) is invalid due to its files</span></span>
-<span><span class="co">## <span style="color: #00BB00;">✔</span> 20240903-120338-2315eb1e (random) is valid</span></span>
-<span><span class="co">## <span style="color: #00BB00;">✔</span> 20240903-120338-a291ae09 (use_shared) is valid</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BB00;">✔</span> 20240905-080228-ca5e5af6 (incoming_data) is valid</span></span>
+<span><span class="co">## <span style="color: #00BB00;">✔</span> 20240905-080229-4bf6295e (analysis) is valid</span></span>
+<span><span class="co">## <span style="color: #BB0000;">✖</span> 20240905-080229-95242471 (incoming_data) is invalid due to its files</span></span>
+<span><span class="co">## <span style="color: #00BB00;">✔</span> 20240905-080229-d73f20b5 (random) is valid</span></span>
+<span><span class="co">## <span style="color: #00BB00;">✔</span> 20240905-080230-53670258 (use_shared) is valid</span></span></code></pre></div>
 <p>If we had the option <code>core.require_complete_tree</code> enabled,
 then this process would also look for any packets that used our
 now-deleted packet and orphan those too, as we no longer have a complete
@@ -776,16 +776,16 @@ <h2 id="interaction-with-the-outpack-store">Interaction with the outpack store<a
 <span><span class="co">## │   └── outpack.rds</span></span>
 <span><span class="co">## ├── <span style="color: #0000BB; font-weight: bold;">location</span></span></span>
 <span><span class="co">## │   ├── <span style="color: #0000BB; font-weight: bold;">local</span></span></span>
-<span><span class="co">## │   │   ├── 20240903-120337-0e6713a8</span></span>
-<span><span class="co">## │   │   ├── 20240903-120337-93402a04</span></span>
-<span><span class="co">## │   │   ├── 20240903-120338-2315eb1e</span></span>
-<span><span class="co">## │   │   └── 20240903-120338-a291ae09</span></span>
+<span><span class="co">## │   │   ├── 20240905-080228-ca5e5af6</span></span>
+<span><span class="co">## │   │   ├── 20240905-080229-4bf6295e</span></span>
+<span><span class="co">## │   │   ├── 20240905-080229-d73f20b5</span></span>
+<span><span class="co">## │   │   └── 20240905-080230-53670258</span></span>
 <span><span class="co">## │   └── <span style="color: #0000BB; font-weight: bold;">orphan</span></span></span>
 <span><span class="co">## └── <span style="color: #0000BB; font-weight: bold;">metadata</span></span></span>
-<span><span class="co">##     ├── 20240903-120337-0e6713a8</span></span>
-<span><span class="co">##     ├── 20240903-120337-93402a04</span></span>
-<span><span class="co">##     ├── 20240903-120338-2315eb1e</span></span>
-<span><span class="co">##     └── 20240903-120338-a291ae09</span></span></code></pre>
+<span><span class="co">##     ├── 20240905-080228-ca5e5af6</span></span>
+<span><span class="co">##     ├── 20240905-080229-4bf6295e</span></span>
+<span><span class="co">##     ├── 20240905-080229-d73f20b5</span></span>
+<span><span class="co">##     └── 20240905-080230-53670258</span></span></code></pre>
 <p>As can be perhaps inferred from the filenames, the files
 <code>.outpack/metadata/&lt;packet-id&gt;</code> are the metadata for
 each packet as it has been run. The files
@@ -825,16 +825,16 @@ <h2 id="interaction-with-the-outpack-store">Interaction with the outpack store<a
 <span><span class="co">## │   └── outpack.rds</span></span>
 <span><span class="co">## ├── <span style="color: #0000BB; font-weight: bold;">location</span></span></span>
 <span><span class="co">## │   ├── <span style="color: #0000BB; font-weight: bold;">local</span></span></span>
-<span><span class="co">## │   │   ├── 20240903-120337-0e6713a8</span></span>
-<span><span class="co">## │   │   ├── 20240903-120337-93402a04</span></span>
-<span><span class="co">## │   │   ├── 20240903-120338-2315eb1e</span></span>
-<span><span class="co">## │   │   └── 20240903-120338-a291ae09</span></span>
+<span><span class="co">## │   │   ├── 20240905-080228-ca5e5af6</span></span>
+<span><span class="co">## │   │   ├── 20240905-080229-4bf6295e</span></span>
+<span><span class="co">## │   │   ├── 20240905-080229-d73f20b5</span></span>
+<span><span class="co">## │   │   └── 20240905-080230-53670258</span></span>
 <span><span class="co">## │   └── <span style="color: #0000BB; font-weight: bold;">orphan</span></span></span>
 <span><span class="co">## └── <span style="color: #0000BB; font-weight: bold;">metadata</span></span></span>
-<span><span class="co">##     ├── 20240903-120337-0e6713a8</span></span>
-<span><span class="co">##     ├── 20240903-120337-93402a04</span></span>
-<span><span class="co">##     ├── 20240903-120338-2315eb1e</span></span>
-<span><span class="co">##     └── 20240903-120338-a291ae09</span></span></code></pre>
+<span><span class="co">##     ├── 20240905-080228-ca5e5af6</span></span>
+<span><span class="co">##     ├── 20240905-080229-4bf6295e</span></span>
+<span><span class="co">##     ├── 20240905-080229-d73f20b5</span></span>
+<span><span class="co">##     └── 20240905-080230-53670258</span></span></code></pre>
 <p>The files under <code>.outpack/files/</code> should never be modified
 or deleted. This approach to storage naturally deduplicates the file
 archive, so that a large file used in many places is only ever stored
diff --git a/articles/metadata.html b/articles/metadata.html
index 5ffc4d07..e046ff2b 100644
--- a/articles/metadata.html
+++ b/articles/metadata.html
@@ -20,7 +20,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/articles/migrating.html b/articles/migrating.html
index cce4b1ee..b54f7d3f 100644
--- a/articles/migrating.html
+++ b/articles/migrating.html
@@ -20,7 +20,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/articles/orderly2.html b/articles/orderly2.html
index 6934b9f2..f29b050a 100644
--- a/articles/orderly2.html
+++ b/articles/orderly2.html
@@ -20,7 +20,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/articles/plugins.html b/articles/plugins.html
index 14ce1e17..078a6a88 100644
--- a/articles/plugins.html
+++ b/articles/plugins.html
@@ -20,7 +20,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -137,7 +137,7 @@ <h3 id="an-example">An example<a class="anchor" aria-label="anchor" href="#an-ex
 <div class="sourceCode" id="cb2"><pre class="sourceCode yaml"><code class="sourceCode yaml"><span id="cb2-1"><a href="#cb2-1" tabindex="-1"></a><span class="fu">minimum_orderly_version</span><span class="kw">:</span><span class="at"> </span><span class="fl">1.99.0</span></span>
 <span id="cb2-2"><a href="#cb2-2" tabindex="-1"></a><span class="fu">plugins</span><span class="kw">:</span></span>
 <span id="cb2-3"><a href="#cb2-3" tabindex="-1"></a><span class="at">  </span><span class="fu">example.db</span><span class="kw">:</span></span>
-<span id="cb2-4"><a href="#cb2-4" tabindex="-1"></a><span class="at">    </span><span class="fu">path</span><span class="kw">:</span><span class="at"> /tmp/RtmpFxHGsW/file1bfd3c51fc2</span></span></code></pre></div>
+<span id="cb2-4"><a href="#cb2-4" tabindex="-1"></a><span class="at">    </span><span class="fu">path</span><span class="kw">:</span><span class="at"> /tmp/Rtmp01esxs/file1be86b7a0ff0</span></span></code></pre></div>
 <p>Our plugin is called <code>example.db</code> and is listed within the
 <code>plugins</code> section, along with its configuration; in this case
 indicating the path where the SQLite file can be loaded from.</p>
@@ -289,7 +289,7 @@ <h4 id="trying-it-out">Trying it out<a class="anchor" aria-label="anchor" href="
 <p>Now, we can run the report:</p>
 <div class="sourceCode" id="cb13"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"example"</span>, root <span class="op">=</span> <span class="va">path_root</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">example</span>' `20240903-120347-89034989` at 2024-09-03 12:03:47.539182</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">example</span>' `20240905-080238-d2ef47f8` at 2024-09-05 08:02:38.827684</span></span>
 <span><span class="co">## &gt; dat &lt;- example.db::query("SELECT * FROM mtcars WHERE cyl == 4")</span></span>
 <span><span class="co">## &gt; orderly2::orderly_artefact("Summary of data", "data.rds")</span></span>
 <span><span class="co">## Warning: Please use a named argument for the description in 'orderly_artefact()'</span></span>
@@ -303,8 +303,8 @@ <h4 id="trying-it-out">Trying it out<a class="anchor" aria-label="anchor" href="
 <span><span class="co">## future versions of orderly, we will change the order of the arguments to</span></span>
 <span><span class="co">## 'orderly_artefact()' so that 'files' comes first. If you name your calls to</span></span>
 <span><span class="co">## 'description' then you will be compatible when we make this change.</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240903-120347-89034989 at 2024-09-03 12:03:47.741805 (0.2026234 secs)</span></span>
-<span><span class="co">## [1] "20240903-120347-89034989"</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240905-080238-d2ef47f8 at 2024-09-05 08:02:39.001563 (0.1738789 secs)</span></span>
+<span><span class="co">## [1] "20240905-080238-d2ef47f8"</span></span></code></pre></div>
 </div>
 </div>
 <div class="section level2">
@@ -444,7 +444,7 @@ <h2 id="saving-metadata-about-what-the-plugin-did">Saving metadata about what th
 <pre><code><span><span class="co">## <span style="color: #0000BB; font-weight: bold;">.</span></span></span>
 <span><span class="co">## ├── <span style="color: #0000BB; font-weight: bold;">archive</span></span></span>
 <span><span class="co">## │   └── <span style="color: #0000BB; font-weight: bold;">example</span></span></span>
-<span><span class="co">## │       └── <span style="color: #0000BB; font-weight: bold;">20240903-120347-89034989</span></span></span>
+<span><span class="co">## │       └── <span style="color: #0000BB; font-weight: bold;">20240905-080238-d2ef47f8</span></span></span>
 <span><span class="co">## │           ├── data.rds</span></span>
 <span><span class="co">## │           └── <span style="color: #00BB00;">example.R</span></span></span>
 <span><span class="co">## ├── <span style="color: #0000BB; font-weight: bold;">draft</span></span></span>
@@ -496,7 +496,7 @@ <h2 id="saving-metadata-about-what-the-plugin-did">Saving metadata about what th
 several).</p>
 <div class="sourceCode" id="cb21"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="va">id</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"example"</span>, root <span class="op">=</span> <span class="va">path_root</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">example</span>' `20240903-120348-9f1ba3e6` at 2024-09-03 12:03:48.624246</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">example</span>' `20240905-080239-da473a20` at 2024-09-05 08:02:39.855157</span></span>
 <span><span class="co">## &gt; dat &lt;- example.db::query("SELECT * FROM mtcars WHERE cyl == 4")</span></span>
 <span><span class="co">## &gt; orderly2::orderly_artefact("Summary of data", "data.rds")</span></span>
 <span><span class="co">## Warning: Please use a named argument for the description in 'orderly_artefact()'</span></span>
@@ -510,7 +510,7 @@ <h2 id="saving-metadata-about-what-the-plugin-did">Saving metadata about what th
 <span><span class="co">## future versions of orderly, we will change the order of the arguments to</span></span>
 <span><span class="co">## 'orderly_artefact()' so that 'files' comes first. If you name your calls to</span></span>
 <span><span class="co">## 'description' then you will be compatible when we make this change.</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240903-120348-9f1ba3e6 at 2024-09-03 12:03:48.68091 (0.05666375 secs)</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240905-080239-da473a20 at 2024-09-05 08:02:39.90698 (0.0518229 secs)</span></span>
 <span><span class="va">meta</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_metadata.html">orderly_metadata</a></span><span class="op">(</span><span class="va">id</span>, root <span class="op">=</span> <span class="va">path_root</span><span class="op">)</span></span>
 <span><span class="va">meta</span><span class="op">$</span><span class="va">custom</span><span class="op">$</span><span class="va">example.db</span></span>
 <span><span class="co">##                                   sql rows         cols</span></span>
diff --git a/articles/query.html b/articles/query.html
index fff5833b..353445c6 100644
--- a/articles/query.html
+++ b/articles/query.html
@@ -20,7 +20,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/articles/troubleshooting.html b/articles/troubleshooting.html
index 61c2ba72..a81c6bb6 100644
--- a/articles/troubleshooting.html
+++ b/articles/troubleshooting.html
@@ -20,7 +20,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -116,21 +116,21 @@ <h2 id="outpack-files-accidentally-committed-to-git">Outpack files accidentally
 <span><span class="co">##   <span style="color: #0000BB; font-style: italic;">&lt;https://mrc-ide.github.io/orderly2/articles/troubleshooting.html&gt;</span></span></span>
 <span><span class="co">## <span style="color: #555555;">This warning is displayed once per session.</span></span></span></code></pre>
 <pre><code><span><span class="co">## <span style="color: #00BB00;">✔</span> Wrote '.gitignore'</span></span></code></pre>
-<pre><code><span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240903-120355-42792af8` at 2024-09-03 12:03:55.262153</span></span></code></pre>
+<pre><code><span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240905-080246-3b8cf756` at 2024-09-05 08:02:46.235029</span></span></code></pre>
 <pre><code><span><span class="co">## &gt; orderly2::orderly_artefact("data.rds", description = "Final data")</span></span>
 <span><span class="co">## &gt; saveRDS(mtcars, "data.rds")</span></span></code></pre>
 <pre><code><span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">data.R</span></span></span></code></pre>
-<pre><code><span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240903-120355-42792af8 at 2024-09-03 12:03:55.299319 (0.03716588 secs)</span></span></code></pre>
-<pre><code><span><span class="co">## [1] "20240903-120355-42792af8"</span></span></code></pre>
+<pre><code><span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240905-080246-3b8cf756 at 2024-09-05 08:02:46.270178 (0.03514838 secs)</span></span></code></pre>
+<pre><code><span><span class="co">## [1] "20240905-080246-3b8cf756"</span></span></code></pre>
 <p>subsequent calls will not display the warning:</p>
 <div class="sourceCode" id="cb12"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"data"</span><span class="op">)</span></span></code></pre></div>
-<pre><code><span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240903-120355-62d2689e` at 2024-09-03 12:03:55.388593</span></span></code></pre>
+<pre><code><span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240905-080246-5a93f4e7` at 2024-09-05 08:02:46.356176</span></span></code></pre>
 <pre><code><span><span class="co">## &gt; orderly2::orderly_artefact("data.rds", description = "Final data")</span></span>
 <span><span class="co">## &gt; saveRDS(mtcars, "data.rds")</span></span></code></pre>
 <pre><code><span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">data.R</span></span></span></code></pre>
-<pre><code><span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240903-120355-62d2689e at 2024-09-03 12:03:55.411622 (0.02302885 secs)</span></span></code></pre>
-<pre><code><span><span class="co">## [1] "20240903-120355-62d2689e"</span></span></code></pre>
+<pre><code><span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240905-080246-5a93f4e7 at 2024-09-05 08:02:46.376964 (0.02078772 secs)</span></span></code></pre>
+<pre><code><span><span class="co">## [1] "20240905-080246-5a93f4e7"</span></span></code></pre>
 <p>The rest of this section discusses how you might permanently fix the
 issue.</p>
 <div class="section level3">
diff --git a/authors.html b/authors.html
index a05bf245..6108720b 100644
--- a/authors.html
+++ b/authors.html
@@ -7,7 +7,7 @@
 
     <a class="navbar-brand me-2" href="index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -73,13 +73,13 @@ <h2 id="citation">Citation</h2>
 
       <p>FitzJohn R, Ashton R, Hill A (2024).
 <em>orderly2: Orderly Next Generation</em>.
-R package version 1.99.32, <a href="https://github.com/mrc-ide/orderly2" class="external-link">https://github.com/mrc-ide/orderly2</a>.
+R package version 1.99.33, <a href="https://github.com/mrc-ide/orderly2" class="external-link">https://github.com/mrc-ide/orderly2</a>.
 </p>
       <pre>@Manual{,
   title = {orderly2: Orderly Next Generation},
   author = {Rich FitzJohn and Robert Ashton and Alex Hill},
   year = {2024},
-  note = {R package version 1.99.32},
+  note = {R package version 1.99.33},
   url = {https://github.com/mrc-ide/orderly2},
 }</pre>
     </div>
diff --git a/index.html b/index.html
index 2091493d..4f31609d 100644
--- a/index.html
+++ b/index.html
@@ -22,7 +22,7 @@
 
     <a class="navbar-brand me-2" href="index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/pkgdown.yml b/pkgdown.yml
index 2fa3b31d..128513ef 100644
--- a/pkgdown.yml
+++ b/pkgdown.yml
@@ -11,7 +11,7 @@ articles:
   plugins: plugins.html
   query: query.html
   troubleshooting: troubleshooting.html
-last_built: 2024-09-03T12:03Z
+last_built: 2024-09-05T08:02Z
 urls:
   reference: https://mrc-ide.github.io/orderly2/reference
   article: https://mrc-ide.github.io/orderly2/articles
diff --git a/reference/index.html b/reference/index.html
index 85047d53..936033dd 100644
--- a/reference/index.html
+++ b/reference/index.html
@@ -7,7 +7,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_artefact.html b/reference/orderly_artefact.html
index 2b94b49c..8f803bed 100644
--- a/reference/orderly_artefact.html
+++ b/reference/orderly_artefact.html
@@ -11,7 +11,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_cleanup.html b/reference/orderly_cleanup.html
index 2ce84b4e..68a4232b 100644
--- a/reference/orderly_cleanup.html
+++ b/reference/orderly_cleanup.html
@@ -13,7 +13,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -135,24 +135,24 @@ <h2 id="notes-for-user-of-orderly-">Notes for user of orderly1<a class="anchor"
     <h2 id="ref-examples">Examples<a class="anchor" aria-label="anchor" href="#ref-examples"></a></h2>
     <div class="sourceCode"><pre class="sourceCode r"><code><span class="r-in"><span><span class="co"># Create a simple example:</span></span></span>
 <span class="r-in"><span><span class="va">path</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="orderly_example.html">orderly_example</a></span><span class="op">(</span><span class="st">"default"</span><span class="op">)</span></span></span>
-<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpeoP8u3/file1722797ca70d'</span>
+<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpTXa1jo/file170f43324a93'</span>
 <span class="r-in"><span></span></span>
 <span class="r-in"><span><span class="co"># We simulate running a packet interactively by using 'source';</span></span></span>
 <span class="r-in"><span><span class="co"># you might have run this line-by-line, or with the "Source"</span></span></span>
 <span class="r-in"><span><span class="co"># button in Rstudio.</span></span></span>
 <span class="r-in"><span><span class="kw"><a href="https://rdrr.io/r/base/source.html" class="external-link">source</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/r/base/file.path.html" class="external-link">file.path</a></span><span class="op">(</span><span class="va">path</span>, <span class="st">"src/data/data.R"</span><span class="op">)</span>, chdir <span class="op">=</span> <span class="cn">TRUE</span><span class="op">)</span></span></span>
-<span class="r-wrn co"><span class="r-pr">#&gt;</span> <span class="warning">Warning: </span>cannot open file '/tmp/RtmpeoP8u3/file1722797ca70d/src/data/data.R': No such file or directory</span>
+<span class="r-wrn co"><span class="r-pr">#&gt;</span> <span class="warning">Warning: </span>cannot open file '/tmp/RtmpTXa1jo/file170f43324a93/src/data/data.R': No such file or directory</span>
 <span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error in file(filename, "r", encoding = encoding):</span> cannot open the connection</span>
 <span class="r-in"><span></span></span>
 <span class="r-in"><span><span class="co"># Having run this, the output of the report is present in the</span></span></span>
 <span class="r-in"><span><span class="co"># source directory:</span></span></span>
 <span class="r-in"><span><span class="fu">fs</span><span class="fu">::</span><span class="fu"><a href="https://fs.r-lib.org/reference/dir_tree.html" class="external-link">dir_tree</a></span><span class="op">(</span><span class="va">path</span><span class="op">)</span></span></span>
-<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> [ENOENT] Failed to search directory '/tmp/RtmpeoP8u3/file1722797ca70d': no such file or directory</span>
+<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> [ENOENT] Failed to search directory '/tmp/RtmpTXa1jo/file170f43324a93': no such file or directory</span>
 <span class="r-in"><span></span></span>
 <span class="r-in"><span><span class="co"># We can detect what might want cleaning up by running</span></span></span>
 <span class="r-in"><span><span class="co"># "orderly_cleanup_status":</span></span></span>
 <span class="r-in"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu">orderly_cleanup_status</span><span class="op">(</span><span class="st">"data"</span>, root <span class="op">=</span> <span class="va">path</span><span class="op">)</span></span></span>
-<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpeoP8u3/file1722797ca70d'</span>
+<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpTXa1jo/file170f43324a93'</span>
 <span class="r-in"><span></span></span>
 <span class="r-in"><span><span class="co"># Soon this will print more nicely to the screen, but for now you</span></span></span>
 <span class="r-in"><span><span class="co"># can see that the status of "data.rds" is "derived", which means</span></span></span>
@@ -161,7 +161,7 @@ <h2 id="ref-examples">Examples<a class="anchor" aria-label="anchor" href="#ref-e
 <span class="r-in"><span></span></span>
 <span class="r-in"><span><span class="co"># Do the actual deletion:</span></span></span>
 <span class="r-in"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu">orderly_cleanup</span><span class="op">(</span><span class="st">"data"</span>, root <span class="op">=</span> <span class="va">path</span><span class="op">)</span></span></span>
-<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpeoP8u3/file1722797ca70d'</span>
+<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpTXa1jo/file170f43324a93'</span>
 </code></pre></div>
     </div>
   </main><aside class="col-md-3"><nav id="toc" aria-label="Table of contents"><h2>On this page</h2>
diff --git a/reference/orderly_config.html b/reference/orderly_config.html
index a35161e9..89974aa1 100644
--- a/reference/orderly_config.html
+++ b/reference/orderly_config.html
@@ -9,7 +9,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -110,7 +110,7 @@ <h2 id="ref-examples">Examples<a class="anchor" aria-label="anchor" href="#ref-e
 <span class="r-in"><span><span class="co"># A default configuration in a new temporary directory</span></span></span>
 <span class="r-in"><span><span class="va">path</span> <span class="op">&lt;-</span> <span class="fu">withr</span><span class="fu">::</span><span class="fu"><a href="https://withr.r-lib.org/reference/with_tempfile.html" class="external-link">local_tempdir</a></span><span class="op">(</span><span class="op">)</span></span></span>
 <span class="r-in"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="orderly_init.html">orderly_init</a></span><span class="op">(</span><span class="va">path</span><span class="op">)</span></span></span>
-<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpeoP8u3/file172241b2bd8d'</span>
+<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpTXa1jo/file170f7685df08'</span>
 <span class="r-in"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu">orderly_config</span><span class="op">(</span><span class="va">path</span><span class="op">)</span></span></span>
 <span class="r-out co"><span class="r-pr">#&gt;</span> $core</span>
 <span class="r-out co"><span class="r-pr">#&gt;</span> $core$path_archive</span>
diff --git a/reference/orderly_config_set.html b/reference/orderly_config_set.html
index f045467a..47363c13 100644
--- a/reference/orderly_config_set.html
+++ b/reference/orderly_config_set.html
@@ -9,7 +9,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -115,9 +115,9 @@ <h2 id="ref-examples">Examples<a class="anchor" aria-label="anchor" href="#ref-e
 <span class="r-in"><span><span class="co"># saves output within the "archive" directory:</span></span></span>
 <span class="r-in"><span><span class="va">path</span> <span class="op">&lt;-</span> <span class="fu">withr</span><span class="fu">::</span><span class="fu"><a href="https://withr.r-lib.org/reference/with_tempfile.html" class="external-link">local_tempdir</a></span><span class="op">(</span><span class="op">)</span></span></span>
 <span class="r-in"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="orderly_init.html">orderly_init</a></span><span class="op">(</span><span class="va">path</span><span class="op">)</span></span></span>
-<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpeoP8u3/file172222a3b2b0'</span>
+<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpTXa1jo/file170f658cd5a4'</span>
 <span class="r-in"><span><span class="fu">fs</span><span class="fu">::</span><span class="fu"><a href="https://fs.r-lib.org/reference/dir_tree.html" class="external-link">dir_tree</a></span><span class="op">(</span><span class="va">path</span>, all <span class="op">=</span> <span class="cn">TRUE</span><span class="op">)</span></span></span>
-<span class="r-out co"><span class="r-pr">#&gt;</span> <span style="color: #0000BB; font-weight: bold;">/tmp/RtmpeoP8u3/file172222a3b2b0</span></span>
+<span class="r-out co"><span class="r-pr">#&gt;</span> <span style="color: #0000BB; font-weight: bold;">/tmp/RtmpTXa1jo/file170f658cd5a4</span></span>
 <span class="r-out co"><span class="r-pr">#&gt;</span> ├── <span style="color: #0000BB; font-weight: bold;">.outpack</span></span>
 <span class="r-out co"><span class="r-pr">#&gt;</span> │   ├── config.json</span>
 <span class="r-out co"><span class="r-pr">#&gt;</span> │   ├── <span style="color: #0000BB; font-weight: bold;">location</span></span>
@@ -129,7 +129,7 @@ <h2 id="ref-examples">Examples<a class="anchor" aria-label="anchor" href="#ref-e
 <span class="r-in"><span>                             core.path_archive <span class="op">=</span> <span class="cn">NULL</span>,</span></span>
 <span class="r-in"><span>                             root <span class="op">=</span> <span class="va">path</span><span class="op">)</span></span></span>
 <span class="r-in"><span><span class="fu">fs</span><span class="fu">::</span><span class="fu"><a href="https://fs.r-lib.org/reference/dir_tree.html" class="external-link">dir_tree</a></span><span class="op">(</span><span class="va">path</span>, all <span class="op">=</span> <span class="cn">TRUE</span><span class="op">)</span></span></span>
-<span class="r-out co"><span class="r-pr">#&gt;</span> <span style="color: #0000BB; font-weight: bold;">/tmp/RtmpeoP8u3/file172222a3b2b0</span></span>
+<span class="r-out co"><span class="r-pr">#&gt;</span> <span style="color: #0000BB; font-weight: bold;">/tmp/RtmpTXa1jo/file170f658cd5a4</span></span>
 <span class="r-out co"><span class="r-pr">#&gt;</span> ├── <span style="color: #0000BB; font-weight: bold;">.outpack</span></span>
 <span class="r-out co"><span class="r-pr">#&gt;</span> │   ├── config.json</span>
 <span class="r-out co"><span class="r-pr">#&gt;</span> │   ├── <span style="color: #0000BB; font-weight: bold;">files</span></span>
diff --git a/reference/orderly_copy_files.html b/reference/orderly_copy_files.html
index ddbc077c..88cc0d86 100644
--- a/reference/orderly_copy_files.html
+++ b/reference/orderly_copy_files.html
@@ -19,7 +19,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_dependency.html b/reference/orderly_dependency.html
index be6b8f84..79dcdc93 100644
--- a/reference/orderly_dependency.html
+++ b/reference/orderly_dependency.html
@@ -7,7 +7,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_description.html b/reference/orderly_description.html
index d0e79a23..f72be4f6 100644
--- a/reference/orderly_description.html
+++ b/reference/orderly_description.html
@@ -7,7 +7,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_example.html b/reference/orderly_example.html
index fd01fbb4..90ef139a 100644
--- a/reference/orderly_example.html
+++ b/reference/orderly_example.html
@@ -9,7 +9,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -89,9 +89,9 @@ <h2 id="value">Value<a class="anchor" aria-label="anchor" href="#value"></a></h2
     <div class="section level2">
     <h2 id="ref-examples">Examples<a class="anchor" aria-label="anchor" href="#ref-examples"></a></h2>
     <div class="sourceCode"><pre class="sourceCode r"><code><span class="r-in"><span><span class="va">path</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu">orderly_example</span><span class="op">(</span><span class="st">"default"</span><span class="op">)</span></span></span>
-<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpeoP8u3/file1722606e9192'</span>
+<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpTXa1jo/file170f8fdfa80'</span>
 <span class="r-in"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="orderly_list_src.html">orderly_list_src</a></span><span class="op">(</span>root <span class="op">=</span> <span class="va">path</span><span class="op">)</span></span></span>
-<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpeoP8u3/file1722606e9192'</span>
+<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpTXa1jo/file170f8fdfa80'</span>
 </code></pre></div>
     </div>
   </main><aside class="col-md-3"><nav id="toc" aria-label="Table of contents"><h2>On this page</h2>
diff --git a/reference/orderly_gitignore_update.html b/reference/orderly_gitignore_update.html
index f7390af5..4d9c88fe 100644
--- a/reference/orderly_gitignore_update.html
+++ b/reference/orderly_gitignore_update.html
@@ -15,7 +15,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_hash.html b/reference/orderly_hash.html
index 1e73f59e..816d2c0f 100644
--- a/reference/orderly_hash.html
+++ b/reference/orderly_hash.html
@@ -19,7 +19,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_init.html b/reference/orderly_init.html
index 7770ce91..fffbbb0a 100644
--- a/reference/orderly_init.html
+++ b/reference/orderly_init.html
@@ -13,7 +13,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -147,7 +147,7 @@ <h2 id="ref-examples">Examples<a class="anchor" aria-label="anchor" href="#ref-e
 <span class="r-in"><span></span></span>
 <span class="r-in"><span><span class="co"># Initialise a new repository, setting an option:</span></span></span>
 <span class="r-in"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu">orderly_init</span><span class="op">(</span><span class="va">path</span>, use_file_store <span class="op">=</span> <span class="cn">TRUE</span><span class="op">)</span></span></span>
-<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpeoP8u3/file17222cb256f'</span>
+<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpTXa1jo/file170f7409549f'</span>
 <span class="r-in"><span></span></span>
 <span class="r-in"><span><span class="co"># fs::dir_tree(path, all = TRUE)</span></span></span>
 </code></pre></div>
diff --git a/reference/orderly_interactive_set_search_options.html b/reference/orderly_interactive_set_search_options.html
index 8e3c2405..7348c3e8 100644
--- a/reference/orderly_interactive_set_search_options.html
+++ b/reference/orderly_interactive_set_search_options.html
@@ -21,7 +21,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_list_src.html b/reference/orderly_list_src.html
index f927fa79..12e7f909 100644
--- a/reference/orderly_list_src.html
+++ b/reference/orderly_list_src.html
@@ -13,7 +13,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -99,9 +99,9 @@ <h2 id="see-also">See also<a class="anchor" aria-label="anchor" href="#see-also"
     <div class="section level2">
     <h2 id="ref-examples">Examples<a class="anchor" aria-label="anchor" href="#ref-examples"></a></h2>
     <div class="sourceCode"><pre class="sourceCode r"><code><span class="r-in"><span><span class="va">path</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="orderly_example.html">orderly_example</a></span><span class="op">(</span><span class="st">"default"</span><span class="op">)</span></span></span>
-<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpeoP8u3/file17221d5da8de'</span>
+<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpTXa1jo/file170f7c5e1acd'</span>
 <span class="r-in"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu">orderly_list_src</span><span class="op">(</span>root <span class="op">=</span> <span class="va">path</span><span class="op">)</span></span></span>
-<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpeoP8u3/file17221d5da8de'</span>
+<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpTXa1jo/file170f7c5e1acd'</span>
 </code></pre></div>
     </div>
   </main><aside class="col-md-3"><nav id="toc" aria-label="Table of contents"><h2>On this page</h2>
diff --git a/reference/orderly_location_add.html b/reference/orderly_location_add.html
index da2828f6..ed047ccb 100644
--- a/reference/orderly_location_add.html
+++ b/reference/orderly_location_add.html
@@ -17,7 +17,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_location_list.html b/reference/orderly_location_list.html
index 6e2daa96..d94ce89c 100644
--- a/reference/orderly_location_list.html
+++ b/reference/orderly_location_list.html
@@ -13,7 +13,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_location_pull_metadata.html b/reference/orderly_location_pull_metadata.html
index ba08b315..65d9e5c2 100644
--- a/reference/orderly_location_pull_metadata.html
+++ b/reference/orderly_location_pull_metadata.html
@@ -11,7 +11,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_location_pull_packet.html b/reference/orderly_location_pull_packet.html
index 3e00323a..a61fa092 100644
--- a/reference/orderly_location_pull_packet.html
+++ b/reference/orderly_location_pull_packet.html
@@ -13,7 +13,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_location_push.html b/reference/orderly_location_push.html
index 1bc63ca0..a6da2967 100644
--- a/reference/orderly_location_push.html
+++ b/reference/orderly_location_push.html
@@ -17,7 +17,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_location_remove.html b/reference/orderly_location_remove.html
index 74d2e5c2..8a701749 100644
--- a/reference/orderly_location_remove.html
+++ b/reference/orderly_location_remove.html
@@ -11,7 +11,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_location_rename.html b/reference/orderly_location_rename.html
index ea9e2812..35699685 100644
--- a/reference/orderly_location_rename.html
+++ b/reference/orderly_location_rename.html
@@ -7,7 +7,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_metadata.html b/reference/orderly_metadata.html
index e81b0372..510429fe 100644
--- a/reference/orderly_metadata.html
+++ b/reference/orderly_metadata.html
@@ -11,7 +11,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_metadata_extract.html b/reference/orderly_metadata_extract.html
index 0d568fe5..e914f74c 100644
--- a/reference/orderly_metadata_extract.html
+++ b/reference/orderly_metadata_extract.html
@@ -15,7 +15,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_metadata_read.html b/reference/orderly_metadata_read.html
index 16c20351..15b73e7c 100644
--- a/reference/orderly_metadata_read.html
+++ b/reference/orderly_metadata_read.html
@@ -15,7 +15,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_new.html b/reference/orderly_new.html
index 4e0e1cf1..81d6d437 100644
--- a/reference/orderly_new.html
+++ b/reference/orderly_new.html
@@ -7,7 +7,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_parameters.html b/reference/orderly_parameters.html
index 91ab25da..834729c2 100644
--- a/reference/orderly_parameters.html
+++ b/reference/orderly_parameters.html
@@ -19,7 +19,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_parse_file.html b/reference/orderly_parse_file.html
index d83f90ca..d616d7b7 100644
--- a/reference/orderly_parse_file.html
+++ b/reference/orderly_parse_file.html
@@ -7,7 +7,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_plugin_add_metadata.html b/reference/orderly_plugin_add_metadata.html
index 72ce08f8..47f26bdf 100644
--- a/reference/orderly_plugin_add_metadata.html
+++ b/reference/orderly_plugin_add_metadata.html
@@ -11,7 +11,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_plugin_context.html b/reference/orderly_plugin_context.html
index 7e5f0c71..990d086e 100644
--- a/reference/orderly_plugin_context.html
+++ b/reference/orderly_plugin_context.html
@@ -15,7 +15,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_plugin_register.html b/reference/orderly_plugin_register.html
index 722f745b..9c7dcca6 100644
--- a/reference/orderly_plugin_register.html
+++ b/reference/orderly_plugin_register.html
@@ -19,7 +19,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_prune_orphans.html b/reference/orderly_prune_orphans.html
index 81294a57..5be38c0e 100644
--- a/reference/orderly_prune_orphans.html
+++ b/reference/orderly_prune_orphans.html
@@ -17,7 +17,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_query.html b/reference/orderly_query.html
index afe644d0..f1817659 100644
--- a/reference/orderly_query.html
+++ b/reference/orderly_query.html
@@ -9,7 +9,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_query_explain.html b/reference/orderly_query_explain.html
index 5b664067..6f3409cf 100644
--- a/reference/orderly_query_explain.html
+++ b/reference/orderly_query_explain.html
@@ -13,7 +13,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_resource.html b/reference/orderly_resource.html
index 63b40cf0..1a972c0f 100644
--- a/reference/orderly_resource.html
+++ b/reference/orderly_resource.html
@@ -13,7 +13,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_run.html b/reference/orderly_run.html
index 6b401c1c..b0d94064 100644
--- a/reference/orderly_run.html
+++ b/reference/orderly_run.html
@@ -11,7 +11,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -236,19 +236,19 @@ <h2 id="manually-setting-report-source-directory">Manually setting report source
     <h2 id="ref-examples">Examples<a class="anchor" aria-label="anchor" href="#ref-examples"></a></h2>
     <div class="sourceCode"><pre class="sourceCode r"><code><span class="r-in"><span><span class="co"># Create a simple example:</span></span></span>
 <span class="r-in"><span><span class="va">path</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="orderly_example.html">orderly_example</a></span><span class="op">(</span><span class="st">"default"</span><span class="op">)</span></span></span>
-<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpeoP8u3/file1722ab3760d'</span>
+<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpTXa1jo/file170f1f0afcb1'</span>
 <span class="r-in"><span></span></span>
 <span class="r-in"><span><span class="co"># Run the 'data' task:</span></span></span>
 <span class="r-in"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu">orderly_run</span><span class="op">(</span><span class="st">"data"</span>, root <span class="op">=</span> <span class="va">path</span><span class="op">)</span></span></span>
-<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpeoP8u3/file1722ab3760d'</span>
+<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpTXa1jo/file170f1f0afcb1'</span>
 <span class="r-in"><span></span></span>
 <span class="r-in"><span><span class="co"># After running, a finished packet appears in the archive:</span></span></span>
 <span class="r-in"><span><span class="fu">fs</span><span class="fu">::</span><span class="fu"><a href="https://fs.r-lib.org/reference/dir_tree.html" class="external-link">dir_tree</a></span><span class="op">(</span><span class="va">path</span><span class="op">)</span></span></span>
-<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> [ENOENT] Failed to search directory '/tmp/RtmpeoP8u3/file1722ab3760d': no such file or directory</span>
+<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> [ENOENT] Failed to search directory '/tmp/RtmpTXa1jo/file170f1f0afcb1': no such file or directory</span>
 <span class="r-in"><span></span></span>
 <span class="r-in"><span><span class="co"># and we can query the metadata:</span></span></span>
 <span class="r-in"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="orderly_metadata_extract.html">orderly_metadata_extract</a></span><span class="op">(</span>name <span class="op">=</span> <span class="st">"data"</span>, root <span class="op">=</span> <span class="va">path</span><span class="op">)</span></span></span>
-<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpeoP8u3/file1722ab3760d'</span>
+<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpTXa1jo/file170f1f0afcb1'</span>
 </code></pre></div>
     </div>
   </main><aside class="col-md-3"><nav id="toc" aria-label="Table of contents"><h2>On this page</h2>
diff --git a/reference/orderly_run_info.html b/reference/orderly_run_info.html
index df60579c..a8234b90 100644
--- a/reference/orderly_run_info.html
+++ b/reference/orderly_run_info.html
@@ -21,7 +21,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_search.html b/reference/orderly_search.html
index 96b1933b..a8f0775d 100644
--- a/reference/orderly_search.html
+++ b/reference/orderly_search.html
@@ -17,7 +17,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_search_options.html b/reference/orderly_search_options.html
index 66291959..05f09aa9 100644
--- a/reference/orderly_search_options.html
+++ b/reference/orderly_search_options.html
@@ -15,7 +15,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_shared_resource.html b/reference/orderly_shared_resource.html
index 162d39fc..de86f3b1 100644
--- a/reference/orderly_shared_resource.html
+++ b/reference/orderly_shared_resource.html
@@ -17,7 +17,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_strict_mode.html b/reference/orderly_strict_mode.html
index 3acaae2b..e06c8f2e 100644
--- a/reference/orderly_strict_mode.html
+++ b/reference/orderly_strict_mode.html
@@ -23,7 +23,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_validate_archive.html b/reference/orderly_validate_archive.html
index a7d2beb6..8d29fadb 100644
--- a/reference/orderly_validate_archive.html
+++ b/reference/orderly_validate_archive.html
@@ -9,7 +9,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/query_index.html b/reference/query_index.html
index 4fd0e9b9..c2475dcc 100644
--- a/reference/query_index.html
+++ b/reference/query_index.html
@@ -7,7 +7,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.32</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.33</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/search.json b/search.json
index d345a87d..b4c111ff 100644
--- a/search.json
+++ b/search.json
@@ -1 +1 @@
-[{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"type-of-location","dir":"Articles","previous_headings":"","what":"Type of location","title":"Collaborative analysis","text":"orderly2 supports two types locations default: path: orderly2 working copy can find filesystem. Note copy people can see (example network share, cloud-synced file system Dropbox OneDrive) see Sharing packets collaborators using shared file system details http: require running HTTP API, either via outpack_server packit location system somewhat extensible, details subject change. documentation behave regardless location stored mechanism transport, can focus instead workflows.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"an-example","dir":"Articles","previous_headings":"","what":"An example","title":"Collaborative analysis","text":"Suppose two researchers, Alice Bob, collaborating piece analysis. purposes example (find common situation practice) share source tree analysis git. Alice Bob clone repo onto machines, using favourite git client (terminal git, via GitHub desktop, gert) moment, including hidden files (except .git), Alice sees: point, cloning Alice .outpack directory (see ) Alice needs run orderly::orderly_init() first: plan work Bob, sharing results shared server, Alice adds orderly2 configuration orderly2::orderly_location_add(): done, can run analysis: Perhaps takes several goes Alice happy analysis, point something ready share. can “push” final packet onto server: Now, consider Bob. also needs source code cloned, orderly initialised, server added: (, Bob also used name server refer shared server, used anything; similar git’s use origin.) Bob can now query packets available server: seen new “data” packet , can pull locally (TODO: mrc-4414 makes nicer): Now Bob position develop packet Alice ran (20240903-120329-47d2bd65)","code":"fs::dir_tree(\".\", all = TRUE, glob = \".git\", invert = TRUE) ## . ## ├── orderly_config.yml ## └── src ##     └── data ##         └── data.R orderly2::orderly_list_src() ## [1] \"data\" orderly2::orderly_init(\".\") ## ✔ Created orderly root at '/tmp/Rtmp0EydQ2/file1a81213bceca/alice' ## ✔ Wrote '.gitignore' orderly2::orderly_list_src() ## [1] \"data\" orderly2::orderly_location_add(   \"server\", \"http\", list(url = \"http://orderly.example.com\")) id <- orderly2::orderly_run(\"data\") ## ℹ Starting packet 'data' `20240903-120329-47d2bd65` at 2024-09-03 12:03:29.286355 ## > orderly2::orderly_artefact(\"data.rds\", description = \"Final data\") ## > saveRDS(mtcars, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240903-120329-47d2bd65 at 2024-09-03 12:03:29.345561 (0.05920625 secs) orderly2::orderly_location_push(id, \"server\") orderly2::orderly_init(\".\") ## ✔ Created orderly root at '/tmp/Rtmp0EydQ2/file1a81213bceca/bob' ## ✔ Wrote '.gitignore' orderly2::orderly_location_add(   \"server\", \"http\", list(url = \"http://orderly.example.com\")) orderly2::orderly_metadata_extract(   name = \"data\",   options = list(allow_remote = TRUE, pull_metadata = TRUE)) ##                         id name parameters ## 1 20240903-120329-47d2bd65 data orderly2::orderly_location_pull_packet(id) ## ℹ Looking for suitable files already on disk ## ℹ Need to fetch 2 files (1.3 kB) from 1 location ## ⠙ Fetching file 1/2 (95 B) from 'server' | ETA:  0s [3ms] ## ✔ Fetched 2 files (1.3 kB) from 'server' in 20ms. ##"},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"possible-working-patterns","dir":"Articles","previous_headings":"","what":"Possible working patterns","title":"Collaborative analysis","text":"seen several broad patterns distributing packets. Central server, push allowed: case ; flexible requires everyone relaxed packets created ultimately trusting colleagues exercising good environment hygiene. Just rules enforced computer though, doesn’t mean might ideas within group can push. may useful push HPC system running computationally demanding tasks . Central server, final copy run : case, Alice Bob can run things server, “canonical” copies, nothing pushed server. Reports can run server ssh-ing onto server running packets manually (TODO: mrc-4412) using web interface (possible OrderlyWeb possible packit). approach works well primary goal confidence packets everyone works dependencies created “clean” environment unexpected global dependencies. web version, can also enforce things like running default branch. Alice Bob end collection packets local archives mix canonical ones (server) locally-created ones, local ones always dead end never get shared anyone. result, Alice Bob may delete archive directories without great concern. running packets, make sure packet run similar way packet run server, Bob may want ensure dependencies found server considered. , can create search options pass orderly2::orderly_run running report. developing, Bob may find passing orderly2::orderly_interactive_set_search_options() helpful mean calls orderly2::orderly_description() use packets server. Staging production servers: might set multiple servers, one staging, one production. staging servers never source truth might (might ) allow people push , less strict rules gets run (perhaps allowing running packets feature branch). setting, users end packets created locally, staging machines production server local archive. staging server probably end mix packets packets production. approach specifying search options useful choosing included final packet.","code":"search_options <- orderly2::orderly_search_options(location = \"server\")"},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"pulling-complete-trees-or-not","dir":"Articles","previous_headings":"","what":"Pulling complete trees, or not","title":"Collaborative analysis","text":"pull packets server (orderly2::orderly_location_pull_packet()), depending settings either end just packet request, packet plus recursive dependencies. controlled configuration option core.require_complete_tree, FALSE default; suitable users, servers set core.require_complete_tree = TRUE. situations , users never holding canonical source packets - matter don’t dependencies. hold metadata packets (, recursively) download dependencies later wanted inspect . Typically though, just extra space used disk. push packet server, always pushes complete tree.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"moving-as-little-data-as-possible","dir":"Articles","previous_headings":"","what":"Moving as little data as possible","title":"Collaborative analysis","text":"pulling packets, first user pulls metadata server. orderly2::orderly_location_metadata_pull run, metadata server updated client (mrc-4444 make lazier). now know everything server holds, hashes files contained packet. likely already know files within packet requested, client first looks see exist within archive, verifying changed. requests server files resolve locally. recursive pull (core.require_complete_tree = TRUE) look union files set packets currently missing archive. frequently much less full set files within packets. algorithm similar push: look full dependency chain target packet find file ids ask server yet know unknown subset get union file hashes ask server subset files yet know push files - done one time present done simultaneously async requests future files sever upload missing metadata, chronological order server verifies packet metadata dependencies referenced files multi-step process means avoid copying data already known , well avoiding moving data multiple times. also means process pushing (pulling) interrupted can safely resumed state always consistent.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"interaction-between-source-tree-and--outpack","dir":"Articles","previous_headings":"","what":"Interaction between source tree and .outpack","title":"Collaborative analysis","text":"important .outpack directory shared via git; warn now, can use orderly2::ordery_gitignore_update() automatically create suitable .gitignore file prevent accidentally committed. However, Alice Bob starting new machine : clone source repository run orderly2::orderly_init() within directory run orderly2 commands normal Without running orderly2::orderly_init(), get error prompting initialise repo. requirement different repositories share configuration options passed orderly2::orderly_init(); can enable disable file store different sets locations enabled, depending workflows. means , practice, association convention set related orderly2 locations source tree can orderly2 repository points locations refer different source trees!","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"sharing-packets-with-collaborators-using-a-shared-file-system","dir":"Articles","previous_headings":"","what":"Sharing packets with collaborators using a shared file system","title":"Collaborative analysis","text":"One simplest ways share packets collaborator shared file system, example network share, cloud-synced file system Dropbox OneDrive. can follows. Create new folder orderly remote shared file system. Alice synced path_sharepoint_alice. Initialise orderly location shared file system Create orderly store file store complete tree. See orderly2::orderly_init() details. Add location Push packets want share available collaborator pull. Note data pushed file store .outpack directory. .dot files considered hidden operating system, “show hidden files” file browser see pushed packet. collaborator can now pull shared file. , : Sync drive location machine. Bob synced path_sharepoint_bob Add location Pull metadata use packets desired","code":"orderly2::orderly_init(   root = path_sharepoint_alice,    path_archive = NULL,    use_file_store = TRUE,    require_complete_tree = TRUE ) ## ✔ Created orderly root at '/tmp/Rtmp0EydQ2/file1a81213bceca/sharepoint' orderly2::orderly_location_add(   \"sharepoint\", \"path\", list(path = path_sharepoint_alice)) orderly2::orderly_location_push(id, \"sharepoint\") orderly2::orderly_location_add(   \"alices_orderly\", \"path\", list(path = path_sharepoint_bob)) orderly2::orderly_location_pull_metadata(   \"alices_orderly\" )"},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"using-dependencies","dir":"Articles","previous_headings":"","what":"Using dependencies","title":"Dependencies between packets","text":", show practically use dependencies common scenarios increasing complexity. code examples purposefully -simple order keep presentation straightforward, see end document discussion complex pieces code might “optimally” .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"basic-use","dir":"Articles","previous_headings":"Using dependencies","what":"Basic use","title":"Dependencies between packets","text":"primary mechanism using dependencies call orderly2::orderly_dependency() within orderly file; finds suitable completed packet copies files found within packet current report. src/analysis/analysis.R contains: , ’ve used orderly2::orderly_dependency() pull file data.rds recent version (latest()) data packet, ’ve used file normal make plot, ’ve saved analysis.png (similar example vignette(\"introduction\"), get us started). look metadata packet created analysis report, can see used 20240903-120332-bc493756 dependency: (indeed , one copy data packet pick ).","code":"## src ## ├── analysis ## │   └── analysis.R ## └── data ##     ├── data.R ##     └── data.csv orderly2::orderly_dependency(\"data\", \"latest()\", \"data.rds\") d <- readRDS(\"data.rds\") png(\"analysis.png\") plot(y ~ x, d) dev.off() id1 <- orderly2::orderly_run(\"data\") ## ℹ Starting packet 'data' `20240903-120332-bc493756` at 2024-09-03 12:03:32.737999 ## > d <- read.csv(\"data.csv\") ## > d$z <- resid(lm(y ~ x, d)) ## > saveRDS(d, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240903-120332-bc493756 at 2024-09-03 12:03:32.779139 (0.04114032 secs) id2 <- orderly2::orderly_run(\"analysis\") ## ℹ Starting packet 'analysis' `20240903-120332-ce9ea708` at 2024-09-03 12:03:32.809472 ## > orderly2::orderly_dependency(\"data\", \"latest()\", \"data.rds\") ## ℹ Depending on data @ `20240903-120332-bc493756` (via latest(name == \"data\")) ## > d <- readRDS(\"data.rds\") ## > png(\"analysis.png\") ## > plot(y ~ x, d) ## > dev.off() ## agg_png  ##       2 ## ✔ Finished running analysis.R ## ℹ Finished 20240903-120332-ce9ea708 at 2024-09-03 12:03:32.932545 (0.1230724 secs) orderly2::orderly_metadata(id2)$depends ##                     packet                  query        files ## 1 20240903-120332-bc493756 latest(name == \"data\") data.rds...."},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"filtering-candidates-by-parameters","dir":"Articles","previous_headings":"Using dependencies","what":"Filtering candidates by parameters","title":"Dependencies between packets","text":"example, query simple — recently created packet name data. One common pattern see analysis might parameter (example country name) downstream analysis might share parameter want pull data country. src/data/data.R containing: can run several values cyl: follow-analysis contains: query latest(parameter:cyl == :cyl) says “find recent packet ’s parameter”cyl” (parameter:cyl) parameter currently running report (:cyl).","code":"## src ## ├── analysis ## │   └── analysis.R ## └── data ##     └── data.R orderly2::orderly_parameters(cyl = NULL) d <- mtcars[mtcars$cyl == cyl, ] saveRDS(d, \"data.rds\") orderly2::orderly_run(\"data\", list(cyl = 4)) ## ℹ Starting packet 'data' `20240903-120333-4dff39a4` at 2024-09-03 12:03:33.307221 ## ℹ Parameters: ## • cyl: 4 ## > orderly2::orderly_parameters(cyl = NULL) ## > d <- mtcars[mtcars$cyl == cyl, ] ## > saveRDS(d, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240903-120333-4dff39a4 at 2024-09-03 12:03:33.339484 (0.03226328 secs) ## [1] \"20240903-120333-4dff39a4\" orderly2::orderly_run(\"data\", list(cyl = 6)) ## ℹ Starting packet 'data' `20240903-120333-5bc6889c` at 2024-09-03 12:03:33.360801 ## ℹ Parameters: ## • cyl: 6 ## > orderly2::orderly_parameters(cyl = NULL) ## > d <- mtcars[mtcars$cyl == cyl, ] ## > saveRDS(d, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240903-120333-5bc6889c at 2024-09-03 12:03:33.389766 (0.02896452 secs) ## [1] \"20240903-120333-5bc6889c\" orderly2::orderly_run(\"data\", list(cyl = 8)) ## ℹ Starting packet 'data' `20240903-120333-693f179c` at 2024-09-03 12:03:33.413622 ## ℹ Parameters: ## • cyl: 8 ## > orderly2::orderly_parameters(cyl = NULL) ## > d <- mtcars[mtcars$cyl == cyl, ] ## > saveRDS(d, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240903-120333-693f179c at 2024-09-03 12:03:33.440027 (0.02640557 secs) ## [1] \"20240903-120333-693f179c\" orderly2::orderly_parameters(cyl = NULL) orderly2::orderly_dependency(   \"data\",   \"latest(parameter:cyl == this:cyl)\",   \"data.rds\") d <- readRDS(\"data.rds\") png(\"analysis.png\") plot(mpg ~ disp, d) dev.off() orderly2::orderly_run(\"analysis\", list(cyl = 4)) ## ℹ Starting packet 'analysis' `20240903-120333-97e3f27b` at 2024-09-03 12:03:33.596035 ## ℹ Parameters: ## • cyl: 4 ## > orderly2::orderly_parameters(cyl = NULL) ## > orderly2::orderly_dependency( ## +   \"data\", ## +   \"latest(parameter:cyl == this:cyl)\", ## +   \"data.rds\") ## ℹ Depending on data @ `20240903-120333-4dff39a4` (via latest(parameter:cyl == this:cyl && name == \"data\")) ## > d <- readRDS(\"data.rds\") ## > png(\"analysis.png\") ## > plot(mpg ~ disp, d) ## > dev.off() ## agg_png  ##       2 ## ✔ Finished running analysis.R ## ℹ Finished 20240903-120333-97e3f27b at 2024-09-03 12:03:33.649306 (0.05327106 secs) ## [1] \"20240903-120333-97e3f27b\""},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"interpreting-errors","dir":"Articles","previous_headings":"Using dependencies","what":"Interpreting errors","title":"Dependencies between packets","text":"query fails resolve candidate error: error message tries fairly self explanatory; failed find packet satisfies querylatest(parameter:cyl == :cyl && name == \"data\"); note report name data become part query, two conditions matched . error suggests running rlang::last_error()$explanation information, can : orderly_query_explain object, tries come reasons query might matched; ’ll expand future let us know might like see. tells query can decomposed two subqueries (match parameter cyl 9000), matched packets B (match packet name data), matched 3 packets. subquery matched packets pairs don’t try guide towards problematic pairs. can also ask orderly2 explain query : save object can explore detail: (worked rlang::last_error()$explanation$parts$). can also use orderly2::orderly_metadata_extract work values might looked :","code":"orderly2::orderly_run(\"analysis\", list(cyl = 9000)) ## ℹ Starting packet 'analysis' `20240903-120333-bb5814a0` at 2024-09-03 12:03:33.734306 ## ℹ Parameters: ## • cyl: 9000 ## > orderly2::orderly_parameters(cyl = NULL) ## > orderly2::orderly_dependency( ## +   \"data\", ## +   \"latest(parameter:cyl == this:cyl)\", ## +   \"data.rds\") ## ✖ Error running analysis.R ## ℹ Finished 20240903-120333-bb5814a0 at 2024-09-03 12:03:33.798243 (0.06393719 secs) ## Error in `orderly2::orderly_run()`: ## ! Failed to run report ## Caused by error in `outpack_packet_use_dependency()`: ## ! Failed to find packet for query 'latest(parameter:cyl == this:cyl && ##   name == \"data\")' ## ℹ See 'rlang::last_error()$explanation' for details rlang::last_error()$explanation ## Evaluated query: 'latest(A && B)' and found 0 packets ## • A (parameter:cyl == this:cyl): 0 packets ##  ## • B (name == \"data\"): 3 packets orderly2::orderly_query_explain(   quote(latest(parameter:cyl == 9000)), name = \"data\") ## Evaluated query: 'latest(A && B)' and found 0 packets ## • A (parameter:cyl == 9000): 0 packets ##  ## • B (name == \"data\"): 3 packets explanation <- orderly2::orderly_query_explain(   quote(latest(parameter:cyl == 9000)), name = \"data\") explanation$parts$B ## $name ## [1] \"B\" ##  ## $str ## [1] \"name == \\\"data\\\"\" ##  ## $expr ## name == \"data\" ##  ## $n ## [1] 3 ##  ## $found ## [1] \"20240903-120333-4dff39a4\" \"20240903-120333-5bc6889c\" ## [3] \"20240903-120333-693f179c\" orderly2::orderly_metadata_extract(   name = \"data\",   extract = c(cyl = \"parameters.cyl is number\")) ##                         id cyl ## 1 20240903-120333-4dff39a4   4 ## 2 20240903-120333-5bc6889c   6 ## 3 20240903-120333-693f179c   8"},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"filtering-candidates-in-other-ways","dir":"Articles","previous_headings":"Using dependencies","what":"Filtering candidates in other ways","title":"Dependencies between packets","text":"saw two types filtering candidates: latest() selected recent packet latest(parameter:cyl == :cyl) found packet whose parameter matched one parameters. used latest(parameter:cyl == 4) hard code specific parameter value, used latest(parameter:cyl == environment:cyl) match whatever value cyl took evaluating environment. Instead query, can provide single id (e.g, 20240903-120333-97e3f27b), mean even new copies data packet created, dependency always resolve value. can chain together logical operations && (sides must true) || (either side must true), group conditions parentheses. addition ==, usual complement comparison operators work. might complex queries like practice people queries series restrictions &&.","code":"latest((parameter:x == 1 || parameter:x == 2) && parameter:y > 10)"},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"computing-dependencies-and-using-many-dependencies-at-once","dir":"Articles","previous_headings":"Using dependencies","what":"Computing dependencies and using many dependencies at once","title":"Dependencies between packets","text":"One common pattern map-reduce pattern set orderly reports. , set packets created vector parameters, perhaps chain different reports parameter, combined together. parameter p takes values “x”, “y” “z”, might look like: , D want combine three copies C packet, one p “x”, “y” “z”. Especially three values hard coded, might just write Note call vary second argument select different parameter value, third argument naming destination file different name (end three files data/). can write loop: , second argument use environment:p fetch value p calling environment - looping value take three values. name third argument, use special interpolation format ${p} substitute value p build filename.","code":"B(p = \"x\") -- C(p = \"x\")   /                          \\ A - B(p = \"y\") -- C(p = \"y\") - D   \\                          /     B(p = \"z\") -- C(p = \"z\") orderly2::orderly_dependency(\"C\", quote(latest(parameter:p == \"x\")),                              c(\"data/x.rds\" = \"result.rds\")) orderly2::orderly_dependency(\"C\", quote(latest(parameter:p == \"y\")),                              c(\"data/y.rds\" = \"result.rds\")) orderly2::orderly_dependency(\"C\", quote(latest(parameter:p == \"z\")),                              c(\"data/z.rds\" = \"result.rds\")) for (p in c(\"x\", \"y\", \"z\")) {   orderly2::orderly_dependency(\"C\", quote(latest(parameter:p == environment:p)),                                c(\"data/${p}.rds\" = \"result.rds\")) }"},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"how-dependencies-interact-with-locations","dir":"Articles","previous_headings":"","what":"How dependencies interact with locations","title":"Dependencies between packets","text":"default, packet unpacked local archive considered candidate inclusion orderly_dependency(). always want. locations selected, packets within considered candidates can controlled search_options argument orderly2::orderly_run (note argument orderly_run(), orderly_dependency() effect controlled runner report, writer report). three components affect packets selected location: character vector locations, matching location names. packets can found locations considered. mix locally created packets well ones people can see, specifying location = \"server\" limit packets available server, means end dependencies colleagues also get. allow_remote: controls willing download files location order satisfy dependency. TRUE, run report, might download files recent packets available location locally. pull_metadata: effect allow_remote also TRUE; causes metadata refreshed dependency resolution. discussion details ?orderly_run","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"other-points","dir":"Articles","previous_headings":"","what":"Other points","title":"Dependencies between packets","text":"used systems like targets, easy make reports smaller need . ’s real need make small, picking right size challenge. small, ’ll end writing lot code orchestrate running different reports pulling things together. ’ll end spending lot time whether things “date” one another really group things always wants run together. ’re big might end work want , order make change part piece analysis must run whole thing .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"installation","dir":"Articles","previous_headings":"","what":"Installation","title":"Introduction to orderly2","text":"","code":"install.packages(   \"orderly2\",   repos = c(\"https://mrc-ide.r-universe.dev\", \"https://cloud.r-project.org\"))"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"creating-an-empty-orderly-repository","dir":"Articles","previous_headings":"","what":"Creating an empty orderly repository","title":"Introduction to orderly2","text":"first step initialise empty orderly2 repository. orderly2 repository directory file orderly_config.yml within , since version 2 also directory .outpack/. Files within .outpack/ directory never directly modified users directory excluded version control (see orderly2::orderly_gitignore_update). Create orderly2 repository calling orderly2::orderly_init(): creates files: step performed completely empty directory, otherwise error thrown. Later, re-initialise orderly2 repository cloning new machine, working others; discussed vignette(\"collaboration\"). orderly_config.yml file contains little default: vignette, created orderly root R’s per-session temporary directory, deleted R exits. want use directory persist across restarting R (certainly want using orderly2 real project!) replace path within home directory, location control. rest vignette evaluate commands within directory, changing directory path ’ve created:","code":"path <- tempfile() # we'll use a temporary directory here - see note below orderly2::orderly_init(path) ## ✔ Created orderly root at '/tmp/RtmpkGakDB/file1b0bcebb068' ## . ## ├── .outpack ## │   ├── config.json ## │   ├── location ## │   └── metadata ## └── orderly_config.yml minimum_orderly_version: \"1.99.0\" setwd(path)"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"creating-your-first-orderly-report","dir":"Articles","previous_headings":"","what":"Creating your first orderly report","title":"Introduction to orderly2","text":"orderly report directory src/<name> containing orderly file <name>.R. file may special commands , now ’ll create one simple possible; ’ll create random data save disk. seems silly, imagine standing something like: downloading file external site resource running simulation saving output fitting model data merging set files together create final data set directory structure (ignoring .outpack) looks like: src/incoming_data/incoming_data.R contains: run report create new packet, use orderly2::orderly_run(): id created new identifier packet unique among packets (within reason) chronologically sortable. packet id sorts another packet’s id started packet. run report, directory structure looks like: things changed : file created ran report (data.rds; see script ) log happened report run packet created incoming_data.R data.csv, original input come source tree empty directory draft/incoming_data created orderly ran report first place addition, quite files changed within .outpack directory, covered . ’s ! Notice initial script just plain R script, can develop interactively within src/incoming_data directory. Note however, paths referred within relative src/incoming_data orderly repository root. important reports see world relative incoming_data.R file. created, can refer report id pull files wherever need , context another orderly report just copy desktop email someone. example, copy file data.rds created location outside orderly’s control copies data.rds new temporary directory dest name final.rds. uses orderly2’s outpack_ functions, designed interact outpack archives regardless created (orderly2 program creates outpack archives). Typically lower-level orderly_ functions.","code":"## . ## ├── orderly_config.yml ## └── src ##     └── incoming_data ##         ├── data.csv ##         └── incoming_data.R d <- read.csv(\"data.csv\") d$z <- resid(lm(y ~ x, d)) saveRDS(d, \"data.rds\") id <- orderly2::orderly_run(\"incoming_data\") ## ℹ Starting packet 'incoming_data' `20240903-120337-0e6713a8` at 2024-09-03 12:03:37.060674 ## > d <- read.csv(\"data.csv\") ## > d$z <- resid(lm(y ~ x, d)) ## > saveRDS(d, \"data.rds\") ## ✔ Finished running incoming_data.R ## ℹ Finished 20240903-120337-0e6713a8 at 2024-09-03 12:03:37.125051 (0.06437659 secs) id ## [1] \"20240903-120337-0e6713a8\" ## . ## ├── archive ## │   └── incoming_data ## │       └── 20240903-120337-0e6713a8 ## │           ├── data.csv ## │           ├── data.rds ## │           └── incoming_data.R ## ├── draft ## │   └── incoming_data ## ├── orderly_config.yml ## └── src ##     └── incoming_data ##         ├── data.csv ##         └── incoming_data.R dest <- tempfile() fs::dir_create(dest) orderly2::orderly_copy_files(id, files = c(\"final.rds\" = \"data.rds\"),                              dest = dest)"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"depending-on-packets-from-another-report","dir":"Articles","previous_headings":"","what":"Depending on packets from another report","title":"Introduction to orderly2","text":"Creating new dataset mostly useful someone else can use . introduce first special orderly commands can use orderly file src/ directory now looks like: src/analysis/analysis.R contains: , ’ve used orderly2::orderly_dependency() pull file data.rds recent version (latest()) data packet filename incoming.rds, ’ve used file normal make plot, ’ve saved analysis.png. can run just , using orderly2::orderly_run(): information dependencies, see vignette(\"dependencies\").","code":"## src ## ├── analysis ## │   └── analysis.R ## └── incoming_data ##     ├── data.csv ##     └── incoming_data.R orderly2::orderly_dependency(\"incoming_data\", \"latest()\",                              c(\"incoming.rds\" = \"data.rds\")) d <- readRDS(\"incoming.rds\") png(\"analysis.png\") plot(y ~ x, d) dev.off() id <- orderly2::orderly_run(\"analysis\") ## ℹ Starting packet 'analysis' `20240903-120337-93402a04` at 2024-09-03 12:03:37.57766 ## > orderly2::orderly_dependency(\"incoming_data\", \"latest()\", ## +                              c(\"incoming.rds\" = \"data.rds\")) ## ℹ Depending on incoming_data @ `20240903-120337-0e6713a8` (via latest(name == \"incoming_data\")) ## > d <- readRDS(\"incoming.rds\") ## > png(\"analysis.png\") ## > plot(y ~ x, d) ## > dev.off() ## agg_png  ##       2 ## ✔ Finished running analysis.R ## ℹ Finished 20240903-120337-93402a04 at 2024-09-03 12:03:37.660659 (0.08299875 secs)"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"available-in-report-orderly-commands","dir":"Articles","previous_headings":"","what":"Available in-report orderly commands","title":"Introduction to orderly2","text":"function orderly2::orderly_dependency() designed operate packet runs. functions act adding metadata final packet, perhaps copying files directory. orderly2::orderly_description(): Provide longer name description report; can reflected tooling uses orderly metadata much informative short name. orderly2::orderly_parameters(): Declares parameters can passed control behaviour report. Parameters key-value pairs simple data (booleans, numbers, strings) report can respond . can also used queries orderly2::orderly_dependency() find packets satisfy criteria. orderly2::orderly_resource(): Declares file resource; file input report, comes source directory. default, orderly treats files directory resource, can useful mark explicitly, necessary “strict mode” (see ). Files marked resource immutable may deleted modified. orderly2::orderly_shared_resource(): Copies file “shared resources” directory shared/, can data files source code located root orderly repository. can reasonable way sharing data commonly used code among several reports. orderly2::orderly_artefact(): Declares file (set files) created report, even run. makes easier check report behaves expected can allow reasoning related set reports without running . declaring something artefact (especially conjunction “strict mode”) also easier clean src directories used interactive development (see ). orderly2::orderly_dependency(): Copy files one packet packet runs, seen . orderly2::orderly_strict_mode(): Declares report run “strict mode” (see ). addition, also function orderly::orderly_run_info() can used running report returns information currently running report (id, resolved dependencies etc). Let’s add additional annotations previous reports: , ’ve added block special orderly commands; go anywhere, example files refer . strict mode enabled (see ) orderly2::orderly_resource calls must go files used made available point (see ).","code":"orderly2::orderly_strict_mode() orderly2::orderly_resource(\"data.csv\") orderly2::orderly_artefact(\"Processed data\", \"data.rds\")  d <- read.csv(\"data.csv\") d$z <- resid(lm(y ~ x, d)) saveRDS(d, \"data.rds\") id <- orderly2::orderly_run(\"incoming_data\") ## ℹ Starting packet 'incoming_data' `20240903-120337-deea8c28` at 2024-09-03 12:03:37.872477 ## > orderly2::orderly_strict_mode() ## > orderly2::orderly_resource(\"data.csv\") ## > orderly2::orderly_artefact(\"Processed data\", \"data.rds\") ## Warning: Please use a named argument for the description in 'orderly_artefact()' ## In future versions of orderly, we will change the order of the arguments to ## 'orderly_artefact()' so that 'files' comes first. If you name your calls to ## 'description' then you will be compatible when we make this change. ## > d <- read.csv(\"data.csv\") ## > d$z <- resid(lm(y ~ x, d)) ## > saveRDS(d, \"data.rds\") ## ✔ Finished running incoming_data.R ## ! 1 warning found: ## • Please use a named argument for the description in 'orderly_artefact()' In ## future versions of orderly, we will change the order of the arguments to ## 'orderly_artefact()' so that 'files' comes first. If you name your calls to ## 'description' then you will be compatible when we make this change. ## ℹ Finished 20240903-120337-deea8c28 at 2024-09-03 12:03:37.919764 (0.04728723 secs)"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"parameterised-reports","dir":"Articles","previous_headings":"","what":"Parameterised reports","title":"Introduction to orderly2","text":"Much flexibility comes orderly graph comes using parameterised reports; reports take set parameters change behaviour based parameters. Downstream reports can depend parameterised report filter based suitable parameters. example, consider simple report generate samples based parameter: creates report single parameter n_samples default value 10. used define parameter default, defined multiple parameters can anything report switches value parameter: might read different URLs fetch different underlying data might fit different analysis might read different shared resources (see ) might depend different dependencies might produce different artefacts However, see parameters relatively heavyweight things try consistent set packets created report. report use control size generated data set. resulting file 15 rows, parameter passed affected report: can use parameters orderly’s search functions. example can find recent version packet running: can also pass parameter queries : can used within orderly2::orderly_dependency() (name == \"random\" part implied first name argument), example case report querying also parameters can use within query, using prefix. suppose downstream report simply uses n number samples might write: depend recent packet called random parameter n_samples value current report’s parameter n. See outpack query documentation much detail .","code":"orderly2::orderly_parameters(n_samples = 10) x <- seq_len(n_samples) d <- data.frame(x = x, y = x + rnorm(n_samples)) saveRDS(d, \"data.rds\") orderly2::orderly_parameters(n_samples = NULL) orderly2::orderly_parameters(n_samples = 10, distribution = \"normal\") id <- orderly2::orderly_run(\"random\", list(n_samples = 15)) ## ℹ Starting packet 'random' `20240903-120338-2315eb1e` at 2024-09-03 12:03:38.139638 ## ℹ Parameters: ## • n_samples: 15 ## > orderly2::orderly_parameters(n_samples = 10) ## > x <- seq_len(n_samples) ## > d <- data.frame(x = x, y = x + rnorm(n_samples)) ## > saveRDS(d, \"data.rds\") ## ✔ Finished running random.R ## ℹ Finished 20240903-120338-2315eb1e at 2024-09-03 12:03:38.167204 (0.02756643 secs) orderly2::orderly_copy_files(id, files = c(\"random.rds\" = \"data.rds\"),                              dest = dest) readRDS(file.path(dest, \"random.rds\")) ##     x          y ## 1   1  0.4463006 ## 2   2  2.6289820 ## 3   3  5.0650249 ## 4   4  2.3690106 ## 5   5  5.5124269 ## 6   6  4.1369885 ## 7   7  6.4779875 ## 8   8  7.9473981 ## 9   9  9.5429963 ## 10 10  9.0859252 ## 11 11 11.4681544 ## 12 12 12.3629513 ## 13 13 11.6954565 ## 14 14 14.7377763 ## 15 15 16.8885049 orderly2::orderly_search('latest(name == \"random\")') ## [1] \"20240903-120338-2315eb1e\" orderly2::orderly_search('latest(name == \"random\" && parameter:n_samples > 10)') ## [1] \"20240903-120338-2315eb1e\" orderly2::orderly_dependency(\"random\", \"latest(parameter:n_samples > 10)\",                              c(\"randm.rds\" = \"data.rds\")) orderly2::orderly_dependency(\"random\", \"latest(parameter:n_samples == this:n)\",                              c(\"randm.rds\" = \"data.rds\"))"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"shared-resources","dir":"Articles","previous_headings":"","what":"Shared resources","title":"Introduction to orderly2","text":"Sometimes useful share data different reports, example common source utilities don’t warrant package, common data. , create directory shared orderly root put files directories might want share. Suppose shared directory contains file data.csv: can write orderly report use_shared uses shared file, use_shared.R containing: can run : resulting archive, file used shared directory present: general property orderly: tries save inputs alongside final results analysis, later can check see went analysis might changed versions.","code":"## . ## ├── archive ## │   ├── analysis ## │   │   └── 20240903-120337-93402a04 ## │   │       ├── analysis.R ## │   │       ├── analysis.png ## │   │       └── incoming.rds ## │   ├── incoming_data ## │   │   ├── 20240903-120337-0e6713a8 ## │   │   │   ├── data.csv ## │   │   │   ├── data.rds ## │   │   │   └── incoming_data.R ## │   │   └── 20240903-120337-deea8c28 ## │   │       ├── data.csv ## │   │       ├── data.rds ## │   │       └── incoming_data.R ## │   └── random ## │       └── 20240903-120338-2315eb1e ## │           ├── data.rds ## │           └── random.R ## ├── draft ## │   ├── analysis ## │   ├── incoming_data ## │   └── random ## ├── orderly_config.yml ## ├── shared ## │   └── data.csv ## └── src ##     ├── analysis ##     │   └── analysis.R ##     ├── incoming_data ##     │   ├── data.csv ##     │   └── incoming_data.R ##     └── random ##         └── random.R orderly2::orderly_shared_resource(\"data.csv\") orderly2::orderly_artefact(\"analysis\", \"analysis.png\")  d <- read.csv(\"data.csv\") png(\"analysis.png\") plot(y ~ x, d) dev.off() id <- orderly2::orderly_run(\"use_shared\") ## ℹ Starting packet 'use_shared' `20240903-120338-a291ae09` at 2024-09-03 12:03:38.63762 ## > orderly2::orderly_shared_resource(\"data.csv\") ## > orderly2::orderly_artefact(\"analysis\", \"analysis.png\") ## Warning: Please use a named argument for the description in 'orderly_artefact()' ## In future versions of orderly, we will change the order of the arguments to ## 'orderly_artefact()' so that 'files' comes first. If you name your calls to ## 'description' then you will be compatible when we make this change. ## > d <- read.csv(\"data.csv\") ## > png(\"analysis.png\") ## > plot(y ~ x, d) ## > dev.off() ## agg_png  ##       2 ## ✔ Finished running use_shared.R ## ! 1 warning found: ## • Please use a named argument for the description in 'orderly_artefact()' In ## future versions of orderly, we will change the order of the arguments to ## 'orderly_artefact()' so that 'files' comes first. If you name your calls to ## 'description' then you will be compatible when we make this change. ## ℹ Finished 20240903-120338-a291ae09 at 2024-09-03 12:03:38.685079 (0.04745865 secs) ## archive/use_shared ## └── 20240903-120338-a291ae09 ##     ├── analysis.png ##     ├── data.csv ##     └── use_shared.R"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"strict-mode","dir":"Articles","previous_headings":"","what":"Strict mode","title":"Introduction to orderly2","text":"previous version orderly (orderly1; see vignette(\"migrating\")) fussy input strictly declared report run, clear really required order run something. version 2 relaxed default, can opt old behaviours checks adding anywhere within orderly file (conventionally top). may make granular future, adding : copy files source directory (src/<reportname>/) draft directory report runs (draft/<reportname>/<packet-id>) declared orderly2::orderly_resource; leaves behind extra files left development warn end running packet files found part artefact Using strict mode also helps orderly2 clean src/<reportname> directory effectively interactive development (see next section).","code":"orderly2::orderly_strict_mode()"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"interactive-development","dir":"Articles","previous_headings":"","what":"Interactive development","title":"Introduction to orderly2","text":"Set working directory src/<reportname> orderly script fully executable (e.g., source Rstudio’s Source button, R’s source() function). Dependencies copied needed. , mix files within source directory. recommend per-source-directory .gitignore keep files version control (see ). soon implement support cleaning generated files directory. example, suppose interactively run incoming_data/incoming_data.R script, leave behind generated files. can report orderly2::orderly_cleanup_status: files unknown orderly tell prompt tell explicitly. can clean generated files running (suggested message): dry_run = TRUE argument can pass want see deleted without using status function. can also keep files git using orderly2::orderly_gitignore_update function: creates (updates) .gitignore file within report generated files included git. already accidentally committed gitignore real effect git surgery, see git manuals handy, profane, guide.","code":"orderly2::orderly_cleanup_status(\"incoming_data\") ## ✖ incoming_data is not clean: ## ℹ 1 file can be deleted by running 'orderly2::orderly_cleanup(\"incoming_data\")': ##   • data.rds orderly2::orderly_cleanup(\"incoming_data\") ## ℹ Deleting 1 file from 'incoming_data': ## • data.rds orderly2::orderly_gitignore_update(\"incoming_data\") ## ✔ Wrote 'src/incoming_data/.gitignore'"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"deleting-things-from-the-archive","dir":"Articles","previous_headings":"","what":"Deleting things from the archive","title":"Introduction to orderly2","text":"delete packets archive/ directory puts orderly2 inconsistent state metadata store. Sometimes matter (e.g., delete old copies never candidates inclusion orderly2::orderly_dependency never notice). However, delete recent copy packet try depend , get error. moment, two copies incoming_data task: run analysis task, pull recent version (20240903-120337-deea8c28). However, deleted manually (e.g., save space accidentally) corrupted (e.g., opening output Excel letting save changes) able included, running analysis fail: error tries fairly informative, telling us failed copying files 20240903-120337-deea8c28 found packet corrupt, file data.rds found archive. also suggests fix; can tell orderly2 20240903-120337-deea8c28 “orphaned” considered inclusion look dependencies. can carry suggestion just validate packet running can validate packets : option core.require_complete_tree enabled, process also look packets used now-deleted packet orphan , longer complete tree includes . want remove references orphaned packets, can use orderly2::orderly_prune_orphans() remove entirely:","code":"orderly2::orderly_metadata_extract(   name = \"incoming_data\",   extract = c(time = \"time.start\")) ##                         id                time ## 1 20240903-120337-0e6713a8 2024-09-03 12:03:37 ## 2 20240903-120337-deea8c28 2024-09-03 12:03:37 orderly2::orderly_run(\"analysis\") ## ℹ Starting packet 'analysis' `20240903-120339-46b913ff` at 2024-09-03 12:03:39.278753 ## > orderly2::orderly_dependency(\"incoming_data\", \"latest()\", ## +                              c(\"incoming.rds\" = \"data.rds\")) ## ✖ Error running analysis.R ## ℹ Finished 20240903-120339-46b913ff at 2024-09-03 12:03:39.356496 (0.07774282 secs) ## Error in `orderly2::orderly_run()`: ## ! Failed to run report ## Caused by error in `orderly_copy_files()`: ## ! Unable to copy files, due to deleted packet 20240903-120337-deea8c28 ## ℹ Consider 'orderly2::orderly_validate_archive(\"20240903-120337-deea8c28\", ##   action = \"orphan\")' to remove this packet from consideration ## Caused by error: ## ! File not found in archive ## ✖ data.rds orderly2::orderly_validate_archive(\"20240903-120337-deea8c28\", action = \"orphan\") orderly2::orderly_validate_archive(action = \"orphan\") ## ✔ 20240903-120337-0e6713a8 (incoming_data) is valid ## ✔ 20240903-120337-93402a04 (analysis) is valid ## ✖ 20240903-120337-deea8c28 (incoming_data) is invalid due to its files ## ✔ 20240903-120338-2315eb1e (random) is valid ## ✔ 20240903-120338-a291ae09 (use_shared) is valid orderly2::orderly_prune_orphans() ## ℹ Pruning 1 orphan packet"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"debugging-and-coping-with-errors","dir":"Articles","previous_headings":"","what":"Debugging and coping with errors","title":"Introduction to orderly2","text":"(written)","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"interaction-with-version-control","dir":"Articles","previous_headings":"","what":"Interaction with version control","title":"Introduction to orderly2","text":"guidelines: Make sure exclude files git listing .gitignore: .outpack/ - nothing suitable version control archive/ - core.archive_path set non-null value, excluded. default archive draft/ - temporary draft directory orderly_envir.yml - used setting machine-specific configuration absolutely version control files: src/ main source analyses orderly_config.yml - high level configuration suitable sharing shared resource directory (configured orderly_config.yml) probably version controlled source repository end multiple people’s machines, configured differently. configuration option set via orderly2::orderly_config_set designed (potentially) different different users, configuration needs version controlled. also means reports/packets can’t directly refer values set . includes directory used save archive packets (enabled) names locations (equivalent git remotes). may find useful include scripts help users set common locations, like git, different users may use different names remote (e.g., one user may location called data another called data-incoming, depending perspective use location). orderly2 always try save information current state git source repository alongside packet metadata. includes current branch, commit (sha) remote url. try create links final version packet upstream source repository.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"interaction-with-the-outpack-store","dir":"Articles","previous_headings":"","what":"Interaction with the outpack store","title":"Introduction to orderly2","text":"alluded , .outpack directory contains lots information packets run, typically “bounds” normal use. effectively “database” information packets run. Understanding directory structured required using orderly, included avoidance mystery! See outpack documentation (vignette(\"outpack\")) details ideas . work , directory structure looks like: can perhaps inferred filenames, files .outpack/metadata/<packet-id> metadata packet run. files .outpack/location/<location-id>/<packet-id> holds information packet first known location (location special “local” location). default orderly configuration store final files directory called archive/, alternatively (additionally) can use content- addressable file store. enabled, .outpack directory looks like: files .outpack/files/ never modified deleted. approach storage naturally deduplicates file archive, large file used many places ever stored .","code":"## .outpack ## ├── config.json ## ├── index ## │   └── outpack.rds ## ├── location ## │   ├── local ## │   │   ├── 20240903-120337-0e6713a8 ## │   │   ├── 20240903-120337-93402a04 ## │   │   ├── 20240903-120338-2315eb1e ## │   │   └── 20240903-120338-a291ae09 ## │   └── orphan ## └── metadata ##     ├── 20240903-120337-0e6713a8 ##     ├── 20240903-120337-93402a04 ##     ├── 20240903-120338-2315eb1e ##     └── 20240903-120338-a291ae09 ## .outpack ## ├── config.json ## ├── files ## │   └── sha256 ## │       ├── 0a ## │       │   └── a82571c21c4e5f1f435e8bef2328dda5ef47e177d78d63d1c4ec647a5a388a ## │       ├── 25 ## │       │   └── 4947c281b203719c72949745123a1d017e2f9b50c048b1d24a0803d73ba0b8 ## │       ├── 40 ## │       │   └── f8c6c3993686ee8acf38ce936911d6f65372d489e78748ec6d1f27d277daa7 ## │       ├── 42 ## │       │   └── 438203a847c7cd496e3e603ddeeceb4e136f1bd18f1efb81332fa093964f7b ## │       ├── 51 ## │       │   └── f8070f04f57a8040ad215474b976bcddc891bfc544e6c6a0519e74d79cb104 ## │       ├── 5f ## │       │   └── 96f49230c2791c05706f24cb2335cd0fad5d3625dc6bca124c44a51857f3f8 ## │       ├── a6 ## │       │   └── 80ab7c65a52327a3d9c5499d114f513f18eabe7f63a98f9fc308c2b3744c82 ## │       ├── d9 ## │       │   └── 1699ae410cbd811e1f028f8a732e5162b7df854eec08d921141f965851272d ## │       ├── de ## │       │   └── 1329bc0e9f8ee0c8b3376eb09b71e93587f707dfeaa01d1a03ae32f97c928a ## │       └── ec ## │           └── b53285781a4d36c65168c80ee14f2af2c885423c6166b9425f40c3c6cd8297 ## ├── index ## │   └── outpack.rds ## ├── location ## │   ├── local ## │   │   ├── 20240903-120337-0e6713a8 ## │   │   ├── 20240903-120337-93402a04 ## │   │   ├── 20240903-120338-2315eb1e ## │   │   └── 20240903-120338-a291ae09 ## │   └── orphan ## └── metadata ##     ├── 20240903-120337-0e6713a8 ##     ├── 20240903-120337-93402a04 ##     ├── 20240903-120338-2315eb1e ##     └── 20240903-120338-a291ae09"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"relationship-between-orderly-and-outpack","dir":"Articles","previous_headings":"","what":"Relationship between orderly and outpack","title":"Introduction to orderly2","text":"orderly2 package built metadata file storage system called outpack; implementing support working metadata archives languages (see outpack_server server implementation Rust outpack-py Python). metadata discussed detail vignette(\"metadata\") document general ideas fully mrc-ide/outpack","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"basic-overview","dir":"Articles","previous_headings":"","what":"Basic overview","title":"Outpack metadata","text":"“packet” conceptually directory, corresponding particular analysis data product, though necessarily stored. internal representation includes: metadata packet - files contains hashes along information done create environment created etc (see ) actual file contents every included file. might stored simple content-addressable file storage system (see ) actual copies directories originally created Every packet referenced uniquely primary key. use key format encodes current date time, well random data avoid collisions. exists dependency graph among packets, one packet depends another. edge graph hard link (one packet another id) also query (e.g., latest packet name) used find packet. means many ways looking thinking dependency graph. packets available locally, outpack repositories, typically (always) machines accessed HTTP API. conceptually similar git “remotes”. need distinguish packets “unpacked” (, packets every file available current archive) packets merely known (metadata files). sometimes refer unpacked packets “local” known “local” location special. use terms “archive” “repository” fairly interchangeably try nail .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"what-goes-into-a-packet","dir":"Articles","previous_headings":"","what":"What goes into a packet","title":"Outpack metadata","text":"packet must things: name (example model_fits). changed (rather changes tracked) currently way namespacing different repositories id (see “outpack id” ) created outpack point packet started set parameters - mapping strings scalar type (boolean, number, string). Packets may zero many parameters. set files - local (relative) path within packet (e.g., output/data.csv) also hash (e.g., sha256:69f6cf230416cf40828da251a0dad17cbbf078587883e826f3345ff08d1aaa7d) Information packet run (start end time), represented seconds post epoch, UTC (see ). addition may contain information : dependencies used, tracking name included packet, query used find , resolved id, set files extracted packet (source name, hash destination name) names scripts run create packet (typically also included files within final packet) information state git point packet run (current branch, hash, etc) information system used run packet (format unspecified might include hostname, operating system version, package/module versions, etc) custom metadata added implementation","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"types-of-users-of-outpack","dir":"Articles","previous_headings":"","what":"Types of users of outpack","title":"Outpack metadata","text":"types “persona” outpack user imagine exist guide decisions abut layout . extremes : user desktop wants develop run analyses, using files present packets available server. want visually inspect results running packets, need see files disk way makes sense human, care complete copy dependencies graph. server instance responsible delivering packets users (e.g., persona ). expected (able fetch) part outpack graph, expect anyone want visually inspect packets directly machine (, storage need human-readable). impacts two configuration options associated parts directory structure : user use content-addressable “file store” human-readable “archive” (explained detail ) require archive contains complete tree dependencies? expect first persona wants human readable archive contain full tree, second wants opposite.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"directory-layout","dir":"Articles","previous_headings":"","what":"Directory layout","title":"Outpack metadata","text":"section discusses files directory make outpack work, much come ; see . typical .outpack directory layout looks like : (note archive/ .outpack level). directories necessarily present; indeed required file .outpack/config.json.","code":".outpack/   config.json   files/   location/   metadata/ archive/"},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"configuration--outpackconfig-json","dir":"Articles","previous_headings":"Directory layout","what":"Configuration (.outpack/config.json)","title":"Outpack metadata","text":"outpack configuration schema defined config.json configuration format still subject change…","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"packet-metadata--outpackmetadata","dir":"Articles","previous_headings":"Directory layout","what":"Packet metadata (.outpack/metadata/)","title":"Outpack metadata","text":"file within directory filename outpack id (matching regular expression ^[0-9]{8}-[0-9]{6}-[0-9a-f]{8}$, see . file json file conforming schema metadata.json. present means outpack implementation can report information back packet (created, files contains, etc), packet metadata meaningful ; want know might come (location distributing packet) copy packet locally.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"location-information--outpacklocation","dir":"Articles","previous_headings":"Directory layout","what":"Location information (.outpack/location/)","title":"Outpack metadata","text":"directory matches regular expression ^[0-9]{8}$ (e.g., 457f4f2a) “location id” (see ) corresponding “location”. file within directory outpack id name, contains json location unpacked (installed) packet, hash metadata. file conforms schema location.json.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"a-file-store--outpackfiles","dir":"Articles","previous_headings":"Directory layout","what":"A file store (.outpack/files)","title":"Outpack metadata","text":"configuration option core.use_file_store true, outpack keeps content addressable file store files knows . much space efficient entire packet unpacked automatically deduplicates shared content among packets (e.g., large file present two packets stored ). file store layout described . storage format human-readable (indeed present within hidden directory .outpack). can enabled either server user","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"an-archive-archive-by-default","dir":"Articles","previous_headings":"Directory layout","what":"An archive (archive/ by default)","title":"Outpack metadata","text":"configuration option core.path_archive non-null directory path containing unpacked packets. packet available path <name> “name” packet, <id> outpack id. several files per packet, possibly directories. storage approach designed human readable, typically enabled outpack repository used laptop user wants interactively work files.","code":"archive/<name>/<id>/<files...>"},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"adding-a-packet","dir":"Articles","previous_headings":"","what":"Adding a packet","title":"Outpack metadata","text":"order make packet available locally, need import metadata files, mark packet available. roughly creating packet (.e., first place packet ever existed) importing packet elsewhere. Making packet available allows used dependency, allows serving packet acting location (http file protocols), guarantees files actually present locally.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"adding-metadata","dir":"Articles","previous_headings":"Adding a packet","what":"Adding metadata","title":"Outpack metadata","text":"can simply copy metadata file .outpack/metadata/<packet id> yet exist. make available anything yet known location. Dangling metadata (, metadata present directory known anywhere) currently mostly ignored.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"adding-files","dir":"Articles","previous_headings":"Adding a packet","what":"Adding files","title":"Outpack metadata","text":"repository uses file store, fill first, much easier think . can easily get difference files used packet (list files packet manifest) already file store looking hash turn. request missing files insert store. may leave “dangling” files (files referred packet) problem. repository human-readable archive uses file store, files present file store easy enough check file store requested path (local relative path packet manifest). update file store first, files guaranteed present. repository uses human readable archive, simplest thing request file remote. However, might efficient check locally previously fetched copies files content, verify modified, copy place rather re-downloading.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"marking-the-packet-as-unpacked-and-known-locally","dir":"Articles","previous_headings":"Adding a packet","what":"Marking the packet as unpacked and known locally","title":"Outpack metadata","text":"local location id, write file conforming location.json schema, containing packet id, time marked unpacked hash metadata.","code":".outpack/<local location id>/<packet id>"},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"ordering-of-operations","dir":"Articles","previous_headings":"Adding a packet","what":"Ordering of operations","title":"Outpack metadata","text":"need files metadata packet marked unpacked; note configurations guarantee every packet unpacked complete tree. can import files first, metadata; lot disadvantage either. mark package unpacked known locally though components present.","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"outpack-ids","dir":"Articles","previous_headings":"Details","what":"Outpack ids","title":"Outpack metadata","text":"Outpack ids match regular expression ^[0-9]{8}-[0-9]{6}-[0-9a-f]{8}$; encode UTC date-time prefix YYYYMMDD-HHMMSS- followed 8 hexadecimal digits. R implementation, encode current second first four digits (2 bytes) append 2 bytes cryptographically random data. id tries balance reasonable degree collision resistance (65536 combinations per millisecond), lexicographic sortability reasonable degree meaningfulness.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"location-ids","dir":"Articles","previous_headings":"Details","what":"Location ids","title":"Outpack metadata","text":"Location ids meaningless 4-byte (8 character) hex strings. immutable created different different machines even point location. location id mapped (via .outpack/config.json) location name human-readable name (e.g., production staging). requirement name different machines. One directories represents local location; can find mapping within configuration.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"representing-hashes","dir":"Articles","previous_headings":"Details","what":"Representing hashes","title":"Outpack metadata","text":"Outpack typically uses sha256 hashes, want able change future. wherever hash presented, algorithm included part string. example instead used md5 algorithm written schema currently assumes hash value represented hex string.","code":"sha256:69f6cf230416cf40828da251a0dad17cbbf078587883e826f3345ff08d1aaa7d md5:bd57f7123c6bfb95c3234ff56373b7f4"},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"times","dir":"Articles","previous_headings":"Details","what":"Times","title":"Outpack metadata","text":"store information times places (e.g., times packet run, imported, etc). Rather trying deal strings, always store time seconds since 1970-01-01 00:00.00 UTC (including fractional seconds, whatever accuracy system allows).","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"file-store","dir":"Articles","previous_headings":"Details","what":"File store","title":"Outpack metadata","text":"file store designed simple, sophisticated git, whose object store similar thing. general layout looks like: hopefully fairly obvious structure. Paths format: reason second level prevent performance degradation directories containing millions files, copying git. store designed cope different hashing algorithms, though R implementation outpack supports sha256 now. Multiple hashing algorithms supported hard linking content multiple places tree, might link ","code":".outpack/files   sha256/     5d/       dfaf1f4a2e15e8fe46dbed145bf2f84bba1b3367d0a56f73de08f8585dd153       ...     77/     ... <algorithm>/<first two bytes>/<remaining bytes> sha256/5d/dfaf1f4a2e15e8fe46dbed145bf2f84bba1b3367d0a56f73de08f8585dd153 md5/84/0bc6ad3ae479dccc1c49a1910b37bd"},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"so-long-yaml-and-thanks-for-all-the-whitespace-errors","dir":"Articles","previous_headings":"Summary of changes","what":"So long YAML and thanks for all the whitespace errors","title":"Migrating from orderly (1.x)","text":"obvious user-facing change (almost) YAML, definition inputs outputs report now defined within orderly file, <reportname>.R. orderly report previously orderly.yml file looked like end within orderly file looks like: think much clearer, comes documentation autocomplete support IDEs. fact, simple reports, special functions required, though ’ll find useful (see vignette(\"orderly2\")) specific changes: longer need list packages packages:, instead just use library() ordinary script. record state session regardless get record used longer need use sources: list scripts want source, instead use source() normal global_resources become orderly2::orderly_shared_resource (aren’t really global much shared). Note directory now always shared/ orderly root, may configure .","code":"parameters:   n_min:     default: 10 script: script.R source:   - functions.R resources:   - metadata.csv depends:   raw_data:     id: latest     use:       raw_data.csv: data.csv artefacts:   data:     description: Processed data     filenames: data.rds orderly2::orderly_parameters(n_min = 10) orderly2::orderly_dependency(\"raw_data\", \"latest\",                               files = c(\"raw_data.csv\" = \"data.csv\")) orderly2::orderly_resource(\"metadata.csv\") orderly2::orderly_artefact(\"Processed data\", \"data.rds\") source(\"functions.R\")"},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"implications","dir":"Articles","previous_headings":"Summary of changes > So long YAML and thanks for all the whitespace errors","what":"Implications","title":"Migrating from orderly (1.x)","text":"change widespread implications: can program things like dependencies, creating loop series parameter values, conditionally depending reports R script can basis orderly report, can add orderly functions annotating inputs outputs progressively","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"database-support-has-been-moved-into-a-plugin","dir":"Articles","previous_headings":"Summary of changes","what":"Database support has been moved into a plugin","title":"Migrating from orderly (1.x)","text":"version 1, built-support accessing data SQL databases; moved within orderly.db plugin. major features supported.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"no-more-commit","dir":"Articles","previous_headings":"Summary of changes","what":"No more commit","title":"Migrating from orderly (1.x)","text":"orderly2 longer requires separate orderly_commit() call orderly_run(); longer make distinction local draft archive packets. Instead, added finer-grained control dependencies resolved (locally, subset servers), generalises way draft/archive used practice. See ?orderly_run details dependencies resolved. implications deleting things; draft directory always easy target deletion, now deletion need tell orderly2 deleted things. See vignette(\"introduction\") details (section “Deleting things archive”).","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"no-more-testing-or-development-mode","dir":"Articles","previous_headings":"Summary of changes","what":"No more testing or development mode","title":"Migrating from orderly (1.x)","text":"two different, unsatisfactory, mechanisms developing orderly report: orderly_test_start (version 1.1.0) orderly_develop_start (1.1.0 onwards) worked initial setup, copying dependencies etc location work (new draft orderly_test_start source directory orderly_develop_start). orderly2 can just work directly within source directory, long working directory set src/<report-name>, orderly2 commands work expected. orderly1, need careful commit (git) results running analysis, encourage per-report .gitignore files help .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"new-language-agnostic-backend","dir":"Articles","previous_headings":"Summary of changes","what":"New, language-agnostic, backend","title":"Migrating from orderly (1.x)","text":"biggest change, perhaps least visible, orderly now built open spec outpack can implemented language. develop Python implementation , possibly languages. takes control metadata. split “orderly_” “outpack_” functions package, information see last section vignette(\"introduction\") also vignette(\"outpack\").","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"other-smaller-changes","dir":"Articles","previous_headings":"Summary of changes","what":"Other, smaller changes","title":"Migrating from orderly (1.x)","text":"“Global” resources become “shared” resources, always live shared/ orderly root (.e., longer configurable). reason want reports able run fairly independently orderly2 configuration (exception plugins). practice people really vary .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"what-is-missing-compared-with-orderly1","dir":"Articles","previous_headings":"","what":"What is missing compared with orderly1","title":"Migrating from orderly (1.x)","text":"changelog feature (implement something support ) extraction secrets vault (support ) automatic handling README files (may implement support )","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"how-to-migrate","dir":"Articles","previous_headings":"","what":"How to migrate","title":"Migrating from orderly (1.x)","text":"two parts migration: updating canonical copy orderly archive (ideally one ) updating source tree. steps done via outpack.orderly package. migrate archive first. every archive want retain (might archives stored locally, production servers staging servers). Archive migration happens place; , modify anything original location. archive old used old versions orderly1 possible process hiccups. Please let us know case. result process end new directory contains new archive conforming outpack spec containing orderly2 metadata. Next, migrate source tree. done place done fresh clone source git repository. report, examine orderly.yml files script files (often script.R), delete , write new orderly file adapt report work orderly2. possible perfect might need minor tweaking hopefully reasonable. One thing preserved (probably ) comments yaml often refer yaml formatting orderly1 features hopefully much problem. probably want manually tweak generated code anyway, take advantage new orderly2 features able compute dependencies. using OrderlyWeb, probably need pause migrating, replacement yet ready.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"what-about-the-packages","dir":"Articles","previous_headings":"","what":"What about the packages?","title":"Migrating from orderly (1.x)","text":"merge orderly2 orderly package, ready release can use . However, anticipate period coexistence legacy orderly1 systems develop orderly2. help small helper package orderly.helper can smooth namespace differences; may useful interact versions.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"what-were-the-problems-in-version-1","dir":"Articles","previous_headings":"","what":"What were the problems in version 1","title":"Migrating from orderly (1.x)","text":"YAML format inflexible, error prone users, leads duplication focussed around initial needs Vaccine Impact Modelling Consortium fairly easy get archive SQLite database inconsistent state (e.g., deleting moving files archive) SQLite database behaved poorly shared file systems","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/orderly2.html","id":"collaborative-analysis","dir":"Articles","previous_headings":"","what":"Collaborative analysis","title":"orderly2","text":"Many analyses involve single person single machine; case number workflow tools make orchestrating analysis easy. workflow model graph dependencies analysis, data flows. example might update data, whole pipeline rerun. update code forecast, forecasts report rerun. experience, model works well single-user setting falls collaborative setting, especially analysis partitioned person; Alice handling data pipeline, Bob running fits, Carol organising forecasts final report. context, changes upstream affect downstream analysis, require set care around integration might used version controlling source code. example, Alice dealing change incoming data format going break analysis time Bob trying get model fits working, Bob trying integrate code changes Alice’s new data. typically deal source code using branches within git; code Bob work branch isolated Alice’s changes. contexts like () data analysis products git. needed way versioning outputs step analysis controlling integrated subsequent analyses. Another way looking problem seek way making analysis composable way functions OOP achieve programs, way docker containerisation achieved deploying software. need way putting interfaces around pieces analysis allow people refer fetch somewhere run. conceptual pieces needed : way referring unambiguously (globally) piece analysis can depended upon, everyone can agree ’re talking piece analysis system storage results running analysis can shared among group system control integration pieces analysis takes place refer transportable unit analysis “packet”. conceptually directory files created running code, atomic unit work point view orderly. packet underlying source form, anyone can run. However, time people use pre-run packets collaborators run inputs onward analyses (see vignette(\"dependencies\") vignette(\"collaboration\") details).","code":"[raw data] -> [processed data] -> [model fits] -> [forecasts] -> [report]"},{"path":"https://mrc-ide.github.io/orderly2/articles/orderly2.html","id":"reproducible-analyses","dir":"Articles","previous_headings":"","what":"Reproducible analyses","title":"orderly2","text":"degree collaboration style requires reproducibility, several aspects . system describe , even though everyone can typically run step analysis, typically don’t. differs workflow tools, users may familiar .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/orderly2.html","id":"difference-from-a-workflow-system","dir":"Articles","previous_headings":"Reproducible analyses","what":"Difference from a workflow system","title":"orderly2","text":"Workflow systems hugely influential scientific computing, people co-opting build systems like make sophisticated systems designed parallel running large complex workflows nextflow. general approach define interdependencies among parts analysis, forming graph parts analysis track inputs outputs workflow. model computation lots good points: defines interface analysis allows (encourages) breaking monolithic analysis component pieces can reasoned allows high-level parallelism making obvious parts workflow can run concurrently tracking way data flows analysis, allows minimum amount recalculation done change, downstream parts triggered shared workspace online runner, allows degree collaboration long everyone happy working constantly changing set code analysis artefacts designed orderly2 working patterns suit . motivating reasons include: nodes computational graph expensive compute require exotic hardware. Workflows upstream data never settle, need know version data ends used particular analysis Workflows upstream analyses used many downstream analyses, upstream developer may know much downstream use Nondeterministic analyses, e.g., involving stochastic simulations, rerunning node expected return exact numerical results, can’t rely different users recovering results different occasions[*] cases missing piece need way versioning nodes within computational graph, shifting emphasis automatically rerunning portions graph tracking data flowed graph. turn shifts reproducibility emphasis “everyone run code get results” “everyone run code, instead work results”. familiar docker, approach similar working pre-built docker images, whereas workflow approach similar working directly Dockerfiles; many situations end result , approaches differ guarantees, computation happens, users refer versions. [*] discourage trying force determinism manually setting seeds, potential violate statistical properties random number streams, fragile best.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/orderly2.html","id":"what-is-reproducibility-anyway","dir":"Articles","previous_headings":"Reproducible analyses","what":"What is reproducibility anyway?","title":"orderly2","text":"Reproducibility means different things different people, even within narrow sense “rerun analysis retrieve results”. last decade, idea one able rerun piece analysis retrieve results slightly morphed one must rerun piece analysis. Similarly, emphasis utility reproducibility shifted authors able rerun work (confidence rerun ) hypothetical third party wanting rerun analysis. approach flips perspective around bit, based experiences collaborative research projects, draws (overly) ambitious aim : Can prove given set inputs produced given set outputs? quickly found impossible, provided systems place one satisfied statement given level trust system. piece analysis comes server primary way people run analyses web front-end (currently OrderlyWeb, soon Packit) know analysis run end--end modification orderly2 preserves inputs alongside outputs files present final packet files went analysis, recorded R package versions full set used. system naturally involves running multiple machines (typically analysts’ laptops, server perhaps HPC environment), way orderly2 treats paths, practically little problem getting analyses working multiple places, trivially satisfying typical reproducibility aim, even though people typically focussed . shift focus proved valuable. analysis run one occasion (e.g., regular reporting, simply updating figure final submission manuscript revision), outputs may change. Understanding changes happened important. orderly2 automatically saves lot metadata run easy find things might changed. , can start interrogating graph among packets find effect change ; find previously run packets pulled old version data set, used previous release package.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"the-basic-idea","dir":"Articles","previous_headings":"","what":"The basic idea","title":"Creating plugins","text":"plugin provided package, possibly thing package provides. plugin name must (currently) package name. functions package needs call orderly2::orderly_plugin orderly2::orderly_plugin_register create register plugin, respectively. make plugin available orderly project, two new bits configuration may present orderly_config.yml - one declares plugin used, configures plugin. use plugin individual report, functions plugin used, configure use plugin. Finally, can save information back final orderly2 metadata plugin . yaml-less design orderly2 (see vignette(\"migrating\") familiar orderly1), line plugin just package code fairly blurred, reasons writing plugin typically want make something easier reports, want action reflected orderly metadata.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"an-example","dir":"Articles","previous_headings":"","what":"An example","title":"Creating plugins","text":"example, ’ll implement stripped version database plugin inspired work (see `orderly.db fuller implementation). make work need functions: …process additional fields orderly_config.yml describe find database …can called orderly file access database …can add metadata final orderly metadata done ’ll start report side things, describing want happen, work implementation. directory structure minimal project orderly_config.yml file contains information shared possible uses plugin - case connection information database: plugin called example.db listed within plugins section, along configuration; case indicating path SQLite file can loaded . example.R file contains information use database specific report; case, making results query SELECT * mtcars cyl == 4 database available R object dat Normally, imagine calculation kept minimal purpose demonstration. implement need : create package write function handle configuration orderly_config.yml write function query() used example.R query ","code":"## . ## ├── orderly_config.yml ## └── src ##     └── example ##         └── example.R minimum_orderly_version: 1.99.0 plugins:   example.db:     path: /tmp/RtmpFxHGsW/file1bfd3c51fc2 dat <- example.db::query(\"SELECT * FROM mtcars WHERE cyl == 4\") orderly2::orderly_artefact(\"Summary of data\", \"data.rds\")  saveRDS(summary(dat), \"data.rds\")"},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"create-a-tiny-package","dir":"Articles","previous_headings":"An example","what":"Create a tiny package","title":"Creating plugins","text":"lots package skeleton tools , favourite, usethis::create_package() probably reasonable job. thing package needs contain Imports: orderly2 DESCRIPTION field. simple package may structure like , DESCRIPTION file contains: NAMESPACE R/plugin.R files shown .","code":"## . ## ├── DESCRIPTION ## ├── NAMESPACE ## └── R ##     └── plugin.R Package: example.db Version: 0.0.1 License: CC0 Title: Orderly Database Example Plugin Description: Simple example of an orderly plugin. Authors@R: person('Orderly Authors', role = c('aut', 'cre'),     email = 'email@example.com') Imports: orderly2"},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"handle-the-configuration","dir":"Articles","previous_headings":"An example","what":"Handle the configuration","title":"Creating plugins","text":"required function plugin needs provide one process data orderly_config.yml. probably primarily concerned validation can fairly simple first, later ’ll expand report errors nicely: arguments data: deserialised section orderly_config.yml specific plugin filename: full path orderly_config.yml return value data argument auxiliary data added validation.","code":"db_config <- function(data, filename) {   data }"},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"evaluate-the-query","dir":"Articles","previous_headings":"An example","what":"Evaluate the query","title":"Creating plugins","text":"Finally, minimal example, need function actually query; example example.db::query: arguments whatever want user provide – nothing special orderly2. important function call orderly2::orderly_plugin_context returns information can use make plugin work. explained ?orderly2::orderly_plugin_context, example use just one element, config, configuration plugin (.e., return value function db_config); see orderly2::orderly_plugin_context context can accessed . last bit package code register plugin, calling orderly2::orderly_plugin_register within .onLoad() special R function called package loaded. means whenever packages loaded (regardless whether attached) register plugin. (important name argument matches package name, orderly2 trigger loading package based name configuration; may support multiple plugins within one package later.) Note query function appear within registration, just function read process configuration. final (minimal) package code : NAMESPACE file contains","code":"query <- function(sql) {   ctx <- orderly2::orderly_plugin_context(\"example.db\")   dbname <- ctx$config$path   con <- DBI::dbConnect(RSQLite::SQLite(), dbname)   on.exit(DBI::dbDisconnect(con))   DBI::dbGetQuery(con, sql) } .onLoad <- function(...) {   orderly2::orderly_plugin_register(     name = \"example.db\",     config = db_config) } db_config <- function(data, filename) {   data }  query <- function(sql) {   ctx <- orderly2::orderly_plugin_context(\"example.db\")   dbname <- ctx$config$path   con <- DBI::dbConnect(RSQLite::SQLite(), dbname)   on.exit(DBI::dbDisconnect(con))   DBI::dbGetQuery(con, sql) }  .onLoad <- function(...) {   orderly2::orderly_plugin_register(     name = \"example.db\",     config = db_config) } export(query)"},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"trying-it-out","dir":"Articles","previous_headings":"An example","what":"Trying it out","title":"Creating plugins","text":"order test package, needs loaded. can either installing package using pkgload::load_all() (may find pkgload::load_all(export_all = FALSE) gives reliable experience. Now, can run report:","code":"pkgload::load_all() ## ℹ Loading example.db orderly2::orderly_run(\"example\", root = path_root) ## ℹ Starting packet 'example' `20240903-120347-89034989` at 2024-09-03 12:03:47.539182 ## > dat <- example.db::query(\"SELECT * FROM mtcars WHERE cyl == 4\") ## > orderly2::orderly_artefact(\"Summary of data\", \"data.rds\") ## Warning: Please use a named argument for the description in 'orderly_artefact()' ## In future versions of orderly, we will change the order of the arguments to ## 'orderly_artefact()' so that 'files' comes first. If you name your calls to ## 'description' then you will be compatible when we make this change. ## > saveRDS(summary(dat), \"data.rds\") ## ✔ Finished running example.R ## ! 1 warning found: ## • Please use a named argument for the description in 'orderly_artefact()' In ## future versions of orderly, we will change the order of the arguments to ## 'orderly_artefact()' so that 'files' comes first. If you name your calls to ## 'description' then you will be compatible when we make this change. ## ℹ Finished 20240903-120347-89034989 at 2024-09-03 12:03:47.741805 (0.2026234 secs) ## [1] \"20240903-120347-89034989\""},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"making-the-plugin-more-robust","dir":"Articles","previous_headings":"","what":"Making the plugin more robust","title":"Creating plugins","text":"plugin fairly fragile validation input data orderly_config.yml orderly.yml. fairly annoying yaml incredibly flexible reporting back information user might gone wrong hard. case, expect single key-value pair orderly_config.yml key path value path SQLite database. can easily expand configuration function report better back user misconfigure plugin: acceptable job preventing poor input suggesting user might look within configuration fix . Note return configuration data , can augment (otherwise change) data need.","code":"db_config <- function(data, filename) {   if (!is.list(data) || is.null(names(data)) || length(data) == 0) {     stop(\"Expected a named list for orderly_config.yml:example.db\")   }   if (length(data$path) != 1 || !is.character(data$path)) {     stop(\"Expected a string for orderly_config.yml:example.db:path\")   }   if (!file.exists(data$path)) {     stop(sprintf(       \"The database '%s' does not exist (orderly_config:example.db:path)\",       data$path))   }   data }"},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"saving-metadata-about-what-the-plugin-did","dir":"Articles","previous_headings":"","what":"Saving metadata about what the plugin did","title":"Creating plugins","text":"Nothing plugin saved report metadata unless save . Partly orderly.yml, saved final directory, serves sort record. However, probably want know something data returned . example might want save query string later can query without read process orderly.yml file statistics size data (e.g., number rows returned, columns) perhaps summary content hash can see content changed different versions report save metadata, use function orderly2::orderly_plugin_add_metadata; takes arguments plugin name, string like structure saved metadata (’ll use query) whatever data want save: function otherwise minimal version . also need provide serialisation function ensure metadata saved expected. saved metadata key query, get list back element query unnamed list many elements query calls given report. , ensure everything except cols length 1 (everything) gets turned scalar (1 [1]) serialise jsonlite::toJSON auto_unbox TRUE. Taking step , can also specify schema metadata conform save file inst/schema.json within package (path within inst fine). Finally, can also add deserialiation hook convert loaded metadata nice data.frame: Now, register plugin, provide path schema, along serialisation deserialisation functions: Now, orderly metadata saved (just running script part report) validate output passed orderly2::orderly_plugin_add_metadata schema, jsonvalidate installed (currently requires development version) R option outpack.schema_validate set TRUE (e.g., running options(outpack.schema_validate = TRUE)). final package structure: DESCRIPTION file NAMESPACE unchanged , schema shown just . plugin.R file contains code collected : (code .R file package, across several).","code":"query <- function(sql) {   ctx <- orderly2::orderly_plugin_context(\"example.db\")   dbname <- ctx$config$path   con <- DBI::dbConnect(RSQLite::SQLite(), dbname)   on.exit(DBI::dbDisconnect(con))   d <- DBI::dbGetQuery(con, sql)   info <- list(sql = sql, rows = nrow(d), cols = names(d))   orderly2::orderly_plugin_add_metadata(\"example.db\", \"query\", info)   d } db_serialise <- function(data) {   for (i in seq_along(data$query)) {     # Always save cols as a vector, even if length 1:     data$query[[i]]$cols <- I(data$query[[i]]$cols)   }   jsonlite::toJSON(data$query, auto_unbox = TRUE) } {     \"$schema\": \"http://json-schema.org/draft-07/schema#\",      \"type\": \"array\",     \"items\": {         \"type\": \"object\",         \"properties\": {             \"sql\": {                 \"type\": \"string\"             },             \"rows\": {                 \"type\": \"number\"             },             \"cols\": {                 \"type\": \"array\",                 \"items\": {                     \"type\": \"string\"                 }             }         },         \"required\": [\"sql\", \"rows\", \"cols\"],         \"additionalProperties\": false     } } .onLoad <- function(...) {   orderly2::orderly_plugin_register(     name = \"example.db\",     config = db_config,     serialise = db_serialise,     deserialise = db_deserialise,     schema = \"schema.json\") } ## . ## ├── archive ## │   └── example ## │       └── 20240903-120347-89034989 ## │           ├── data.rds ## │           └── example.R ## ├── draft ## │   └── example ## ├── orderly_config.yml ## └── src ##     └── example ##         └── example.R db_config <- function(data, filename) {   if (!is.list(data) || is.null(names(data)) || length(data) == 0) {     stop(\"Expected a named list for orderly_config.yml:example.db\")   }   if (length(data$path) != 1 || !is.character(data$path)) {     stop(\"Expected a string for orderly_config.yml:example.db:path\")   }   if (!file.exists(data$path)) {     stop(sprintf(       \"The database '%s' does not exist (orderly_config:example.db:path)\",       data$path))   }   data }  query <- function(sql) {   ctx <- orderly2::orderly_plugin_context(\"example.db\")   dbname <- ctx$config$path   con <- DBI::dbConnect(RSQLite::SQLite(), dbname)   on.exit(DBI::dbDisconnect(con))   d <- DBI::dbGetQuery(con, sql)   info <- list(sql = sql, rows = nrow(d), cols = names(d))   orderly2::orderly_plugin_add_metadata(\"example.db\", \"query\", info)   d }  .onLoad <- function(...) {   orderly2::orderly_plugin_register(     name = \"example.db\",     config = db_config,     serialise = db_serialise,     deserialise = db_deserialise,     schema = \"schema.json\") } id <- orderly2::orderly_run(\"example\", root = path_root) ## ℹ Starting packet 'example' `20240903-120348-9f1ba3e6` at 2024-09-03 12:03:48.624246 ## > dat <- example.db::query(\"SELECT * FROM mtcars WHERE cyl == 4\") ## > orderly2::orderly_artefact(\"Summary of data\", \"data.rds\") ## Warning: Please use a named argument for the description in 'orderly_artefact()' ## In future versions of orderly, we will change the order of the arguments to ## 'orderly_artefact()' so that 'files' comes first. If you name your calls to ## 'description' then you will be compatible when we make this change. ## > saveRDS(summary(dat), \"data.rds\") ## ✔ Finished running example.R ## ! 1 warning found: ## • Please use a named argument for the description in 'orderly_artefact()' In ## future versions of orderly, we will change the order of the arguments to ## 'orderly_artefact()' so that 'files' comes first. If you name your calls to ## 'description' then you will be compatible when we make this change. ## ℹ Finished 20240903-120348-9f1ba3e6 at 2024-09-03 12:03:48.68091 (0.05666375 secs) meta <- orderly2::orderly_metadata(id, root = path_root) meta$custom$example.db ##                                   sql rows         cols ## 1 SELECT * FROM mtcars WHERE cyl == 4   11 mpg, cyl...."},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"potential-uses","dir":"Articles","previous_headings":"","what":"Potential uses","title":"Creating plugins","text":"need functionality similar example - pulling database functionality original version orderly something independent, turns useful fraction orderly use-cases. can imagine potential uses though, : Non-DBI-based database data extraction, customised routines pulling data database Download files shared location just use (e.g., SharePoint, OneDrive, AWS). orderly_config.yml contain account connection details orderly.yml contain mapping remote data/files local files. Rather writing environment , use path argument copy files correct place. Pull data web API just running follow basic pattern requiring configuration order able connect resource service, specification resources fetched, action actually fetch resource put place.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"structure-of-queries","dir":"Articles","previous_headings":"","what":"Structure of queries","title":"Orderly query DSL","text":"simple query finds recent packet; unlikely useful without scoping - see . complex queries expressed syntax valid R (also valid Julia close valid Python). complex query composed “tests” Every “test” uses boolean operator (<, >, <=, >=, ==, !=) left right hand side can one : lookup outpack metadata (parameter:x value parameter called x, name name packet, id id packet) lookup provided data pars (:x value pars$x) literal value (e.g., \"some_name\", 1, TRUE) Tests can grouped together (, !, &&, || might expect: parameter:x == 1 || parameter:x == 2 finds packets parameter x 1 2 name == \"data\" && parameter:x > 3 finds packets called “data” parameter x greater 3 (parameter:y == 2) && !(parameter:x == 1 || parameter:x == 2) finds parameter y 2 parameter x anything 1 2 (also written (parameter:y == 2) && (parameter:x != 1 && parameter:x != 2)) four functions latest(expr) finds latest packet satisfying expr - always returns length 1 character, NA_character_ suitable packet found. expr given latest packets returned. single(expr) like latest(expr) except error expr evaluate exactly one packet id usedby(expr, FALSE) expr either literal id expression returns 1 id. finds packets used generation packet id returned expr (see dependencies section details).","code":"latest() name == \"some_name\" parameter:x > 1"},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"special-simple-queries","dir":"Articles","previous_headings":"","what":"Special simple queries","title":"Orderly query DSL","text":"two shorthand queries: latest equivalent latest() (useful applied scope) string matching regular expression id (^([0-9]{8}-[0-9]{6}-[[:xdigit:]]{8})$) equivalent single(id == \"<id>\") \"<id>\" string provided)","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"scoping-queries","dir":"Articles","previous_headings":"","what":"Scoping queries","title":"Orderly query DSL","text":"WARNING: may remove Scoping queries can used shorthand filtering returned packets. future used reduce set packets searched speed query evaluation. join together main query (scope) && (expr), except expr call latest single. case combine latest((scope) && (expr)) single((scope) && (expr)). useful want limit search particular name location perform detailed search. example, query equivalent orderly2 uses functionality resolving dependencies orderly2::orderly_dependency.","code":"orderly_query(quote(parameter:x == 1), scope = quote(name == \"data\")) orderly_query(quote(parameter:x == 1 && name == \"data\"))"},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"scoping-on-name","dir":"Articles","previous_headings":"Scoping queries","what":"Scoping on name","title":"Orderly query DSL","text":"often users want scope name instead passing scope argument shorthand name argument usability. equivalent ","code":"orderly_query(quote(parameter:x == 1), name = \"data\") orderly_query(quote(parameter:x == 1), scope = quote(name == \"data\"))"},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"dependencies","dir":"Articles","previous_headings":"","what":"Dependencies","title":"Orderly query DSL","text":"2 packets, B depends output (.e. call (id_a, ...) running packet B) can draw .  equivalently say dependency B, B depends used B , B uses parent B, B child tree dependencies among packets might want search packets used another packet. can use query function usedby(id) list packets used id. search recursively packets used id parents parents’ parents . optional second arg immediate FALSE default, set TRUE search immediate (e.g. level 1) dependencies. able search dependencies like means packet structure like  want know id used C can find using orderly_search usedby can combined groupings scope: depth usedby recurse can controlled setting depth e.g. search just immediate parents C. depth can positive integer, default depth recurse finds parents. usedby can simplified using subqueries. Subqueries denoted curly braces {} can either named passed subquery arg can anonymous. query equivalent uses subquery C. two important things note usedby: query inside usedby search entire index, ignoring scope name parameters. want find packets used latest C. subquery C scoped return results. query inside usedby must return single result. ensure must either literal id, call latest call single well searching dependency tree using usedby can search uses function. setup reports , B C want know id C uses can find using uses usedby can combined search complex arrangements dependencies. something like  want search version E depends version used latest C can via searches tree C tree find version E. Note important added name == \"\" condition , missing usedby(latest(name == \"C\")) also return B single throw error multiple packets. can also search tree find D e.g. note E packet uses D need add name == \"E\" clause. can combine usedby uses complex searches, find D C run","code":"orderly_search(quote(usedby(latest(name == \"C\"))), name = \"A\") orderly_search(quote(usedby(latest(name == \"C\")) && parameter:year == 2022),                name = \"A\") orderly_search(quote(usedby(latest(name == \"C\"), depth = 1)), name = \"A\") orderly_search(quote(usedby({C}) && parameter:year == 2022),                name = \"A\",                subquery = list(C = quote(latest(name == \"C\")))) orderly_search(quote(uses(latest(name == \"A\"))),                name = \"C\") orderly_search(   quote(latest(uses(single(usedby(latest(name == \"C\")) && name == \"A\")))),   name = \"E\") orderly_query(   quote(usedby(single(uses(name == \"D\")))),   name = \"A\") orderly_query(   quote(usedby(single(uses(single(usedby(latest(name == \"C\")) && name == \"A\"))) && name == \"E\"))),   name = \"D\")"},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"simple-things","dir":"Articles","previous_headings":"Possible future queries and interface improvements","what":"Simple things","title":"Orderly query DSL","text":"orderly1 supports .null(parameter:x) might generalise support However, Python None Julia nothing, complicates things. Alternatively use missing(parameter:x)?","code":"parameter:x == NULL"},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"explain-the-query","dir":"Articles","previous_headings":"Possible future queries and interface improvements","what":"Explain the query","title":"Orderly query DSL","text":"Often, people want know “packet match”? good show query set packets fail query excluded. definitely bit work.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/troubleshooting.html","id":"outpack-files-accidentally-committed-to-git","dir":"Articles","previous_headings":"","what":"Outpack files accidentally committed to git","title":"Troubleshooting","text":"discussed orderly introduction, want commit files .outpack/, drafts/ archive/ (used) git create sorts problems line. directed , probably ended files git want undo situation. least painful way depends situation. now put guard rails try prevent happening, still happen modify .gitignore file force-add files example. situation, orderly2 shout : may directed page. just want continue working anyway, run suggested command: things work warning first time session: subsequent calls display warning: rest section discusses might permanently fix issue.","code":"orderly2::orderly_run(\"data\") ## Error in `orderly2::orderly_run()`: ## ! Detected 6 outpack files committed to git ## ✖ Detected files were found in '.outpack/' and 'archive/' ## ℹ For tips on resolving this, please see ##   <https://mrc-ide.github.io/orderly2/articles/troubleshooting.html> ## ℹ To turn this into a warning and continue anyway set the option ##   'orderly_git_error_is_warning' to TRUE by running ##   options(orderly_git_error_is_warning = TRUE) options(orderly_git_error_is_warning = TRUE) orderly2::orderly_run(\"data\") ## Warning in orderly2::orderly_run(\"data\"): Detected 6 outpack files committed to git ## ✖ Detected files were found in '.outpack/' and 'archive/' ## ℹ For tips on resolving this, please see ##   <https://mrc-ide.github.io/orderly2/articles/troubleshooting.html> ## This warning is displayed once per session. ## ✔ Wrote '.gitignore' ## ℹ Starting packet 'data' `20240903-120355-42792af8` at 2024-09-03 12:03:55.262153 ## > orderly2::orderly_artefact(\"data.rds\", description = \"Final data\") ## > saveRDS(mtcars, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240903-120355-42792af8 at 2024-09-03 12:03:55.299319 (0.03716588 secs) ## [1] \"20240903-120355-42792af8\" orderly2::orderly_run(\"data\") ## ℹ Starting packet 'data' `20240903-120355-62d2689e` at 2024-09-03 12:03:55.388593 ## > orderly2::orderly_artefact(\"data.rds\", description = \"Final data\") ## > saveRDS(mtcars, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240903-120355-62d2689e at 2024-09-03 12:03:55.411622 (0.02302885 secs) ## [1] \"20240903-120355-62d2689e\""},{"path":"https://mrc-ide.github.io/orderly2/articles/troubleshooting.html","id":"i-dont-care-about-my-history-at-all","dir":"Articles","previous_headings":"Outpack files accidentally committed to git","what":"I don’t care about my history at all","title":"Troubleshooting","text":"case just started project, yet collaborating anyone else (person willing re-clone sources). simplest thing : Delete .git directory entirely Run orderly2::orderly_gitignore_update(\"(root)\") set reasonable .gitignore prevent situation happening Run git add . add everything back (review git status make sure ’re happy) Run git commit -m \"Initial commit\" create single commit contains files currently repo history, also .outpack files previously pushed repo GitHub similar need set git remote add origin https://github.com/user/repo (replacing user/repo path, using git@github.com:user/repo use ssh talk GitHub) git branch -M main assuming using main default branch, now common git push --force -u origin main Note destructive require coordination collaborators changed history.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/troubleshooting.html","id":"i-just-want-this-to-go-away-and-nothing-i-have-committed-is-very-large","dir":"Articles","previous_headings":"Outpack files accidentally committed to git","what":"I just want this to go away and nothing I have committed is very large","title":"Troubleshooting","text":"care history, also committed files (e.g., committed files .outpack/ small large 100MB file archive/ preventing pushing GitHub) just delete offending files git without updating history, affecting local copies. Run git rm --cached .outpack (repeating draft archive needed) Run orderly2::orderly_gitignore_update(\"(root)\") set reasonable .gitignore prevent situation happening Run git add .gitignore also stage Run git commit -m \"Delete unwanted outpack files\" can push without issues.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/troubleshooting.html","id":"i-care-about-my-history-but-want-this-stuff-gone","dir":"Articles","previous_headings":"Outpack files accidentally committed to git","what":"I care about my history but want this stuff gone","title":"Troubleshooting","text":"working branch, unwanted files committed branch, simplest thing copy changes made somewhere safe, create new branch current main copy changes . somewhat automatically generating applying patch: unwanted files committed onto default branch, potentially gory history rewriting. See StackOverflow question, git docs currently recommended tool . Good luck!","code":"git diff -- src > changes.patch git checkout main git checkout -b changes-attempt2 git apply changes.patch git push -u origin changes-attempt2"},{"path":"https://mrc-ide.github.io/orderly2/authors.html","id":null,"dir":"","previous_headings":"","what":"Authors","title":"Authors and Citation","text":"Rich FitzJohn. Author, maintainer. Robert Ashton. Author. Alex Hill. Author. Imperial College Science, Technology Medicine. Copyright holder.","code":""},{"path":"https://mrc-ide.github.io/orderly2/authors.html","id":"citation","dir":"","previous_headings":"","what":"Citation","title":"Authors and Citation","text":"FitzJohn R, Ashton R, Hill (2024). orderly2: Orderly Next Generation. R package version 1.99.32, https://github.com/mrc-ide/orderly2.","code":"@Manual{,   title = {orderly2: Orderly Next Generation},   author = {Rich FitzJohn and Robert Ashton and Alex Hill},   year = {2024},   note = {R package version 1.99.32},   url = {https://github.com/mrc-ide/orderly2}, }"},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"orderly2","dir":"","previous_headings":"","what":"Orderly Next Generation","title":"Orderly Next Generation","text":"attendant hospital responsible non-medical care patients maintenance order cleanliness. soldier carries orders performs minor tasks officer. orderly2 package designed help make analysis reproducible. principal aim automate series basic steps process writing analyses, making easy : track inputs analysis (packages, code, data resources) store multiple versions analysis repeated track outputs analysis create analyses depend outputs previous analyses","code":""},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"get-started","dir":"","previous_headings":"","what":"Get started","title":"Orderly Next Generation","text":"Two vignettes provide overview package, depending tastes: vignette(\"orderly2\") describes problem orderly2 tries solve (read package website) vignette(\"introduction\") describes orderly2 example, walking basic features (read package website)","code":""},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"roadmap","dir":"","previous_headings":"","what":"Roadmap","title":"Orderly Next Generation","text":"package ground-rewrite orderly package, now ready use new projects adventurous users original package. next months orderly renamed orderly1, finalise API package. stable, rename back orderly2 orderly release CRAN replace existing package. migration path existing users described vignette(\"migrating\") updated based experience migrating people. web framework OrderlyWeb rewritten parallel (packit) provide migration path .","code":""},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"installation","dir":"","previous_headings":"","what":"Installation","title":"Orderly Next Generation","text":"Please install r-universe: prefer, can install GitHub remotes:","code":"install.packages(   \"orderly2\",   repos = c(\"https://mrc-ide.r-universe.dev\", \"https://cloud.r-project.org\")) remotes::install_github(\"mrc-ide/orderly2\")"},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"testing","dir":"","previous_headings":"","what":"Testing","title":"Orderly Next Generation","text":"run tests, need outpack_server available system path. One way achieve run tests also require development versions gert jsonvalidate. can installed rOpenSci r-universe, : Without , tests skipped, suite still run.","code":"cargo install --git https://github.com/mrc-ide/outpack_server install.packages(   c(\"gert\", \"jsonvalidate\"),   repos = c(\"https://ropensci.r-universe.dev\", \"https://cloud.r-project.org\"))"},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"licence","dir":"","previous_headings":"","what":"Licence","title":"Orderly Next Generation","text":"MIT © Imperial College Science, Technology Medicine","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_artefact.html","id":null,"dir":"Reference","previous_headings":"","what":"Declare orderly artefacts — orderly_artefact","title":"Declare orderly artefacts — orderly_artefact","text":"Declare artefact. turn number orderly features; see Details . can multiple calls function within orderly script.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_artefact.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Declare orderly artefacts — orderly_artefact","text":"","code":"orderly_artefact(description = NULL, files)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_artefact.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Declare orderly artefacts — orderly_artefact","text":"description name artefact files files within artefact","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_artefact.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Declare orderly artefacts — orderly_artefact","text":"Undefined","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_artefact.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Declare orderly artefacts — orderly_artefact","text":"(1) files matching copied src directory draft directory unless also listed resource orderly_resource(). feature enabled call function top level orderly script contains string literals (variables). (2) script fails produce files, orderly_run() fail, guaranteeing task really produce things need . (3) within final metadata, artefacts additional metadata; description provide grouping","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":null,"dir":"Reference","previous_headings":"","what":"Clean up source directory — orderly_cleanup","title":"Clean up source directory — orderly_cleanup","text":"Find, delete, file generated running report. comfortable , strongly recommended run orderly_cleanup_status first see deleted.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Clean up source directory — orderly_cleanup","text":"","code":"orderly_cleanup(name = NULL, dry_run = FALSE, root = NULL, locate = TRUE)  orderly_cleanup_status(name = NULL, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Clean up source directory — orderly_cleanup","text":"name Name report directory clean (.e., look src/<name> relative orderly root dry_run Logical, indicating delete anything, instead just print information root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, just outpack (see orderly_init details). locate Logical, indicating configuration searched .  TRUE config given, orderly looks working directory parents finds .outpack directory","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Clean up source directory — orderly_cleanup","text":"(currently unstable) object class orderly_cleanup_status within element delete indicates files deleted (orderly_cleanup_status) deleted (orderly_cleanup)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Clean up source directory — orderly_cleanup","text":"file deletion, look remove empty directories; orderly2 similar semantics git directories never directly tracked. recent gert (yet CRAN) ask git files ignored; ignored good candidates deletion! encourage keep per-report .gitignore lists files copy source directory, can use information clean files generation. Importantly, even file matches ignore rule committed repository, longer match ignore rule.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"notes-for-user-of-orderly-","dir":"Reference","previous_headings":"","what":"Notes for user of orderly1","title":"Clean up source directory — orderly_cleanup","text":"orderly1 function quite different semantics, full set possible files always knowable yaml file. , start point view list files compare directory.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Clean up source directory — orderly_cleanup","text":"","code":"# Create a simple example: path <- orderly2::orderly_example(\"default\") #> ✔ Created orderly root at '/tmp/RtmpeoP8u3/file1722797ca70d'  # We simulate running a packet interactively by using 'source'; # you might have run this line-by-line, or with the \"Source\" # button in Rstudio. source(file.path(path, \"src/data/data.R\"), chdir = TRUE) #> Warning: cannot open file '/tmp/RtmpeoP8u3/file1722797ca70d/src/data/data.R': No such file or directory #> Error in file(filename, \"r\", encoding = encoding): cannot open the connection  # Having run this, the output of the report is present in the # source directory: fs::dir_tree(path) #> Error: [ENOENT] Failed to search directory '/tmp/RtmpeoP8u3/file1722797ca70d': no such file or directory  # We can detect what might want cleaning up by running # \"orderly_cleanup_status\": orderly2::orderly_cleanup_status(\"data\", root = path) #> Error: Directory does not exist: '/tmp/RtmpeoP8u3/file1722797ca70d'  # Soon this will print more nicely to the screen, but for now you # can see that the status of \"data.rds\" is \"derived\", which means # that orderly knows that it is subject to being cleaned up; the # \"delete\" element shows what will be deleted.  # Do the actual deletion: orderly2::orderly_cleanup(\"data\", root = path) #> Error: Directory does not exist: '/tmp/RtmpeoP8u3/file1722797ca70d'"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config.html","id":null,"dir":"Reference","previous_headings":"","what":"Read configuration — orderly_config","title":"Read configuration — orderly_config","text":"Read current orderly configuration, stored within outpack root, along orderly-specific extensions.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Read configuration — orderly_config","text":"","code":"orderly_config(root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Read configuration — orderly_config","text":"root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Read configuration — orderly_config","text":"list configuration options: core: important options outpack store, containing: path_archive: path human-readable packet archive, NULL disabled (set orderly_config_set core.path_archive) use_file_store: Indicates content-addressable file store enabled (core.use_file_store) require_complete_tree: Indicates outpack store requires dependencies fully available (core.require_complete_tree) hash_algorithm: hash algorithm used (currently modifiable) location: Information locations; see orderly_location_add, orderly_location_rename orderly_location_remove interact configuration, orderly_location_list simply list available locations. Returns data.frame columns name, id, priority, type args, args list column. orderly: list orderly-specific configuration; just minimum required version (minimum_orderly_version).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Read configuration — orderly_config","text":"","code":"# A default configuration in a new temporary directory path <- withr::local_tempdir() orderly2::orderly_init(path) #> ✔ Created orderly root at '/tmp/RtmpeoP8u3/file172241b2bd8d' orderly2::orderly_config(path) #> $core #> $core$path_archive #> [1] \"archive\" #>  #> $core$use_file_store #> [1] FALSE #>  #> $core$require_complete_tree #> [1] FALSE #>  #> $core$hash_algorithm #> [1] \"sha256\" #>  #>  #> $location #>    name  type args #> 1 local local      #>  #> $orderly #> $orderly$minimum_orderly_version #> [1] ‘1.99.0’ #>  #>"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":null,"dir":"Reference","previous_headings":"","what":"Set configuration options — orderly_config_set","title":"Set configuration options — orderly_config_set","text":"Set configuration options. can currently set; expanded time. See Details.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Set configuration options — orderly_config_set","text":"","code":"orderly_config_set(..., options = list(...), root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Set configuration options — orderly_config_set","text":"... Named options set (e.g., pass argument core.require_complete_tree = TRUE) options alternative ..., can pass list named options (e.g., list(core.require_complete_tree = TRUE)).  interface typically easier program . root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Set configuration options — orderly_config_set","text":"Nothing","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Set configuration options — orderly_config_set","text":"Options set order provided.  Currently, setting one option fails, options processed previous ones (rely behaviour, may change). Currently can set: core.require_complete_tree See orderly_init description options.","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Set configuration options — orderly_config_set","text":"","code":"# The default configuration does not include a file store, and # saves output within the \"archive\" directory: path <- withr::local_tempdir() orderly2::orderly_init(path) #> ✔ Created orderly root at '/tmp/RtmpeoP8u3/file172222a3b2b0' fs::dir_tree(path, all = TRUE) #> /tmp/RtmpeoP8u3/file172222a3b2b0 #> ├── .outpack #> │   ├── config.json #> │   ├── location #> │   └── metadata #> └── orderly_config.yml  # Change this after the fact: orderly2::orderly_config_set(core.use_file_store = TRUE,                              core.path_archive = NULL,                              root = path) fs::dir_tree(path, all = TRUE) #> /tmp/RtmpeoP8u3/file172222a3b2b0 #> ├── .outpack #> │   ├── config.json #> │   ├── files #> │   ├── index #> │   │   └── outpack.rds #> │   ├── location #> │   └── metadata #> └── orderly_config.yml"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_copy_files.html","id":null,"dir":"Reference","previous_headings":"","what":"Copy files from a packet — orderly_copy_files","title":"Copy files from a packet — orderly_copy_files","text":"Copy files packet anywhere. Similar orderly_dependency except used active packet context. can use function pull files outpack root directory outside control outpack, example. Note arguments need must provided name, position, exception id query.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_copy_files.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Copy files from a packet — orderly_copy_files","text":"","code":"orderly_copy_files(   ...,   files,   dest,   overwrite = TRUE,   envir = parent.frame(),   options = NULL,   root = NULL,   locate = TRUE )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_copy_files.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Copy files from a packet — orderly_copy_files","text":"... Arguments passed orderly_query, perhaps just query expression files Files copy packet. can (1) character vector, case files copied without changing names, (2) named character vector, case name used destination name, (3) data.frame (including tbl_df, data.frame objects) containing columns , case files copied names . cases, want import directory files packet, must refer source trailing slash (e.g., c(= \"/\")), create local directory /... files upstream packet directory /. omit slash error thrown suggesting add slash intended. can use limited form string interpolation names argument; using ${variable} pick values envir substitute string.  similar interpolation might familiar glue::glue similar, much simpler concatenation fancy features supported. Note unfortunate, (us) avoidable inconsistency ; interpolation values environment query done using environment:x destination filename ${x}. dest directory copy overwrite Overwrite files destination; typically want, set FALSE prefer error thrown destination file already exists. envir Optionally, environment substitute query (using environment: prefix). default use calling environment, can explicitly pass want control lookup happens. options Optionally, orderly_search_options object controlling search performed, packets considered scope. provided, default options used (.e., orderly2::orderly_search_options()) root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_copy_files.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Copy files from a packet — orderly_copy_files","text":"Nothing, invisibly. Primarily called side effect copying files packet directory dest","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_copy_files.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Copy files from a packet — orderly_copy_files","text":"can call function id string, case search packet proceed regardless whether id present.  called arguments (e.g., string match id format, named argument name, subquery parameters) interpret arguments query orderly_search find id. error query return exactly one packet id, probably want use latest(). different ways might fail (recover failure): id known metadata store (known unpacked also known present remote) fail impossible resolve files. Consider refreshing metadata orderly_location_pull_metadata refresh . id unpacked local copy files referred can found, error default (see next option). However, sometimes file refer might also present downloaded packet depended , content file unchanged packet version locally. id unpacked, local copy file allow_remote TRUE try request file whatever remote selected orderly_location_pull_packet packet. Note empty directories might created failure.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_dependency.html","id":null,"dir":"Reference","previous_headings":"","what":"Declare a dependency — orderly_dependency","title":"Declare a dependency — orderly_dependency","text":"Declare dependency another packet","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_dependency.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Declare a dependency — orderly_dependency","text":"","code":"orderly_dependency(name, query, files)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_dependency.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Declare a dependency — orderly_dependency","text":"name name packet depend query query search ; often simply string latest, indicating recent version. may want complex query though. files Files copy packet. can (1) character vector, case files copied without changing names, (2) named character vector, case name used destination name, (3) data.frame (including tbl_df, data.frame objects) containing columns , case files copied names . cases, want import directory files packet, must refer source trailing slash (e.g., c(= \"/\")), create local directory /... files upstream packet directory /. omit slash error thrown suggesting add slash intended. can use limited form string interpolation names argument; using ${variable} pick values envir substitute string.  similar interpolation might familiar glue::glue similar, much simpler concatenation fancy features supported. Note unfortunate, (us) avoidable inconsistency ; interpolation values environment query done using environment:x destination filename ${x}.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_dependency.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Declare a dependency — orderly_dependency","text":"Undefined","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_dependency.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Declare a dependency — orderly_dependency","text":"See orderly_run details search options used select locations packets found , data fetched network. running interactively, obviously work, use orderly_interactive_set_search_options() set options function respond .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_description.html","id":null,"dir":"Reference","previous_headings":"","what":"Describe the current packet — orderly_description","title":"Describe the current packet — orderly_description","text":"Describe current packet","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_description.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Describe the current packet — orderly_description","text":"","code":"orderly_description(display = NULL, long = NULL, custom = NULL)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_description.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Describe the current packet — orderly_description","text":"display friendly name report; displayed locations web interface, packit. given, must scalar character. long longer description report. given, must scalar character. custom additional metadata. given, must named list, elements scalar atomics (character, number, logical).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_description.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Describe the current packet — orderly_description","text":"Undefined","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_example.html","id":null,"dir":"Reference","previous_headings":"","what":"Copy a simple orderly example — orderly_example","title":"Copy a simple orderly example — orderly_example","text":"Copy simple orderly example use docs. function form part workflow!","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_example.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Copy a simple orderly example — orderly_example","text":"","code":"orderly_example(name, ..., dest = NULL)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_example.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Copy a simple orderly example — orderly_example","text":"name name example copy. Currently \"default\" supported. ... Arguments passed orderly_init() dest destination. default use withr::local_tempfile() create temporary directory clean . suitable use orderly examples, may prefer provide path. path must already exist.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_example.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Copy a simple orderly example — orderly_example","text":"Invisibly, path example.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_example.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Copy a simple orderly example — orderly_example","text":"","code":"path <- orderly2::orderly_example(\"default\") #> ✔ Created orderly root at '/tmp/RtmpeoP8u3/file1722606e9192' orderly2::orderly_list_src(root = path) #> Error: Directory does not exist: '/tmp/RtmpeoP8u3/file1722606e9192'"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_gitignore_update.html","id":null,"dir":"Reference","previous_headings":"","what":"Update a gitignore file — orderly_gitignore_update","title":"Update a gitignore file — orderly_gitignore_update","text":"Update gitignore, useful prevent accidentally committing files source control generated. includes artefacts, shared resources dependencies (within report directory) global level contents .outpack directory, draft folder archive directory.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_gitignore_update.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Update a gitignore file — orderly_gitignore_update","text":"","code":"orderly_gitignore_update(name, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_gitignore_update.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Update a gitignore file — orderly_gitignore_update","text":"name name gitignore file update, string \"(root)\" root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, just outpack (see orderly_init details). locate Logical, indicating configuration searched .  TRUE config given, orderly looks working directory parents finds .outpack directory","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_gitignore_update.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Update a gitignore file — orderly_gitignore_update","text":"Nothing, called side effects","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_gitignore_update.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Update a gitignore file — orderly_gitignore_update","text":"function fails message edit '.gitignore', markers corrupted, look special markers within .gitignore file.  look like   edit file : lines appears file anything first two lines order get error message, search remove lines rerun.","code":"# ---VVV--- added by orderly ---VVV---------------- # Don't manually edit content between these markers ... patterns # ---^^^--- added by orderly ---^^^----------------"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_hash.html","id":null,"dir":"Reference","previous_headings":"","what":"Compute a hash — orderly_hash_file","title":"Compute a hash — orderly_hash_file","text":"Use orderly2's hashing functions.  intended advanced users, particular want create hashes consistent orderly2 within plugins. default behaviour use algorithm used orderly root (via root argument, usual root location approach). However, string provided algorithm can use alternative algorithm.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_hash.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Compute a hash — orderly_hash_file","text":"","code":"orderly_hash_file(path, algorithm = NULL, root = NULL, locate = TRUE)  orderly_hash_data(data, algorithm = NULL, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_hash.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Compute a hash — orderly_hash_file","text":"path name file hash algorithm name algorithm use, overriding orderly root. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml data string hash","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_hash.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Compute a hash — orderly_hash_file","text":"string format <algorithm>:<digest>","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_hash.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Compute a hash — orderly_hash_file","text":"","code":"orderly2::orderly_hash_data(\"hello\", \"md5\") #> [1] \"md5:5d41402abc4b2a76b9719d911017c592\""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":null,"dir":"Reference","previous_headings":"","what":"Initialise an orderly repository — orderly_init","title":"Initialise an orderly repository — orderly_init","text":"Initialise empty orderly repository, initialise source copy orderly repository (see Details). orderly repository defined presence file orderly_config.yml root, along directory .outpack/ level.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Initialise an orderly repository — orderly_init","text":"","code":"orderly_init(   root = \".\",   path_archive = \"archive\",   use_file_store = FALSE,   require_complete_tree = FALSE,   force = FALSE )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Initialise an orderly repository — orderly_init","text":"root path initialise repository root .  repository already initialised, operation checks options passed set repository (erroring ), otherwise nothing.  default path current working directory. path_archive Path archive directory, used store human-readable copies packets.  NULL, copy made, file_store must TRUE use_file_store Logical, indicating use content-addressable file-store source truth packets.  archive non-NULL, file-store used source truth duplicated files archive exist convenience. require_complete_tree Logical, indicating require complete tree packets.  currently affects orderly_location_pull_packet, requiring always operates recursive mode.  FALSE default, set TRUE want archive behave well location; TRUE always packets hold metadata . force Logical, indicating shold initialise orderly even directory empty.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Initialise an orderly repository — orderly_init","text":"full, normalised, path root, invisibly. Typically called side effect.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Initialise an orderly repository — orderly_init","text":"expected orderly_config.yml saved version control, .outpack excluded version control; means every clone project need call orderly2::orderly_init() initialise .outpack directory. forget , error thrown reminding need . can safely call orderly2::orderly_init() already-initialised directory, however, arguments passed must exactly match configuration current root, otherwise error thrown. Please use orderly_config_set change configuration, ensures change configuration possible. configuration options given match directory already uses, nothing happens. repository call orderly2::orderly_init() already initialised .outpack directory orderly_config.yml file, write file .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Initialise an orderly repository — orderly_init","text":"","code":"# We'll use an automatically cleaned-up directory for the root: path <- withr::local_tempdir()  # Initialise a new repository, setting an option: orderly2::orderly_init(path, use_file_store = TRUE) #> ✔ Created orderly root at '/tmp/RtmpeoP8u3/file17222cb256f'  # fs::dir_tree(path, all = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_interactive_set_search_options.html","id":null,"dir":"Reference","previous_headings":"","what":"Set search options for interactive use — orderly_interactive_set_search_options","title":"Set search options for interactive use — orderly_interactive_set_search_options","text":"Set search options interactive use orderly; see orderly_dependency orderly_run details. may either orderly_search_options object, list coerced one point use (NULL). applies current session, applies interactive uses orderly functions might received copy search_options via orderly_run","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_interactive_set_search_options.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Set search options for interactive use — orderly_interactive_set_search_options","text":"","code":"orderly_interactive_set_search_options(options = NULL)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_interactive_set_search_options.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Set search options for interactive use — orderly_interactive_set_search_options","text":"options Optional control locations, used orderly_dependency; see Details section orderly_run.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_interactive_set_search_options.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Set search options for interactive use — orderly_interactive_set_search_options","text":"Nothing, called side effects","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_list_src.html","id":null,"dir":"Reference","previous_headings":"","what":"List source reports — orderly_list_src","title":"List source reports — orderly_list_src","text":"List source reports - , directories within src/ look suitable running orderly; directories contain entrypoint file - .R file name directory (e.g., src/data/data.R corresponds data).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_list_src.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"List source reports — orderly_list_src","text":"","code":"orderly_list_src(root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_list_src.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"List source reports — orderly_list_src","text":"root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, just outpack (see orderly_init details). locate Logical, indicating configuration searched .  TRUE config given, orderly looks working directory parents finds .outpack directory","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_list_src.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"List source reports — orderly_list_src","text":"character vector names source reports, suitable passing orderly_run","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_list_src.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"List source reports — orderly_list_src","text":"","code":"path <- orderly2::orderly_example(\"default\") #> ✔ Created orderly root at '/tmp/RtmpeoP8u3/file17221d5da8de' orderly2::orderly_list_src(root = path) #> Error: Directory does not exist: '/tmp/RtmpeoP8u3/file17221d5da8de'"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":null,"dir":"Reference","previous_headings":"","what":"Add a new location — orderly_location_add","title":"Add a new location — orderly_location_add","text":"Add new location - place packets might found pulled local archive.  Currently file http based locations supported, limited support custom locations. Note adding location pull metadata , need call orderly_location_pull_metadata first.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Add a new location — orderly_location_add","text":"","code":"orderly_location_add(name, type, args, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Add a new location — orderly_location_add","text":"name short name location use.  use, one local orphan type type location add. Currently supported values path (location exists elsewhere filesystem) http (location accessed outpack's http API). args Arguments location driver. arguments vary depending type used, see Details. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Add a new location — orderly_location_add","text":"Nothing","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Add a new location — orderly_location_add","text":"currently support two types locations - path, points outpack archive accessible path (e.g., computer mounted network share), http, requires outpack server running url uses HTTP API communicate, packit, uses Packit web server.  types may added later, configuration options location types definitely needed future. Configuration options different location types: Path locations: path: path archive root. generally absolute path, behaviour outpack unreliable. HTTP locations: Accessing outpack HTTP requires outpack server running. interface expected change expand API, also move support things like TLS authentication. url: location server, including protocol, example http://example.com:8080 Packit locations: Packit locations work HTTPS, include everything outpack location also provide authentication later capabilities think. url: location server token: value login token (currently GitHub token read:org scope).  Later expand authentication modes supported. Custom locations: outpack implementations expected support path http locations, standard arguments .  expect implementations support custom locations, argument lists may vary implementations. allow , can pass location type \"custom\" list arguments.  expect argument 'driver' present among list.  example action, see outpack.sharepoint package. warned may change interface future, case may need update configuration.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":"warning","dir":"Reference","previous_headings":"","what":"Warning","title":"Add a new location — orderly_location_add","text":"API may change move support different types locations.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_list.html","id":null,"dir":"Reference","previous_headings":"","what":"List known pack locations — orderly_location_list","title":"List known pack locations — orderly_location_list","text":"List known locations.  special name local always present within output function (packets known current root), though typically interested locations.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_list.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"List known pack locations — orderly_location_list","text":"","code":"orderly_location_list(verbose = FALSE, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_list.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"List known pack locations — orderly_location_list","text":"verbose Logical, indicating return data.frame includes information location. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_list.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"List known pack locations — orderly_location_list","text":"Depending value verbose: verbose = FALSE: character vector location names.  default behaviour. verbose = TRUE: data.frame columns name, type args.  args column list column, element key-value pair arguments location.","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_metadata.html","id":null,"dir":"Reference","previous_headings":"","what":"Pull metadata from a location — orderly_location_pull_metadata","title":"Pull metadata from a location — orderly_location_pull_metadata","text":"Pull metadata location, updating index.  always relatively quick updates small files contain information can found remote packets.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_metadata.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Pull metadata from a location — orderly_location_pull_metadata","text":"","code":"orderly_location_pull_metadata(location = NULL, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_metadata.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Pull metadata from a location — orderly_location_pull_metadata","text":"location name location pull (see orderly_location_list possible values).  given, pulls locations.  \"local\" \"orphan\" locations always date pulling metadata nothing. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_metadata.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Pull metadata from a location — orderly_location_pull_metadata","text":"Nothing","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_packet.html","id":null,"dir":"Reference","previous_headings":"","what":"Pull one or more packets from a location — orderly_location_pull_packet","title":"Pull one or more packets from a location — orderly_location_pull_packet","text":"Pull one packets (including files) archive one locations. make files available use dependencies (e.g., orderly_dependency).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_packet.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Pull one or more packets from a location — orderly_location_pull_packet","text":"","code":"orderly_location_pull_packet(   ...,   options = NULL,   recursive = NULL,   root = NULL,   locate = TRUE )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_packet.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Pull one or more packets from a location — orderly_location_pull_packet","text":"... Arguments passed orderly_search. special case first argument character vector ids named dot arguments, interpret argument vector ids directly. careful , query may pull lot data particular, passing NULL match everything every remote ! options Options passed orderly_search. option allow_remote must TRUE otherwise packet possibly pulled, error thrown FALSE. recursive non-NULL, logical, indicating recursively pull packets referenced packets specified id.  might copy lot data!  NULL, default value given configuration option require_complete_tree. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_packet.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Pull one or more packets from a location — orderly_location_pull_packet","text":"Invisibly, ids packets pulled","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_packet.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Pull one or more packets from a location — orderly_location_pull_packet","text":"possible take long time pull packets, moving lot data operating slow connection.  Cancelling resuming pull fairly efficient, keep track files copied even case interrupted pull.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_push.html","id":null,"dir":"Reference","previous_headings":"","what":"Push tree to location — orderly_location_push","title":"Push tree to location — orderly_location_push","text":"Push tree location. function works packets known location files required create . pushes files required build packets pushes missing metadata server. process interrupted safe resume transfer files packets missed previous call.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_push.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Push tree to location — orderly_location_push","text":"","code":"orderly_location_push(packet_id, location, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_push.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Push tree to location — orderly_location_push","text":"packet_id One packets push server location name location push (see orderly_location_list possible values). root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_push.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Push tree to location — orderly_location_push","text":"Invisibly, details information actually moved (might less requested, depending dependencies packets already known location).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_remove.html","id":null,"dir":"Reference","previous_headings":"","what":"Remove a location — orderly_location_remove","title":"Remove a location — orderly_location_remove","text":"Remove existing location. packets location known elsewhere now associated 'orphan' location instead.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_remove.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Remove a location — orderly_location_remove","text":"","code":"orderly_location_remove(name, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_remove.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Remove a location — orderly_location_remove","text":"name short name location. remove local orphan root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_remove.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Remove a location — orderly_location_remove","text":"Nothing","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_rename.html","id":null,"dir":"Reference","previous_headings":"","what":"Rename a location — orderly_location_rename","title":"Rename a location — orderly_location_rename","text":"Rename existing location","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_rename.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Rename a location — orderly_location_rename","text":"","code":"orderly_location_rename(old, new, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_rename.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Rename a location — orderly_location_rename","text":"old current short name location. rename local orphan new desired short name location. one local orphan root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_rename.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Rename a location — orderly_location_rename","text":"Nothing","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata.html","id":null,"dir":"Reference","previous_headings":"","what":"Read outpack metadata — orderly_metadata","title":"Read outpack metadata — orderly_metadata","text":"Read metadata particular id. may want use orderly_search find id corresponding particular query.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Read outpack metadata — orderly_metadata","text":"","code":"orderly_metadata(id, root = NULL, locate = FALSE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Read outpack metadata — orderly_metadata","text":"id id fetch metadata . error thrown id known root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Read outpack metadata — orderly_metadata","text":"list metadata. See outpack schema details (https://github.com/mrc-ide/outpack)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":null,"dir":"Reference","previous_headings":"","what":"Extract metadata from orderly2 packets — orderly_metadata_extract","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"Extract metadata group packets.  experimental high-level function interacting metadata way hope useful. expand bit time goes , based feedback get let us know think.  See Details use .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"","code":"orderly_metadata_extract(..., extract = NULL, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"... Arguments passed orderly_search. special case first argument character vector ids named dot arguments, interpret argument vector ids directly. extract character vector columns extract, possibly named. See Details format. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"data.frame, columns vary based names extract; see Details information.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"Extracting data outpack metadata challenging way works data structures familiar R users, naturally tree structured, metadata may present packets (e.g., packet depend another dependency section, one run context without git git metadata). just want raw tree-structured data, can always use orderly_metadata load full metadata packet (even one currently available computer, just known ) structure data remain fairly constant across orderly2 versions. However, sometimes want extract data order ask specific questions like: parameter combinations available across range packets? particular set packets used? files packets produce? Later like ask even complex questions like: version file graph.png change? inputs changed versions? ...able answer questions requires similar approach interrogating metadata across range packets. orderly_metadata_extract function aims simplify process pulling bits metadata arranging data.frame (sorts) .  little mini-language extract argument simple rewriting results, can always . order use function need know metadata available; expand vignette worked examples make easier understand. function works top-level keys, : id: packet id (always returned) name: packet name parameters: key-value pair values, string keys atomic values. guarantee presence keys packets, types. time: key-value pair times, string keys time values (see DateTimeClasses; stored seconds since 1970 actual metadata). present start end always present. files: files present packet. data.frame (per packet), columns path (relative), size (bytes) hash. depends: dependencies used packet. data.frame (per packet), columns packet (id), query (string, used find packet) files (another data.frame columns corresponding filenames upstream packet, respectively) git: either metadata state git null. given sha branch strings, url array strings/character vector (can zero, one elements). session: information session packet run (unstandardised, even orderly version may change) custom: additional metadata added respective engine.  packets run orderly2, orderly field , list: artefacts: data.frame artefact information, containing columns description (string) paths (list column paths). shared: data.frame copied shared resources original name () name copied packet (). role: data.frame identified roles files, columns path role. description: list information orderly_description human-readable descriptions tags. session: list information session run, list platform containing information platform (R version version, operating system os system name system) packages containing columns package , version attached. nesting makes providing universally useful data format difficult; considering files data.frame files column, list data.frames; similar nestedness applies depends orderly custom data. However, able fairly easily process data format need . simplest extraction uses names top-level keys:   creates data.frame columns corresponding keys, one row per packet. name always string, character vector, parameters files complex, list columns. must provide id; always returned always first character vector column. can rename columns providing name entries within extract, example:   , except parameters column renamed pars. interestingly, can index structure like parameters; suppose want value parameter x, write:   allowed packet parameters element list. However, know type x (might vary packets). can add information though write:   create numeric column. packet value x non-integer, call orderly_metadata_extract fail error, packet lacks value x, missing value appropriate type added. Note coercion number, error non-NULL non-numeric value found.  Valid types use <type> boolean, number string (note differ slightly R's names want emphasise scalar quantities; also note integer may produce unexpected errors integer-like numeric values). can also use list default.  Things schema known scalar atomics (name) automatically simplified. can index array-valued elements (files depends) way object-valued elements:   get list column file names per packet another hashes, probably less useful data.frame get extracting just files longer hash information aligned. can index fairly deeply; possible get orderly \"display name\" :   path need extract dot (likely package name plugin, custom.orderly.db) need escape dot backslash (, custom.orderly\\.db). probably need two slashes use raw string (recent versions R).","code":"extract = c(\"name\", \"parameters\", \"files\") extract = c(\"name\", pars = \"parameters\", \"files\") extract = c(x = \"parameters.x\") extract = c(x = \"parameters.x is number\") extract = c(file_path = \"files.path\", file_hash = \"files.hash\") extract = c(display = \"custom.orderly.description.display is string\")"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":"custom-orderly-metadata","dir":"Reference","previous_headings":"","what":"Custom 'orderly' metadata","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"Within custom.orderly, additional fields can extracted. format subject change, stored metadata schema (short term) way deserialise . probably best rely right now, expand section can.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_read.html","id":null,"dir":"Reference","previous_headings":"","what":"Read outpack metadata json file — orderly_metadata_read","title":"Read outpack metadata json file — orderly_metadata_read","text":"Low-level function reading metadata deserialising . function can used directly read metadata json file without reference root contains . may useful context reading metadata file written part failed run.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_read.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Read outpack metadata json file — orderly_metadata_read","text":"","code":"orderly_metadata_read(path, plugins = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_read.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Read outpack metadata json file — orderly_metadata_read","text":"path Path json file plugins Try deserialise data loaded plugins (see Details).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_read.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Read outpack metadata json file — orderly_metadata_read","text":"list outpack metadata; see schema details. contrast reading json file directly jsonlite::fromJSON, function take care convert scalar length-one vectors expected types.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_read.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Read outpack metadata json file — orderly_metadata_read","text":"Custom metadata saved plugins may deserialised expected called function, designed operate separately valid orderly root (.e., load data file regardless came ). plugins TRUE (default) deserialise data matches loaded plugin.  means behaviour function depends loaded plugin packages. can force running orderly2::orderly_config() within orderly directory, load declared plugins.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_new.html","id":null,"dir":"Reference","previous_headings":"","what":"Create a new report — orderly_new","title":"Create a new report — orderly_new","text":"Create new empty report.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_new.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Create a new report — orderly_new","text":"","code":"orderly_new(name, template = NULL, force = FALSE, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_new.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Create a new report — orderly_new","text":"name name report template template use. acceptable values now NULL (uses built-default) FALSE suppresses default content.  may support customisable templates future - let us know useful. force Create orderly file - <name>.R within existing directory src/<name>; may useful already created directory files first want help creating orderly file. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, just outpack (see orderly_init details). locate Logical, indicating configuration searched .  TRUE config given, orderly looks working directory parents finds .outpack directory","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_new.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Create a new report — orderly_new","text":"Nothing, called side effects ","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parameters.html","id":null,"dir":"Reference","previous_headings":"","what":"Declare orderly parameters — orderly_parameters","title":"Declare orderly parameters — orderly_parameters","text":"Declare orderly parameters. one call within file, though enforced! Typically put close top, though order really matter. Parameters scalar atomic values (e.g. string, number boolean) defaults must present literally (.e., may come variable ). Provide NULL default, case parameter required.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parameters.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Declare orderly parameters — orderly_parameters","text":"","code":"orderly_parameters(...)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parameters.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Declare orderly parameters — orderly_parameters","text":"... number parameters","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parameters.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Declare orderly parameters — orderly_parameters","text":"Undefined","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parameters.html","id":"behaviour-in-interactive-sessions","dir":"Reference","previous_headings":"","what":"Behaviour in interactive sessions","title":"Declare orderly parameters — orderly_parameters","text":"running interactively (.e., via source() running orderly file session copy/paste Rstudio), orderly_parameters() function different behaviour. First, look current environment (likely global environment) values parameters - , variables bound names parameters.  parameters found look default values use possible, possible either error prompt based global option orderly_interactive_parameters_missing_error. TRUE, ask enter value parameters (strings need entered quotes).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parse_file.html","id":null,"dir":"Reference","previous_headings":"","what":"Parse the orderly entrypoint script — orderly_parse_file","title":"Parse the orderly entrypoint script — orderly_parse_file","text":"expert use .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parse_file.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Parse the orderly entrypoint script — orderly_parse_file","text":"","code":"orderly_parse_file(path)  orderly_parse_expr(exprs, filename)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parse_file.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Parse the orderly entrypoint script — orderly_parse_file","text":"path Path orderly_* script exprs Parsed AST orderly_* script filename Name orderly_* file include metadata","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parse_file.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Parse the orderly entrypoint script — orderly_parse_file","text":"Parsed orderly entrypoint script","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parse_file.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Parse the orderly entrypoint script — orderly_parse_file","text":"Parses details calls orderly_ -script functions intermediate representation downstream use. Also validates calls orderly_* -script functions well-formed.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_add_metadata.html","id":null,"dir":"Reference","previous_headings":"","what":"Add metadata from plugin — orderly_plugin_add_metadata","title":"Add metadata from plugin — orderly_plugin_add_metadata","text":"Add plugin-specific metadata running packet. take describing. accumulate number bits metadata arbitrary fields, later serialise json.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_add_metadata.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Add metadata from plugin — orderly_plugin_add_metadata","text":"","code":"orderly_plugin_add_metadata(name, field, data)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_add_metadata.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Add metadata from plugin — orderly_plugin_add_metadata","text":"name name plugin; must used orderly_plugin_register orderly_plugin_context field name field add data . required even plugin produces one sort data, case can remove later within serialisation function. data Arbitrary data added currently running packet","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_add_metadata.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Add metadata from plugin — orderly_plugin_add_metadata","text":"Nothing, called side effects","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_context.html","id":null,"dir":"Reference","previous_headings":"","what":"Fetch plugin context — orderly_plugin_context","title":"Fetch plugin context — orderly_plugin_context","text":"Fetch running context, use within plugin.  intention within free functions plugin makes available, call function get information state packet.  typically call orderly_plugin_add_metadata() afterwards.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_context.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Fetch plugin context — orderly_plugin_context","text":"","code":"orderly_plugin_context(name, envir)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_context.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Fetch plugin context — orderly_plugin_context","text":"name Name plugin envir environment calling function. can typically pass parent.frame() (rlang::caller_env()) function calling orderly_plugin_context() function called user. argument effect interactive use (envir almost certainly global environment).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_context.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Fetch plugin context — orderly_plugin_context","text":"list elements: is_active: logical, indicating running orderly_run(); may need change behaviour depending value. path: path running packet. almost always working directory, unless packet contains calls setwd() similar. may create files . config: configuration plugin, processing plugin's read function (see orderly_plugin_register) envir: environment packet running . Often global environment, assume ! may read write environment. src: path packet source directory. different current directory packet running, user interactively working report. may read directory must write parameters: parameters passed run report.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_context.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Fetch plugin context — orderly_plugin_context","text":"plugin function called, orderly2 running one two modes; (1) within orderly_run(), case part way creating packet brand new directory, possibly using special environment evaluation, (2) interactively, user developing report. plugin needs able support modes, function return information state help cope either case.","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_register.html","id":null,"dir":"Reference","previous_headings":"","what":"Register an orderly plugin — orderly_plugin_register","title":"Register an orderly plugin — orderly_plugin_register","text":"Create orderly plugin. plugin typically defined package used extend orderly enabling new functionality, declared orderly_config.yml orderly file, affecting running reports primarily creating new objects report environment.  system discussed detail vignette(\"plugins\"), expanded (likely breaking ways) soon.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_register.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Register an orderly plugin — orderly_plugin_register","text":"","code":"orderly_plugin_register(   name,   config,   serialise = NULL,   deserialise = NULL,   cleanup = NULL,   schema = NULL )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_register.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Register an orderly plugin — orderly_plugin_register","text":"name name plugin, typically package name config function read, check process configuration section orderly_config.yml. function passed deserialised data plugin's section orderly_config.yml, full path file.  order loading plugins defined, plugin must standalone try interact plugins load. return processed copy configuration data, passed second argument read. serialise function serialise metadata added plugin's functions outpack metadata. passed list entries pushed via orderly_plugin_add_metadata(); named list names corresponding field argument orderly_plugin_add_metadata list element unnamed list values corresponding data. NULL, serialisation done, metadata plugin added. deserialise function deserialise metadata serialised serialise function. intended help deal issues disambiguating unserialising objects json (scalars vs arrays lenth 1, data.frames vs lists--lists etc), make plugin nicer work orderly_metadata_extract(). function given single argument data data jsonlite::fromJSON(..., simplifyVector = FALSE) apply required simplifications , returning modified copy argument. cleanup Optionally, function clean state plugin uses. can call orderly_plugin_context within function access anything need . given, cleanup done. schema Optionally path, within package, schema metadata created plugin; omit .json extension.  file contains sources file inst/plugin/myschema.json pass plugin/myschema.  See vignette(\"plugins\") details.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_register.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Register an orderly plugin — orderly_plugin_register","text":"Nothing, function called side effect registering plugin.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_prune_orphans.html","id":null,"dir":"Reference","previous_headings":"","what":"Prune orphan packet metadata — orderly_prune_orphans","title":"Prune orphan packet metadata — orderly_prune_orphans","text":"Prune orphan packets metadata store.  function can used remove references packets longer reachable; happened deleted packet manually archive ran orderly_validate_archive removed location.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_prune_orphans.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Prune orphan packet metadata — orderly_prune_orphans","text":"","code":"orderly_prune_orphans(root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_prune_orphans.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Prune orphan packet metadata — orderly_prune_orphans","text":"root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_prune_orphans.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Prune orphan packet metadata — orderly_prune_orphans","text":"Invisibly, character vector orphaned packet ids","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_prune_orphans.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Prune orphan packet metadata — orderly_prune_orphans","text":"orphan packet used anywhere, can easily drop - never existed.  referenced metadata know elsewhere locally, problem upstream location (one happen). referenced packet run locally, metadata deleted. expose function mostly users want expunge permanently reference previously run packets. hope never need really reason run .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query.html","id":null,"dir":"Reference","previous_headings":"","what":"Construct outpack query — orderly_query","title":"Construct outpack query — orderly_query","text":"Construct outpack query, typically passed orderly_search","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Construct outpack query — orderly_query","text":"","code":"orderly_query(expr, name = NULL, scope = NULL, subquery = NULL)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Construct outpack query — orderly_query","text":"expr query expression. NULL expression matches everything. name Optionally, name packet scope query . intersected scope arg shorthand way running scope = list(name = \"name\") scope Optionally, scope query limit packets searched pars subquery Optionally, named list subqueries can referenced name expr.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Construct outpack query — orderly_query","text":"orderly_query object, modified, can passed orderly_search()","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query_explain.html","id":null,"dir":"Reference","previous_headings":"","what":"Explain a query — orderly_query_explain","title":"Explain a query — orderly_query_explain","text":"Explain query matched. experimental output change. moment, can tell query matches, fails match based one number &&-ed together clauses.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query_explain.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Explain a query — orderly_query_explain","text":"","code":"orderly_query_explain(   ...,   parameters = NULL,   envir = parent.frame(),   options = NULL,   root = NULL )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query_explain.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Explain a query — orderly_query_explain","text":"... Arguments passed orderly_query, perhaps just query expression parameters Optionally, named list parameters substitute query (using : prefix) envir Optionally, environment substitute query (using environment: prefix). default use calling environment, can explicitly pass want control lookup happens. options Optionally, orderly_search_options object controlling search performed, packets considered scope. provided, default options used (.e., orderly2::orderly_search_options()) root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query_explain.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Explain a query — orderly_query_explain","text":"object class orderly_query_explain, can inspected (contents subject change) print method show user-friendly summary query result.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_resource.html","id":null,"dir":"Reference","previous_headings":"","what":"Declare orderly resources — orderly_resource","title":"Declare orderly resources — orderly_resource","text":"Declare file, group files, orderly resource. explicitly declaring files resources orderly mark files immutable inputs validate analysis modify run orderly_run()","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_resource.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Declare orderly resources — orderly_resource","text":"","code":"orderly_resource(files)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_resource.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Declare orderly resources — orderly_resource","text":"files number names files","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_resource.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Declare orderly resources — orderly_resource","text":"Invisibly, character vector resources included call. rely order files expanded directories, likely platform dependent.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":null,"dir":"Reference","previous_headings":"","what":"Run a report — orderly_run","title":"Run a report — orderly_run","text":"Run report.  create new directory drafts/<reportname>, copy declared resources , run script check expected artefacts created.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Run a report — orderly_run","text":"","code":"orderly_run(   name,   parameters = NULL,   envir = NULL,   echo = TRUE,   search_options = NULL,   root = NULL,   locate = TRUE )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Run a report — orderly_run","text":"name Name report run. leading ./ src/ trailing / path parts removed (e.g., added autocomplete). parameters Parameters passed report. named list parameters declared orderly.yml.  parameter must scalar character, numeric, integer logical. envir environment used evaluate report script; default use global environment, may always wanted. echo Optional logical control printing output source() console. search_options Optional control locations, used orderly_dependency; converted orderly_search_options object, see Details. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, just outpack (see orderly_init details). locate Logical, indicating configuration searched .  TRUE config given, orderly looks working directory parents finds .outpack directory","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Run a report — orderly_run","text":"id created report (string)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"locations-used-in-dependency-resolution","dir":"Reference","previous_headings":"","what":"Locations used in dependency resolution","title":"Run a report — orderly_run","text":"packet depends packets, want control locations used find appropriate packets. control passed function argument orderly_dependency property way packet created packet ; importantly different users may different names locations makes little sense encode location name source code. Alternatively, want use different locations different contexts (initial development want include local copies packets possible dependencies vs resolving dependencies resolved one locations! Similarly, might want include packets known locations currently downloaded onto machine - pulling packets take anything seconds hours depending size speed network connection (pulling packets mean packet fails run). allow control can pass argument search_options, orderly_search_options object, allows control names locations use, whether metadata refreshed pull anything packets currently downloaded considered candidates. effect running interactively, case can specify search options (root specific) orderly_interactive_set_search_options","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"which-packets-might-be-selected-from-locations-","dir":"Reference","previous_headings":"","what":"Which packets might be selected from locations?","title":"Run a report — orderly_run","text":"search_options argument controls outpack searches packets given query anything might moved network (one outpack archive another). default everything resolved locally ; can depend packets unpacked within current archive.  pass search_options argument contains allow_remote = TRUE (see orderly_search_options packets known anywhere candidates using dependencies needed pull resolved files remote location. Note even packet locally present might needed - content anywhere else unpacked packet reuse content without re-fetching. pull_metadata = TRUE, refresh location metadata pulling, location argument controls locations pulled .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"equivalence-to-the-old-use-draft-option","dir":"Reference","previous_headings":"","what":"Equivalence to the old use_draft option","title":"Run a report — orderly_run","text":"location handling generalises orderly (v1)'s old use_draft option, terms location argument orderly2::orderly_search_options`: use_draft = TRUE location = \"local\" use_draft = FALSE location = c(...) provide locations except local (setdiff(orderly2::orderly_location_list(), \"local\")) use_draft = \"newer\" location = NULL (last option one people preferred new default behaviour). addition, resolve dependencies currently exist production right now options:   updates current metadata production, runs queries packets known remote, depends even (yet) locally.  functionality never available orderly version 1, though intended support .","code":"location = \"production\", pull_metadata = TRUE, require_unpacked = FALSE"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"running-with-a-source-tree-separate-from-outpack-root","dir":"Reference","previous_headings":"","what":"Running with a source tree separate from outpack root","title":"Run a report — orderly_run","text":"Sometimes useful run things different place disk outpack root. know two cases come : running reports within runner server, make clean clone source tree particular git reference new temporary directory run report , insert orderly repo fixed non-temporary location. user convenient torun report hard drive store archive metadata (larger) shared drive. first instance, source path <src> contains file orderly_config.yml directory src/ source reports, separate path <root> contains directory .outpack/ metadata - may also unpacked archive, .git/ directory depending configuration. (Later make sense support \"bare\" outpack layout.)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"manually-setting-report-source-directory","dir":"Reference","previous_headings":"","what":"Manually setting report source directory","title":"Run a report — orderly_run","text":"manually set report source directory, need set path directory ORDERLY_REPORT_SRC environment variable.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Run a report — orderly_run","text":"","code":"# Create a simple example: path <- orderly2::orderly_example(\"default\") #> ✔ Created orderly root at '/tmp/RtmpeoP8u3/file1722ab3760d'  # Run the 'data' task: orderly2::orderly_run(\"data\", root = path) #> Error: Directory does not exist: '/tmp/RtmpeoP8u3/file1722ab3760d'  # After running, a finished packet appears in the archive: fs::dir_tree(path) #> Error: [ENOENT] Failed to search directory '/tmp/RtmpeoP8u3/file1722ab3760d': no such file or directory  # and we can query the metadata: orderly2::orderly_metadata_extract(name = \"data\", root = path) #> Error: Directory does not exist: '/tmp/RtmpeoP8u3/file1722ab3760d'"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run_info.html","id":null,"dir":"Reference","previous_headings":"","what":"Information about currently running report — orderly_run_info","title":"Information about currently running report — orderly_run_info","text":"Fetch information actively running report.  allows reflect information report back part report, example embedding current report id, information computed dependencies. information slightly different format orderly version 1.x (currently) include information dependencies run outside orderly_run, never reliable previously.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run_info.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Information about currently running report — orderly_run_info","text":"","code":"orderly_run_info()"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run_info.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Information about currently running report — orderly_run_info","text":"list elements name: name current report id: id current report, NA running interactively root: orderly root path depends: data frame information dependencies (available interactively) index: integer sequence along calls orderly_dependency name: name dependency query: query used find dependency id: computed id included packet filename: file used packet : filename used locally","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search.html","id":null,"dir":"Reference","previous_headings":"","what":"Query outpack's database — orderly_search","title":"Query outpack's database — orderly_search","text":"Evaluate query outpack database, returning vector matching packet ids.  Note default searches packets unpacked available direct use computer; search within packets known locations (might know via metadata) need use options argument.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Query outpack's database — orderly_search","text":"","code":"orderly_search(   ...,   parameters = NULL,   envir = parent.frame(),   options = NULL,   root = NULL )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Query outpack's database — orderly_search","text":"... Arguments passed orderly_query, perhaps just query expression parameters Optionally, named list parameters substitute query (using : prefix) envir Optionally, environment substitute query (using environment: prefix). default use calling environment, can explicitly pass want control lookup happens. options Optionally, orderly_search_options object controlling search performed, packets considered scope. provided, default options used (.e., orderly2::orderly_search_options()) root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Query outpack's database — orderly_search","text":"character vector matching ids. case match query returning single value (e.g., latest(...) single(...)) character missing value (NA_character_)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search_options.html","id":null,"dir":"Reference","previous_headings":"","what":"Packet search options — orderly_search_options","title":"Packet search options — orderly_search_options","text":"Options controlling packet searches carried , example via orderly_search orderly_run. details never included metadata alongside query (, part query even though affect ).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search_options.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Packet search options — orderly_search_options","text":"","code":"orderly_search_options(   location = NULL,   allow_remote = FALSE,   pull_metadata = FALSE )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search_options.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Packet search options — orderly_search_options","text":"location Optional vector locations pull . might future expand allow wildcards exceptions. allow_remote Logical, indicating allow packets found currently unpacked (.e., known location metadata ). TRUE, conjunction orderly_dependency might pull large quantity data. pull_metadata Logical, indicating pull metadata immediately search. location given, pass orderly_location_pull_metadata filter locations update.  pulling many packets sequence, want update option FALSE first pull.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search_options.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Packet search options — orderly_search_options","text":"object class orderly_search_options modified creation (see note pull_metadata)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_shared_resource.html","id":null,"dir":"Reference","previous_headings":"","what":"Copy shared resources into a packet directory — orderly_shared_resource","title":"Copy shared resources into a packet directory — orderly_shared_resource","text":"Copy shared resources packet directory. can use share common resources (data code) multiple packets. Additional metadata added keep track files came .  Using function requires shared resources directory shared/ exists orderly root; error raised configured attempt fetch files.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_shared_resource.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Copy shared resources into a packet directory — orderly_shared_resource","text":"","code":"orderly_shared_resource(...)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_shared_resource.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Copy shared resources into a packet directory — orderly_shared_resource","text":"... shared resources copy. arguments named, name destination file value filename within shared resource directory. can use limited form string interpolation names argument; using ${variable} pick values envir substitute string.  similar interpolation might familiar glue::glue similar, much simpler concatenation fancy features supported.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_shared_resource.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Copy shared resources into a packet directory — orderly_shared_resource","text":"Invisibly, data.frame columns (fileames copied running packet) (filenames within shared/).  orderly_resource, rely ordering directory expansion performed.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_strict_mode.html","id":null,"dir":"Reference","previous_headings":"","what":"Enable orderly strict mode — orderly_strict_mode","title":"Enable orderly strict mode — orderly_strict_mode","text":"Put orderly2 \"strict mode\", closer defaults orderly 1.0.0; mode explicitly included files (via orderly_resource orderly_shared_resource) copied running packet, warn unexpected files end run.  Using strict mode allows orderly2 aggressive deletes files within source directory, accurate reports , faster start packets developing interactively.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_strict_mode.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Enable orderly strict mode — orderly_strict_mode","text":"","code":"orderly_strict_mode()"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_strict_mode.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Enable orderly strict mode — orderly_strict_mode","text":"Undefined","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_strict_mode.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Enable orderly strict mode — orderly_strict_mode","text":"future, may extend strict mode allow requiring computation occurs within orderly functions (.e., requirements run packet fully known actually running ).  likely default behaviour orderly_strict_mode gain argument. allow server processes either override value (enabling even explicitly given) /require .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_validate_archive.html","id":null,"dir":"Reference","previous_headings":"","what":"Validate unpacked packets. — orderly_validate_archive","title":"Validate unpacked packets. — orderly_validate_archive","text":"Validate unpacked packets. time, expect function become fully featured, validating .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_validate_archive.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Validate unpacked packets. — orderly_validate_archive","text":"","code":"orderly_validate_archive(..., action = \"inform\", root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_validate_archive.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Validate unpacked packets. — orderly_validate_archive","text":"... Either arguments search can constructed (useful options include name = \"something\"), character vector ids, leave empty validate everything. action action take finding invalid packet. See Details. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_validate_archive.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Validate unpacked packets. — orderly_validate_archive","text":"Invisibly, character vector repaired (invalid) packets.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_validate_archive.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Validate unpacked packets. — orderly_validate_archive","text":"actions can take finding invalid packet : inform (default): just print information problem orphan: mark packet orphaned within metadata, touch files archive (default directory archive/) - safe option leave consistent state without deleting anything. delete: addition marking packet orphan, also delete files archive. Later, add \"repair\" option try fix broken packets. validation interacts option core.require_complete_tree; option TRUE, packet valid (recursive) dependencies also valid, action apply packets also upstream dependencies invalidated.  validation happen even query implied ... include packets complete tree required. validation also interact core.use_file_store repair supported, becomes trivial.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":null,"dir":"Reference","previous_headings":"","what":"Query index — query_index","title":"Query index — query_index","text":"Class managing active index whilst evaluating query","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"public-fields","dir":"Reference","previous_headings":"","what":"Public fields","title":"Query index — query_index","text":"index packet index depends Named list data frames. Names packet ids, values packets depended packet id (.e. parents). uses Named list data frames. Names packet ids, values packets used packet id (.e. children). root outpack root object","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"public-methods","dir":"Reference","previous_headings":"","what":"Public methods","title":"Query index — query_index","text":"query_index$new() query_index$get_packet_depends() query_index$get_packet_uses()","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"method-new-","dir":"Reference","previous_headings":"","what":"Method new()","title":"Query index — query_index","text":"Create new query_index object","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Query index — query_index","text":"","code":"query_index$new(root, index, depends, uses)"},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Query index — query_index","text":"root outpack root object index packet index data frame depends Named list data frames. Names packet ids, values packets depended packet id (.e. parents). uses Named list data frames. Names packet ids, values packets used packet id (.e. children). data depends relationships flow direction.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"method-get-packet-depends-","dir":"Reference","previous_headings":"","what":"Method get_packet_depends()","title":"Query index — query_index","text":"Get ids packets packet depends specified level","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"usage-1","dir":"Reference","previous_headings":"","what":"Usage","title":"Query index — query_index","text":"","code":"query_index$get_packet_depends(id, depth)"},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"arguments-1","dir":"Reference","previous_headings":"","what":"Arguments","title":"Query index — query_index","text":"id id packet get parents depth Depth parents get, depth 1 gets immediate parents depth 2 gets parents parents parents, depth Inf recurse whole tree get parents","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"returns","dir":"Reference","previous_headings":"","what":"Returns","title":"Query index — query_index","text":"ids parents packet","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"method-get-packet-uses-","dir":"Reference","previous_headings":"","what":"Method get_packet_uses()","title":"Query index — query_index","text":"Get ids packets used packet","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"usage-2","dir":"Reference","previous_headings":"","what":"Usage","title":"Query index — query_index","text":"","code":"query_index$get_packet_uses(id, depth)"},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"arguments-2","dir":"Reference","previous_headings":"","what":"Arguments","title":"Query index — query_index","text":"id id packet get children depth Depth children get, depth 1 gets immediate children depth 2 gets children children children, depth Inf recurse whole tree get children","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"returns-1","dir":"Reference","previous_headings":"","what":"Returns","title":"Query index — query_index","text":"ids children packet","code":""}]
+[{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"type-of-location","dir":"Articles","previous_headings":"","what":"Type of location","title":"Collaborative analysis","text":"orderly2 supports two types locations default: path: orderly2 working copy can find filesystem. Note copy people can see (example network share, cloud-synced file system Dropbox OneDrive) see Sharing packets collaborators using shared file system details http: require running HTTP API, either via outpack_server packit location system somewhat extensible, details subject change. documentation behave regardless location stored mechanism transport, can focus instead workflows.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"an-example","dir":"Articles","previous_headings":"","what":"An example","title":"Collaborative analysis","text":"Suppose two researchers, Alice Bob, collaborating piece analysis. purposes example (find common situation practice) share source tree analysis git. Alice Bob clone repo onto machines, using favourite git client (terminal git, via GitHub desktop, gert) moment, including hidden files (except .git), Alice sees: point, cloning Alice .outpack directory (see ) Alice needs run orderly::orderly_init() first: plan work Bob, sharing results shared server, Alice adds orderly2 configuration orderly2::orderly_location_add(): done, can run analysis: Perhaps takes several goes Alice happy analysis, point something ready share. can “push” final packet onto server: Now, consider Bob. also needs source code cloned, orderly initialised, server added: (, Bob also used name server refer shared server, used anything; similar git’s use origin.) Bob can now query packets available server: seen new “data” packet , can pull locally (TODO: mrc-4414 makes nicer): Now Bob position develop packet Alice ran (20240905-080221-6b00c525)","code":"fs::dir_tree(\".\", all = TRUE, glob = \".git\", invert = TRUE) ## . ## ├── orderly_config.yml ## └── src ##     └── data ##         └── data.R orderly2::orderly_list_src() ## [1] \"data\" orderly2::orderly_init(\".\") ## ✔ Created orderly root at '/tmp/RtmpXFj1Og/file1a6b795204b0/alice' ## ✔ Wrote '.gitignore' orderly2::orderly_list_src() ## [1] \"data\" orderly2::orderly_location_add(   \"server\", \"http\", list(url = \"http://orderly.example.com\")) id <- orderly2::orderly_run(\"data\") ## ℹ Starting packet 'data' `20240905-080221-6b00c525` at 2024-09-05 08:02:21.423466 ## > orderly2::orderly_artefact(\"data.rds\", description = \"Final data\") ## > saveRDS(mtcars, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240905-080221-6b00c525 at 2024-09-05 08:02:21.478472 (0.05500579 secs) orderly2::orderly_location_push(id, \"server\") orderly2::orderly_init(\".\") ## ✔ Created orderly root at '/tmp/RtmpXFj1Og/file1a6b795204b0/bob' ## ✔ Wrote '.gitignore' orderly2::orderly_location_add(   \"server\", \"http\", list(url = \"http://orderly.example.com\")) orderly2::orderly_metadata_extract(   name = \"data\",   options = list(allow_remote = TRUE, pull_metadata = TRUE)) ##                         id name parameters ## 1 20240905-080221-6b00c525 data orderly2::orderly_location_pull_packet(id) ## ℹ Looking for suitable files already on disk ## ℹ Need to fetch 2 files (1.3 kB) from 1 location ## ⠙ Fetching file 1/2 (95 B) from 'server' | ETA:  0s [2ms] ## ✔ Fetched 2 files (1.3 kB) from 'server' in 19ms. ##"},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"possible-working-patterns","dir":"Articles","previous_headings":"","what":"Possible working patterns","title":"Collaborative analysis","text":"seen several broad patterns distributing packets. Central server, push allowed: case ; flexible requires everyone relaxed packets created ultimately trusting colleagues exercising good environment hygiene. Just rules enforced computer though, doesn’t mean might ideas within group can push. may useful push HPC system running computationally demanding tasks . Central server, final copy run : case, Alice Bob can run things server, “canonical” copies, nothing pushed server. Reports can run server ssh-ing onto server running packets manually (TODO: mrc-4412) using web interface (possible OrderlyWeb possible packit). approach works well primary goal confidence packets everyone works dependencies created “clean” environment unexpected global dependencies. web version, can also enforce things like running default branch. Alice Bob end collection packets local archives mix canonical ones (server) locally-created ones, local ones always dead end never get shared anyone. result, Alice Bob may delete archive directories without great concern. running packets, make sure packet run similar way packet run server, Bob may want ensure dependencies found server considered. , can create search options pass orderly2::orderly_run running report. developing, Bob may find passing orderly2::orderly_interactive_set_search_options() helpful mean calls orderly2::orderly_description() use packets server. Staging production servers: might set multiple servers, one staging, one production. staging servers never source truth might (might ) allow people push , less strict rules gets run (perhaps allowing running packets feature branch). setting, users end packets created locally, staging machines production server local archive. staging server probably end mix packets packets production. approach specifying search options useful choosing included final packet.","code":"search_options <- orderly2::orderly_search_options(location = \"server\")"},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"pulling-complete-trees-or-not","dir":"Articles","previous_headings":"","what":"Pulling complete trees, or not","title":"Collaborative analysis","text":"pull packets server (orderly2::orderly_location_pull_packet()), depending settings either end just packet request, packet plus recursive dependencies. controlled configuration option core.require_complete_tree, FALSE default; suitable users, servers set core.require_complete_tree = TRUE. situations , users never holding canonical source packets - matter don’t dependencies. hold metadata packets (, recursively) download dependencies later wanted inspect . Typically though, just extra space used disk. push packet server, always pushes complete tree.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"moving-as-little-data-as-possible","dir":"Articles","previous_headings":"","what":"Moving as little data as possible","title":"Collaborative analysis","text":"pulling packets, first user pulls metadata server. orderly2::orderly_location_metadata_pull run, metadata server updated client (mrc-4444 make lazier). now know everything server holds, hashes files contained packet. likely already know files within packet requested, client first looks see exist within archive, verifying changed. requests server files resolve locally. recursive pull (core.require_complete_tree = TRUE) look union files set packets currently missing archive. frequently much less full set files within packets. algorithm similar push: look full dependency chain target packet find file ids ask server yet know unknown subset get union file hashes ask server subset files yet know push files - done one time present done simultaneously async requests future files sever upload missing metadata, chronological order server verifies packet metadata dependencies referenced files multi-step process means avoid copying data already known , well avoiding moving data multiple times. also means process pushing (pulling) interrupted can safely resumed state always consistent.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"interaction-between-source-tree-and--outpack","dir":"Articles","previous_headings":"","what":"Interaction between source tree and .outpack","title":"Collaborative analysis","text":"important .outpack directory shared via git; warn now, can use orderly2::ordery_gitignore_update() automatically create suitable .gitignore file prevent accidentally committed. However, Alice Bob starting new machine : clone source repository run orderly2::orderly_init() within directory run orderly2 commands normal Without running orderly2::orderly_init(), get error prompting initialise repo. requirement different repositories share configuration options passed orderly2::orderly_init(); can enable disable file store different sets locations enabled, depending workflows. means , practice, association convention set related orderly2 locations source tree can orderly2 repository points locations refer different source trees!","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"sharing-packets-with-collaborators-using-a-shared-file-system","dir":"Articles","previous_headings":"","what":"Sharing packets with collaborators using a shared file system","title":"Collaborative analysis","text":"One simplest ways share packets collaborator shared file system, example network share, cloud-synced file system Dropbox OneDrive. can follows. Create new folder orderly remote shared file system. Alice synced path_sharepoint_alice. Initialise orderly location shared file system Create orderly store file store complete tree. See orderly2::orderly_init() details. Add location Push packets want share available collaborator pull. Note data pushed file store .outpack directory. .dot files considered hidden operating system, “show hidden files” file browser see pushed packet. collaborator can now pull shared file. , : Sync drive location machine. Bob synced path_sharepoint_bob Add location Pull metadata use packets desired","code":"orderly2::orderly_init(   root = path_sharepoint_alice,    path_archive = NULL,    use_file_store = TRUE,    require_complete_tree = TRUE ) ## ✔ Created orderly root at '/tmp/RtmpXFj1Og/file1a6b795204b0/sharepoint' orderly2::orderly_location_add(   \"sharepoint\", \"path\", list(path = path_sharepoint_alice)) orderly2::orderly_location_push(id, \"sharepoint\") orderly2::orderly_location_add(   \"alices_orderly\", \"path\", list(path = path_sharepoint_bob)) orderly2::orderly_location_pull_metadata(   \"alices_orderly\" )"},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"using-dependencies","dir":"Articles","previous_headings":"","what":"Using dependencies","title":"Dependencies between packets","text":", show practically use dependencies common scenarios increasing complexity. code examples purposefully -simple order keep presentation straightforward, see end document discussion complex pieces code might “optimally” .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"basic-use","dir":"Articles","previous_headings":"Using dependencies","what":"Basic use","title":"Dependencies between packets","text":"primary mechanism using dependencies call orderly2::orderly_dependency() within orderly file; finds suitable completed packet copies files found within packet current report. src/analysis/analysis.R contains: , ’ve used orderly2::orderly_dependency() pull file data.rds recent version (latest()) data packet, ’ve used file normal make plot, ’ve saved analysis.png (similar example vignette(\"introduction\"), get us started). look metadata packet created analysis report, can see used 20240905-080224-b9674449 dependency: (indeed , one copy data packet pick ).","code":"## src ## ├── analysis ## │   └── analysis.R ## └── data ##     ├── data.R ##     └── data.csv orderly2::orderly_dependency(\"data\", \"latest()\", \"data.rds\") d <- readRDS(\"data.rds\") png(\"analysis.png\") plot(y ~ x, d) dev.off() id1 <- orderly2::orderly_run(\"data\") ## ℹ Starting packet 'data' `20240905-080224-b9674449` at 2024-09-05 08:02:24.726524 ## > d <- read.csv(\"data.csv\") ## > d$z <- resid(lm(y ~ x, d)) ## > saveRDS(d, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240905-080224-b9674449 at 2024-09-05 08:02:24.761707 (0.03518271 secs) id2 <- orderly2::orderly_run(\"analysis\") ## ℹ Starting packet 'analysis' `20240905-080224-c961843d` at 2024-09-05 08:02:24.788789 ## > orderly2::orderly_dependency(\"data\", \"latest()\", \"data.rds\") ## ℹ Depending on data @ `20240905-080224-b9674449` (via latest(name == \"data\")) ## > d <- readRDS(\"data.rds\") ## > png(\"analysis.png\") ## > plot(y ~ x, d) ## > dev.off() ## agg_png  ##       2 ## ✔ Finished running analysis.R ## ℹ Finished 20240905-080224-c961843d at 2024-09-05 08:02:24.896412 (0.1076229 secs) orderly2::orderly_metadata(id2)$depends ##                     packet                  query        files ## 1 20240905-080224-b9674449 latest(name == \"data\") data.rds...."},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"filtering-candidates-by-parameters","dir":"Articles","previous_headings":"Using dependencies","what":"Filtering candidates by parameters","title":"Dependencies between packets","text":"example, query simple — recently created packet name data. One common pattern see analysis might parameter (example country name) downstream analysis might share parameter want pull data country. src/data/data.R containing: can run several values cyl: follow-analysis contains: query latest(parameter:cyl == :cyl) says “find recent packet ’s parameter”cyl” (parameter:cyl) parameter currently running report (:cyl).","code":"## src ## ├── analysis ## │   └── analysis.R ## └── data ##     └── data.R orderly2::orderly_parameters(cyl = NULL) d <- mtcars[mtcars$cyl == cyl, ] saveRDS(d, \"data.rds\") orderly2::orderly_run(\"data\", list(cyl = 4)) ## ℹ Starting packet 'data' `20240905-080225-40f9c854` at 2024-09-05 08:02:25.256076 ## ℹ Parameters: ## • cyl: 4 ## > orderly2::orderly_parameters(cyl = NULL) ## > d <- mtcars[mtcars$cyl == cyl, ] ## > saveRDS(d, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240905-080225-40f9c854 at 2024-09-05 08:02:25.286214 (0.03013873 secs) ## [1] \"20240905-080225-40f9c854\" orderly2::orderly_run(\"data\", list(cyl = 6)) ## ℹ Starting packet 'data' `20240905-080225-4da56319` at 2024-09-05 08:02:25.30542 ## ℹ Parameters: ## • cyl: 6 ## > orderly2::orderly_parameters(cyl = NULL) ## > d <- mtcars[mtcars$cyl == cyl, ] ## > saveRDS(d, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240905-080225-4da56319 at 2024-09-05 08:02:25.332333 (0.02691317 secs) ## [1] \"20240905-080225-4da56319\" orderly2::orderly_run(\"data\", list(cyl = 8)) ## ℹ Starting packet 'data' `20240905-080225-59e680b0` at 2024-09-05 08:02:25.353485 ## ℹ Parameters: ## • cyl: 8 ## > orderly2::orderly_parameters(cyl = NULL) ## > d <- mtcars[mtcars$cyl == cyl, ] ## > saveRDS(d, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240905-080225-59e680b0 at 2024-09-05 08:02:25.378441 (0.02495623 secs) ## [1] \"20240905-080225-59e680b0\" orderly2::orderly_parameters(cyl = NULL) orderly2::orderly_dependency(   \"data\",   \"latest(parameter:cyl == this:cyl)\",   \"data.rds\") d <- readRDS(\"data.rds\") png(\"analysis.png\") plot(mpg ~ disp, d) dev.off() orderly2::orderly_run(\"analysis\", list(cyl = 4)) ## ℹ Starting packet 'analysis' `20240905-080225-8719a52e` at 2024-09-05 08:02:25.530097 ## ℹ Parameters: ## • cyl: 4 ## > orderly2::orderly_parameters(cyl = NULL) ## > orderly2::orderly_dependency( ## +   \"data\", ## +   \"latest(parameter:cyl == this:cyl)\", ## +   \"data.rds\") ## ℹ Depending on data @ `20240905-080225-40f9c854` (via latest(parameter:cyl == this:cyl && name == \"data\")) ## > d <- readRDS(\"data.rds\") ## > png(\"analysis.png\") ## > plot(mpg ~ disp, d) ## > dev.off() ## agg_png  ##       2 ## ✔ Finished running analysis.R ## ℹ Finished 20240905-080225-8719a52e at 2024-09-05 08:02:25.579071 (0.04897404 secs) ## [1] \"20240905-080225-8719a52e\""},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"interpreting-errors","dir":"Articles","previous_headings":"Using dependencies","what":"Interpreting errors","title":"Dependencies between packets","text":"query fails resolve candidate error: error message tries fairly self explanatory; failed find packet satisfies querylatest(parameter:cyl == :cyl && name == \"data\"); note report name data become part query, two conditions matched . error suggests running rlang::last_error()$explanation information, can : orderly_query_explain object, tries come reasons query might matched; ’ll expand future let us know might like see. tells query can decomposed two subqueries (match parameter cyl 9000), matched packets B (match packet name data), matched 3 packets. subquery matched packets pairs don’t try guide towards problematic pairs. can also ask orderly2 explain query : save object can explore detail: (worked rlang::last_error()$explanation$parts$). can also use orderly2::orderly_metadata_extract work values might looked :","code":"orderly2::orderly_run(\"analysis\", list(cyl = 9000)) ## ℹ Starting packet 'analysis' `20240905-080225-a81e462f` at 2024-09-05 08:02:25.658968 ## ℹ Parameters: ## • cyl: 9000 ## > orderly2::orderly_parameters(cyl = NULL) ## > orderly2::orderly_dependency( ## +   \"data\", ## +   \"latest(parameter:cyl == this:cyl)\", ## +   \"data.rds\") ## ✖ Error running analysis.R ## ℹ Finished 20240905-080225-a81e462f at 2024-09-05 08:02:25.717692 (0.05872464 secs) ## Error in `orderly2::orderly_run()`: ## ! Failed to run report ## Caused by error in `outpack_packet_use_dependency()`: ## ! Failed to find packet for query 'latest(parameter:cyl == this:cyl && ##   name == \"data\")' ## ℹ See 'rlang::last_error()$explanation' for details rlang::last_error()$explanation ## Evaluated query: 'latest(A && B)' and found 0 packets ## • A (parameter:cyl == this:cyl): 0 packets ##  ## • B (name == \"data\"): 3 packets orderly2::orderly_query_explain(   quote(latest(parameter:cyl == 9000)), name = \"data\") ## Evaluated query: 'latest(A && B)' and found 0 packets ## • A (parameter:cyl == 9000): 0 packets ##  ## • B (name == \"data\"): 3 packets explanation <- orderly2::orderly_query_explain(   quote(latest(parameter:cyl == 9000)), name = \"data\") explanation$parts$B ## $name ## [1] \"B\" ##  ## $str ## [1] \"name == \\\"data\\\"\" ##  ## $expr ## name == \"data\" ##  ## $n ## [1] 3 ##  ## $found ## [1] \"20240905-080225-40f9c854\" \"20240905-080225-4da56319\" ## [3] \"20240905-080225-59e680b0\" orderly2::orderly_metadata_extract(   name = \"data\",   extract = c(cyl = \"parameters.cyl is number\")) ##                         id cyl ## 1 20240905-080225-40f9c854   4 ## 2 20240905-080225-4da56319   6 ## 3 20240905-080225-59e680b0   8"},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"filtering-candidates-in-other-ways","dir":"Articles","previous_headings":"Using dependencies","what":"Filtering candidates in other ways","title":"Dependencies between packets","text":"saw two types filtering candidates: latest() selected recent packet latest(parameter:cyl == :cyl) found packet whose parameter matched one parameters. used latest(parameter:cyl == 4) hard code specific parameter value, used latest(parameter:cyl == environment:cyl) match whatever value cyl took evaluating environment. Instead query, can provide single id (e.g, 20240905-080225-8719a52e), mean even new copies data packet created, dependency always resolve value. can chain together logical operations && (sides must true) || (either side must true), group conditions parentheses. addition ==, usual complement comparison operators work. might complex queries like practice people queries series restrictions &&.","code":"latest((parameter:x == 1 || parameter:x == 2) && parameter:y > 10)"},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"computing-dependencies-and-using-many-dependencies-at-once","dir":"Articles","previous_headings":"Using dependencies","what":"Computing dependencies and using many dependencies at once","title":"Dependencies between packets","text":"One common pattern map-reduce pattern set orderly reports. , set packets created vector parameters, perhaps chain different reports parameter, combined together. parameter p takes values “x”, “y” “z”, might look like: , D want combine three copies C packet, one p “x”, “y” “z”. Especially three values hard coded, might just write Note call vary second argument select different parameter value, third argument naming destination file different name (end three files data/). can write loop: , second argument use environment:p fetch value p calling environment - looping value take three values. name third argument, use special interpolation format ${p} substitute value p build filename.","code":"B(p = \"x\") -- C(p = \"x\")   /                          \\ A - B(p = \"y\") -- C(p = \"y\") - D   \\                          /     B(p = \"z\") -- C(p = \"z\") orderly2::orderly_dependency(\"C\", quote(latest(parameter:p == \"x\")),                              c(\"data/x.rds\" = \"result.rds\")) orderly2::orderly_dependency(\"C\", quote(latest(parameter:p == \"y\")),                              c(\"data/y.rds\" = \"result.rds\")) orderly2::orderly_dependency(\"C\", quote(latest(parameter:p == \"z\")),                              c(\"data/z.rds\" = \"result.rds\")) for (p in c(\"x\", \"y\", \"z\")) {   orderly2::orderly_dependency(\"C\", quote(latest(parameter:p == environment:p)),                                c(\"data/${p}.rds\" = \"result.rds\")) }"},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"how-dependencies-interact-with-locations","dir":"Articles","previous_headings":"","what":"How dependencies interact with locations","title":"Dependencies between packets","text":"default, packet unpacked local archive considered candidate inclusion orderly_dependency(). always want. locations selected, packets within considered candidates can controlled search_options argument orderly2::orderly_run (note argument orderly_run(), orderly_dependency() effect controlled runner report, writer report). three components affect packets selected location: character vector locations, matching location names. packets can found locations considered. mix locally created packets well ones people can see, specifying location = \"server\" limit packets available server, means end dependencies colleagues also get. allow_remote: controls willing download files location order satisfy dependency. TRUE, run report, might download files recent packets available location locally. pull_metadata: effect allow_remote also TRUE; causes metadata refreshed dependency resolution. discussion details ?orderly_run","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"other-points","dir":"Articles","previous_headings":"","what":"Other points","title":"Dependencies between packets","text":"used systems like targets, easy make reports smaller need . ’s real need make small, picking right size challenge. small, ’ll end writing lot code orchestrate running different reports pulling things together. ’ll end spending lot time whether things “date” one another really group things always wants run together. ’re big might end work want , order make change part piece analysis must run whole thing .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"installation","dir":"Articles","previous_headings":"","what":"Installation","title":"Introduction to orderly2","text":"","code":"install.packages(   \"orderly2\",   repos = c(\"https://mrc-ide.r-universe.dev\", \"https://cloud.r-project.org\"))"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"creating-an-empty-orderly-repository","dir":"Articles","previous_headings":"","what":"Creating an empty orderly repository","title":"Introduction to orderly2","text":"first step initialise empty orderly2 repository. orderly2 repository directory file orderly_config.yml within , since version 2 also directory .outpack/. Files within .outpack/ directory never directly modified users directory excluded version control (see orderly2::orderly_gitignore_update). Create orderly2 repository calling orderly2::orderly_init(): creates files: step performed completely empty directory, otherwise error thrown. Later, re-initialise orderly2 repository cloning new machine, working others; discussed vignette(\"collaboration\"). orderly_config.yml file contains little default: vignette, created orderly root R’s per-session temporary directory, deleted R exits. want use directory persist across restarting R (certainly want using orderly2 real project!) replace path within home directory, location control. rest vignette evaluate commands within directory, changing directory path ’ve created:","code":"path <- tempfile() # we'll use a temporary directory here - see note below orderly2::orderly_init(path) ## ✔ Created orderly root at '/tmp/RtmpqAQMea/file1af65b7c91e8' ## . ## ├── .outpack ## │   ├── config.json ## │   ├── location ## │   └── metadata ## └── orderly_config.yml minimum_orderly_version: \"1.99.0\" setwd(path)"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"creating-your-first-orderly-report","dir":"Articles","previous_headings":"","what":"Creating your first orderly report","title":"Introduction to orderly2","text":"orderly report directory src/<name> containing orderly file <name>.R. file may special commands , now ’ll create one simple possible; ’ll create random data save disk. seems silly, imagine standing something like: downloading file external site resource running simulation saving output fitting model data merging set files together create final data set directory structure (ignoring .outpack) looks like: src/incoming_data/incoming_data.R contains: run report create new packet, use orderly2::orderly_run(): id created new identifier packet unique among packets (within reason) chronologically sortable. packet id sorts another packet’s id started packet. run report, directory structure looks like: things changed : file created ran report (data.rds; see script ) log happened report run packet created incoming_data.R data.csv, original input come source tree empty directory draft/incoming_data created orderly ran report first place addition, quite files changed within .outpack directory, covered . ’s ! Notice initial script just plain R script, can develop interactively within src/incoming_data directory. Note however, paths referred within relative src/incoming_data orderly repository root. important reports see world relative incoming_data.R file. created, can refer report id pull files wherever need , context another orderly report just copy desktop email someone. example, copy file data.rds created location outside orderly’s control copies data.rds new temporary directory dest name final.rds. uses orderly2’s outpack_ functions, designed interact outpack archives regardless created (orderly2 program creates outpack archives). Typically lower-level orderly_ functions.","code":"## . ## ├── orderly_config.yml ## └── src ##     └── incoming_data ##         ├── data.csv ##         └── incoming_data.R d <- read.csv(\"data.csv\") d$z <- resid(lm(y ~ x, d)) saveRDS(d, \"data.rds\") id <- orderly2::orderly_run(\"incoming_data\") ## ℹ Starting packet 'incoming_data' `20240905-080228-ca5e5af6` at 2024-09-05 08:02:28.794513 ## > d <- read.csv(\"data.csv\") ## > d$z <- resid(lm(y ~ x, d)) ## > saveRDS(d, \"data.rds\") ## ✔ Finished running incoming_data.R ## ℹ Finished 20240905-080228-ca5e5af6 at 2024-09-05 08:02:28.855893 (0.06137991 secs) id ## [1] \"20240905-080228-ca5e5af6\" ## . ## ├── archive ## │   └── incoming_data ## │       └── 20240905-080228-ca5e5af6 ## │           ├── data.csv ## │           ├── data.rds ## │           └── incoming_data.R ## ├── draft ## │   └── incoming_data ## ├── orderly_config.yml ## └── src ##     └── incoming_data ##         ├── data.csv ##         └── incoming_data.R dest <- tempfile() fs::dir_create(dest) orderly2::orderly_copy_files(id, files = c(\"final.rds\" = \"data.rds\"),                              dest = dest)"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"depending-on-packets-from-another-report","dir":"Articles","previous_headings":"","what":"Depending on packets from another report","title":"Introduction to orderly2","text":"Creating new dataset mostly useful someone else can use . introduce first special orderly commands can use orderly file src/ directory now looks like: src/analysis/analysis.R contains: , ’ve used orderly2::orderly_dependency() pull file data.rds recent version (latest()) data packet filename incoming.rds, ’ve used file normal make plot, ’ve saved analysis.png. can run just , using orderly2::orderly_run(): information dependencies, see vignette(\"dependencies\").","code":"## src ## ├── analysis ## │   └── analysis.R ## └── incoming_data ##     ├── data.csv ##     └── incoming_data.R orderly2::orderly_dependency(\"incoming_data\", \"latest()\",                              c(\"incoming.rds\" = \"data.rds\")) d <- readRDS(\"incoming.rds\") png(\"analysis.png\") plot(y ~ x, d) dev.off() id <- orderly2::orderly_run(\"analysis\") ## ℹ Starting packet 'analysis' `20240905-080229-4bf6295e` at 2024-09-05 08:02:29.299062 ## > orderly2::orderly_dependency(\"incoming_data\", \"latest()\", ## +                              c(\"incoming.rds\" = \"data.rds\")) ## ℹ Depending on incoming_data @ `20240905-080228-ca5e5af6` (via latest(name == \"incoming_data\")) ## > d <- readRDS(\"incoming.rds\") ## > png(\"analysis.png\") ## > plot(y ~ x, d) ## > dev.off() ## agg_png  ##       2 ## ✔ Finished running analysis.R ## ℹ Finished 20240905-080229-4bf6295e at 2024-09-05 08:02:29.375767 (0.07670498 secs)"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"available-in-report-orderly-commands","dir":"Articles","previous_headings":"","what":"Available in-report orderly commands","title":"Introduction to orderly2","text":"function orderly2::orderly_dependency() designed operate packet runs. functions act adding metadata final packet, perhaps copying files directory. orderly2::orderly_description(): Provide longer name description report; can reflected tooling uses orderly metadata much informative short name. orderly2::orderly_parameters(): Declares parameters can passed control behaviour report. Parameters key-value pairs simple data (booleans, numbers, strings) report can respond . can also used queries orderly2::orderly_dependency() find packets satisfy criteria. orderly2::orderly_resource(): Declares file resource; file input report, comes source directory. default, orderly treats files directory resource, can useful mark explicitly, necessary “strict mode” (see ). Files marked resource immutable may deleted modified. orderly2::orderly_shared_resource(): Copies file “shared resources” directory shared/, can data files source code located root orderly repository. can reasonable way sharing data commonly used code among several reports. orderly2::orderly_artefact(): Declares file (set files) created report, even run. makes easier check report behaves expected can allow reasoning related set reports without running . declaring something artefact (especially conjunction “strict mode”) also easier clean src directories used interactive development (see ). orderly2::orderly_dependency(): Copy files one packet packet runs, seen . orderly2::orderly_strict_mode(): Declares report run “strict mode” (see ). addition, also function orderly::orderly_run_info() can used running report returns information currently running report (id, resolved dependencies etc). Let’s add additional annotations previous reports: , ’ve added block special orderly commands; go anywhere, example files refer . strict mode enabled (see ) orderly2::orderly_resource calls must go files used made available point (see ).","code":"orderly2::orderly_strict_mode() orderly2::orderly_resource(\"data.csv\") orderly2::orderly_artefact(\"Processed data\", \"data.rds\")  d <- read.csv(\"data.csv\") d$z <- resid(lm(y ~ x, d)) saveRDS(d, \"data.rds\") id <- orderly2::orderly_run(\"incoming_data\") ## ℹ Starting packet 'incoming_data' `20240905-080229-95242471` at 2024-09-05 08:02:29.584198 ## > orderly2::orderly_strict_mode() ## > orderly2::orderly_resource(\"data.csv\") ## > orderly2::orderly_artefact(\"Processed data\", \"data.rds\") ## Warning: Please use a named argument for the description in 'orderly_artefact()' ## In future versions of orderly, we will change the order of the arguments to ## 'orderly_artefact()' so that 'files' comes first. If you name your calls to ## 'description' then you will be compatible when we make this change. ## > d <- read.csv(\"data.csv\") ## > d$z <- resid(lm(y ~ x, d)) ## > saveRDS(d, \"data.rds\") ## ✔ Finished running incoming_data.R ## ! 1 warning found: ## • Please use a named argument for the description in 'orderly_artefact()' In ## future versions of orderly, we will change the order of the arguments to ## 'orderly_artefact()' so that 'files' comes first. If you name your calls to ## 'description' then you will be compatible when we make this change. ## ℹ Finished 20240905-080229-95242471 at 2024-09-05 08:02:29.626847 (0.04264855 secs)"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"parameterised-reports","dir":"Articles","previous_headings":"","what":"Parameterised reports","title":"Introduction to orderly2","text":"Much flexibility comes orderly graph comes using parameterised reports; reports take set parameters change behaviour based parameters. Downstream reports can depend parameterised report filter based suitable parameters. example, consider simple report generate samples based parameter: creates report single parameter n_samples default value 10. used define parameter default, defined multiple parameters can anything report switches value parameter: might read different URLs fetch different underlying data might fit different analysis might read different shared resources (see ) might depend different dependencies might produce different artefacts However, see parameters relatively heavyweight things try consistent set packets created report. report use control size generated data set. resulting file 15 rows, parameter passed affected report: can use parameters orderly’s search functions. example can find recent version packet running: can also pass parameter queries : can used within orderly2::orderly_dependency() (name == \"random\" part implied first name argument), example case report querying also parameters can use within query, using prefix. suppose downstream report simply uses n number samples might write: depend recent packet called random parameter n_samples value current report’s parameter n. See outpack query documentation much detail .","code":"orderly2::orderly_parameters(n_samples = 10) x <- seq_len(n_samples) d <- data.frame(x = x, y = x + rnorm(n_samples)) saveRDS(d, \"data.rds\") orderly2::orderly_parameters(n_samples = NULL) orderly2::orderly_parameters(n_samples = 10, distribution = \"normal\") id <- orderly2::orderly_run(\"random\", list(n_samples = 15)) ## ℹ Starting packet 'random' `20240905-080229-d73f20b5` at 2024-09-05 08:02:29.843275 ## ℹ Parameters: ## • n_samples: 15 ## > orderly2::orderly_parameters(n_samples = 10) ## > x <- seq_len(n_samples) ## > d <- data.frame(x = x, y = x + rnorm(n_samples)) ## > saveRDS(d, \"data.rds\") ## ✔ Finished running random.R ## ℹ Finished 20240905-080229-d73f20b5 at 2024-09-05 08:02:29.868695 (0.02541995 secs) orderly2::orderly_copy_files(id, files = c(\"random.rds\" = \"data.rds\"),                              dest = dest) readRDS(file.path(dest, \"random.rds\")) ##     x          y ## 1   1  0.4463006 ## 2   2  2.6289820 ## 3   3  5.0650249 ## 4   4  2.3690106 ## 5   5  5.5124269 ## 6   6  4.1369885 ## 7   7  6.4779875 ## 8   8  7.9473981 ## 9   9  9.5429963 ## 10 10  9.0859252 ## 11 11 11.4681544 ## 12 12 12.3629513 ## 13 13 11.6954565 ## 14 14 14.7377763 ## 15 15 16.8885049 orderly2::orderly_search('latest(name == \"random\")') ## [1] \"20240905-080229-d73f20b5\" orderly2::orderly_search('latest(name == \"random\" && parameter:n_samples > 10)') ## [1] \"20240905-080229-d73f20b5\" orderly2::orderly_dependency(\"random\", \"latest(parameter:n_samples > 10)\",                              c(\"randm.rds\" = \"data.rds\")) orderly2::orderly_dependency(\"random\", \"latest(parameter:n_samples == this:n)\",                              c(\"randm.rds\" = \"data.rds\"))"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"shared-resources","dir":"Articles","previous_headings":"","what":"Shared resources","title":"Introduction to orderly2","text":"Sometimes useful share data different reports, example common source utilities don’t warrant package, common data. , create directory shared orderly root put files directories might want share. Suppose shared directory contains file data.csv: can write orderly report use_shared uses shared file, use_shared.R containing: can run : resulting archive, file used shared directory present: general property orderly: tries save inputs alongside final results analysis, later can check see went analysis might changed versions.","code":"## . ## ├── archive ## │   ├── analysis ## │   │   └── 20240905-080229-4bf6295e ## │   │       ├── analysis.R ## │   │       ├── analysis.png ## │   │       └── incoming.rds ## │   ├── incoming_data ## │   │   ├── 20240905-080228-ca5e5af6 ## │   │   │   ├── data.csv ## │   │   │   ├── data.rds ## │   │   │   └── incoming_data.R ## │   │   └── 20240905-080229-95242471 ## │   │       ├── data.csv ## │   │       ├── data.rds ## │   │       └── incoming_data.R ## │   └── random ## │       └── 20240905-080229-d73f20b5 ## │           ├── data.rds ## │           └── random.R ## ├── draft ## │   ├── analysis ## │   ├── incoming_data ## │   └── random ## ├── orderly_config.yml ## ├── shared ## │   └── data.csv ## └── src ##     ├── analysis ##     │   └── analysis.R ##     ├── incoming_data ##     │   ├── data.csv ##     │   └── incoming_data.R ##     └── random ##         └── random.R orderly2::orderly_shared_resource(\"data.csv\") orderly2::orderly_artefact(\"analysis\", \"analysis.png\")  d <- read.csv(\"data.csv\") png(\"analysis.png\") plot(y ~ x, d) dev.off() id <- orderly2::orderly_run(\"use_shared\") ## ℹ Starting packet 'use_shared' `20240905-080230-53670258` at 2024-09-05 08:02:30.328211 ## > orderly2::orderly_shared_resource(\"data.csv\") ## > orderly2::orderly_artefact(\"analysis\", \"analysis.png\") ## Warning: Please use a named argument for the description in 'orderly_artefact()' ## In future versions of orderly, we will change the order of the arguments to ## 'orderly_artefact()' so that 'files' comes first. If you name your calls to ## 'description' then you will be compatible when we make this change. ## > d <- read.csv(\"data.csv\") ## > png(\"analysis.png\") ## > plot(y ~ x, d) ## > dev.off() ## agg_png  ##       2 ## ✔ Finished running use_shared.R ## ! 1 warning found: ## • Please use a named argument for the description in 'orderly_artefact()' In ## future versions of orderly, we will change the order of the arguments to ## 'orderly_artefact()' so that 'files' comes first. If you name your calls to ## 'description' then you will be compatible when we make this change. ## ℹ Finished 20240905-080230-53670258 at 2024-09-05 08:02:30.372285 (0.04407382 secs) ## archive/use_shared ## └── 20240905-080230-53670258 ##     ├── analysis.png ##     ├── data.csv ##     └── use_shared.R"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"strict-mode","dir":"Articles","previous_headings":"","what":"Strict mode","title":"Introduction to orderly2","text":"previous version orderly (orderly1; see vignette(\"migrating\")) fussy input strictly declared report run, clear really required order run something. version 2 relaxed default, can opt old behaviours checks adding anywhere within orderly file (conventionally top). may make granular future, adding : copy files source directory (src/<reportname>/) draft directory report runs (draft/<reportname>/<packet-id>) declared orderly2::orderly_resource; leaves behind extra files left development warn end running packet files found part artefact Using strict mode also helps orderly2 clean src/<reportname> directory effectively interactive development (see next section).","code":"orderly2::orderly_strict_mode()"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"interactive-development","dir":"Articles","previous_headings":"","what":"Interactive development","title":"Introduction to orderly2","text":"Set working directory src/<reportname> orderly script fully executable (e.g., source Rstudio’s Source button, R’s source() function). Dependencies copied needed. , mix files within source directory. recommend per-source-directory .gitignore keep files version control (see ). soon implement support cleaning generated files directory. example, suppose interactively run incoming_data/incoming_data.R script, leave behind generated files. can report orderly2::orderly_cleanup_status: files unknown orderly tell prompt tell explicitly. can clean generated files running (suggested message): dry_run = TRUE argument can pass want see deleted without using status function. can also keep files git using orderly2::orderly_gitignore_update function: creates (updates) .gitignore file within report generated files included git. already accidentally committed gitignore real effect git surgery, see git manuals handy, profane, guide.","code":"orderly2::orderly_cleanup_status(\"incoming_data\") ## ✖ incoming_data is not clean: ## ℹ 1 file can be deleted by running 'orderly2::orderly_cleanup(\"incoming_data\")': ##   • data.rds orderly2::orderly_cleanup(\"incoming_data\") ## ℹ Deleting 1 file from 'incoming_data': ## • data.rds orderly2::orderly_gitignore_update(\"incoming_data\") ## ✔ Wrote 'src/incoming_data/.gitignore'"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"deleting-things-from-the-archive","dir":"Articles","previous_headings":"","what":"Deleting things from the archive","title":"Introduction to orderly2","text":"delete packets archive/ directory puts orderly2 inconsistent state metadata store. Sometimes matter (e.g., delete old copies never candidates inclusion orderly2::orderly_dependency never notice). However, delete recent copy packet try depend , get error. moment, two copies incoming_data task: run analysis task, pull recent version (20240905-080229-95242471). However, deleted manually (e.g., save space accidentally) corrupted (e.g., opening output Excel letting save changes) able included, running analysis fail: error tries fairly informative, telling us failed copying files 20240905-080229-95242471 found packet corrupt, file data.rds found archive. also suggests fix; can tell orderly2 20240905-080229-95242471 “orphaned” considered inclusion look dependencies. can carry suggestion just validate packet running can validate packets : option core.require_complete_tree enabled, process also look packets used now-deleted packet orphan , longer complete tree includes . want remove references orphaned packets, can use orderly2::orderly_prune_orphans() remove entirely:","code":"orderly2::orderly_metadata_extract(   name = \"incoming_data\",   extract = c(time = \"time.start\")) ##                         id                time ## 1 20240905-080228-ca5e5af6 2024-09-05 08:02:28 ## 2 20240905-080229-95242471 2024-09-05 08:02:29 orderly2::orderly_run(\"analysis\") ## ℹ Starting packet 'analysis' `20240905-080230-f22cf381` at 2024-09-05 08:02:30.948278 ## > orderly2::orderly_dependency(\"incoming_data\", \"latest()\", ## +                              c(\"incoming.rds\" = \"data.rds\")) ## ✖ Error running analysis.R ## ℹ Finished 20240905-080230-f22cf381 at 2024-09-05 08:02:31.020346 (0.07206869 secs) ## Error in `orderly2::orderly_run()`: ## ! Failed to run report ## Caused by error in `orderly_copy_files()`: ## ! Unable to copy files, due to deleted packet 20240905-080229-95242471 ## ℹ Consider 'orderly2::orderly_validate_archive(\"20240905-080229-95242471\", ##   action = \"orphan\")' to remove this packet from consideration ## Caused by error: ## ! File not found in archive ## ✖ data.rds orderly2::orderly_validate_archive(\"20240905-080229-95242471\", action = \"orphan\") orderly2::orderly_validate_archive(action = \"orphan\") ## ✔ 20240905-080228-ca5e5af6 (incoming_data) is valid ## ✔ 20240905-080229-4bf6295e (analysis) is valid ## ✖ 20240905-080229-95242471 (incoming_data) is invalid due to its files ## ✔ 20240905-080229-d73f20b5 (random) is valid ## ✔ 20240905-080230-53670258 (use_shared) is valid orderly2::orderly_prune_orphans() ## ℹ Pruning 1 orphan packet"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"debugging-and-coping-with-errors","dir":"Articles","previous_headings":"","what":"Debugging and coping with errors","title":"Introduction to orderly2","text":"(written)","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"interaction-with-version-control","dir":"Articles","previous_headings":"","what":"Interaction with version control","title":"Introduction to orderly2","text":"guidelines: Make sure exclude files git listing .gitignore: .outpack/ - nothing suitable version control archive/ - core.archive_path set non-null value, excluded. default archive draft/ - temporary draft directory orderly_envir.yml - used setting machine-specific configuration absolutely version control files: src/ main source analyses orderly_config.yml - high level configuration suitable sharing shared resource directory (configured orderly_config.yml) probably version controlled source repository end multiple people’s machines, configured differently. configuration option set via orderly2::orderly_config_set designed (potentially) different different users, configuration needs version controlled. also means reports/packets can’t directly refer values set . includes directory used save archive packets (enabled) names locations (equivalent git remotes). may find useful include scripts help users set common locations, like git, different users may use different names remote (e.g., one user may location called data another called data-incoming, depending perspective use location). orderly2 always try save information current state git source repository alongside packet metadata. includes current branch, commit (sha) remote url. try create links final version packet upstream source repository.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"interaction-with-the-outpack-store","dir":"Articles","previous_headings":"","what":"Interaction with the outpack store","title":"Introduction to orderly2","text":"alluded , .outpack directory contains lots information packets run, typically “bounds” normal use. effectively “database” information packets run. Understanding directory structured required using orderly, included avoidance mystery! See outpack documentation (vignette(\"outpack\")) details ideas . work , directory structure looks like: can perhaps inferred filenames, files .outpack/metadata/<packet-id> metadata packet run. files .outpack/location/<location-id>/<packet-id> holds information packet first known location (location special “local” location). default orderly configuration store final files directory called archive/, alternatively (additionally) can use content- addressable file store. enabled, .outpack directory looks like: files .outpack/files/ never modified deleted. approach storage naturally deduplicates file archive, large file used many places ever stored .","code":"## .outpack ## ├── config.json ## ├── index ## │   └── outpack.rds ## ├── location ## │   ├── local ## │   │   ├── 20240905-080228-ca5e5af6 ## │   │   ├── 20240905-080229-4bf6295e ## │   │   ├── 20240905-080229-d73f20b5 ## │   │   └── 20240905-080230-53670258 ## │   └── orphan ## └── metadata ##     ├── 20240905-080228-ca5e5af6 ##     ├── 20240905-080229-4bf6295e ##     ├── 20240905-080229-d73f20b5 ##     └── 20240905-080230-53670258 ## .outpack ## ├── config.json ## ├── files ## │   └── sha256 ## │       ├── 0a ## │       │   └── a82571c21c4e5f1f435e8bef2328dda5ef47e177d78d63d1c4ec647a5a388a ## │       ├── 25 ## │       │   └── 4947c281b203719c72949745123a1d017e2f9b50c048b1d24a0803d73ba0b8 ## │       ├── 40 ## │       │   └── f8c6c3993686ee8acf38ce936911d6f65372d489e78748ec6d1f27d277daa7 ## │       ├── 42 ## │       │   └── 438203a847c7cd496e3e603ddeeceb4e136f1bd18f1efb81332fa093964f7b ## │       ├── 51 ## │       │   └── f8070f04f57a8040ad215474b976bcddc891bfc544e6c6a0519e74d79cb104 ## │       ├── 5f ## │       │   └── 96f49230c2791c05706f24cb2335cd0fad5d3625dc6bca124c44a51857f3f8 ## │       ├── a6 ## │       │   └── 80ab7c65a52327a3d9c5499d114f513f18eabe7f63a98f9fc308c2b3744c82 ## │       ├── d9 ## │       │   └── 1699ae410cbd811e1f028f8a732e5162b7df854eec08d921141f965851272d ## │       ├── de ## │       │   └── 1329bc0e9f8ee0c8b3376eb09b71e93587f707dfeaa01d1a03ae32f97c928a ## │       └── ec ## │           └── b53285781a4d36c65168c80ee14f2af2c885423c6166b9425f40c3c6cd8297 ## ├── index ## │   └── outpack.rds ## ├── location ## │   ├── local ## │   │   ├── 20240905-080228-ca5e5af6 ## │   │   ├── 20240905-080229-4bf6295e ## │   │   ├── 20240905-080229-d73f20b5 ## │   │   └── 20240905-080230-53670258 ## │   └── orphan ## └── metadata ##     ├── 20240905-080228-ca5e5af6 ##     ├── 20240905-080229-4bf6295e ##     ├── 20240905-080229-d73f20b5 ##     └── 20240905-080230-53670258"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"relationship-between-orderly-and-outpack","dir":"Articles","previous_headings":"","what":"Relationship between orderly and outpack","title":"Introduction to orderly2","text":"orderly2 package built metadata file storage system called outpack; implementing support working metadata archives languages (see outpack_server server implementation Rust outpack-py Python). metadata discussed detail vignette(\"metadata\") document general ideas fully mrc-ide/outpack","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"basic-overview","dir":"Articles","previous_headings":"","what":"Basic overview","title":"Outpack metadata","text":"“packet” conceptually directory, corresponding particular analysis data product, though necessarily stored. internal representation includes: metadata packet - files contains hashes along information done create environment created etc (see ) actual file contents every included file. might stored simple content-addressable file storage system (see ) actual copies directories originally created Every packet referenced uniquely primary key. use key format encodes current date time, well random data avoid collisions. exists dependency graph among packets, one packet depends another. edge graph hard link (one packet another id) also query (e.g., latest packet name) used find packet. means many ways looking thinking dependency graph. packets available locally, outpack repositories, typically (always) machines accessed HTTP API. conceptually similar git “remotes”. need distinguish packets “unpacked” (, packets every file available current archive) packets merely known (metadata files). sometimes refer unpacked packets “local” known “local” location special. use terms “archive” “repository” fairly interchangeably try nail .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"what-goes-into-a-packet","dir":"Articles","previous_headings":"","what":"What goes into a packet","title":"Outpack metadata","text":"packet must things: name (example model_fits). changed (rather changes tracked) currently way namespacing different repositories id (see “outpack id” ) created outpack point packet started set parameters - mapping strings scalar type (boolean, number, string). Packets may zero many parameters. set files - local (relative) path within packet (e.g., output/data.csv) also hash (e.g., sha256:69f6cf230416cf40828da251a0dad17cbbf078587883e826f3345ff08d1aaa7d) Information packet run (start end time), represented seconds post epoch, UTC (see ). addition may contain information : dependencies used, tracking name included packet, query used find , resolved id, set files extracted packet (source name, hash destination name) names scripts run create packet (typically also included files within final packet) information state git point packet run (current branch, hash, etc) information system used run packet (format unspecified might include hostname, operating system version, package/module versions, etc) custom metadata added implementation","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"types-of-users-of-outpack","dir":"Articles","previous_headings":"","what":"Types of users of outpack","title":"Outpack metadata","text":"types “persona” outpack user imagine exist guide decisions abut layout . extremes : user desktop wants develop run analyses, using files present packets available server. want visually inspect results running packets, need see files disk way makes sense human, care complete copy dependencies graph. server instance responsible delivering packets users (e.g., persona ). expected (able fetch) part outpack graph, expect anyone want visually inspect packets directly machine (, storage need human-readable). impacts two configuration options associated parts directory structure : user use content-addressable “file store” human-readable “archive” (explained detail ) require archive contains complete tree dependencies? expect first persona wants human readable archive contain full tree, second wants opposite.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"directory-layout","dir":"Articles","previous_headings":"","what":"Directory layout","title":"Outpack metadata","text":"section discusses files directory make outpack work, much come ; see . typical .outpack directory layout looks like : (note archive/ .outpack level). directories necessarily present; indeed required file .outpack/config.json.","code":".outpack/   config.json   files/   location/   metadata/ archive/"},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"configuration--outpackconfig-json","dir":"Articles","previous_headings":"Directory layout","what":"Configuration (.outpack/config.json)","title":"Outpack metadata","text":"outpack configuration schema defined config.json configuration format still subject change…","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"packet-metadata--outpackmetadata","dir":"Articles","previous_headings":"Directory layout","what":"Packet metadata (.outpack/metadata/)","title":"Outpack metadata","text":"file within directory filename outpack id (matching regular expression ^[0-9]{8}-[0-9]{6}-[0-9a-f]{8}$, see . file json file conforming schema metadata.json. present means outpack implementation can report information back packet (created, files contains, etc), packet metadata meaningful ; want know might come (location distributing packet) copy packet locally.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"location-information--outpacklocation","dir":"Articles","previous_headings":"Directory layout","what":"Location information (.outpack/location/)","title":"Outpack metadata","text":"directory matches regular expression ^[0-9]{8}$ (e.g., 457f4f2a) “location id” (see ) corresponding “location”. file within directory outpack id name, contains json location unpacked (installed) packet, hash metadata. file conforms schema location.json.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"a-file-store--outpackfiles","dir":"Articles","previous_headings":"Directory layout","what":"A file store (.outpack/files)","title":"Outpack metadata","text":"configuration option core.use_file_store true, outpack keeps content addressable file store files knows . much space efficient entire packet unpacked automatically deduplicates shared content among packets (e.g., large file present two packets stored ). file store layout described . storage format human-readable (indeed present within hidden directory .outpack). can enabled either server user","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"an-archive-archive-by-default","dir":"Articles","previous_headings":"Directory layout","what":"An archive (archive/ by default)","title":"Outpack metadata","text":"configuration option core.path_archive non-null directory path containing unpacked packets. packet available path <name> “name” packet, <id> outpack id. several files per packet, possibly directories. storage approach designed human readable, typically enabled outpack repository used laptop user wants interactively work files.","code":"archive/<name>/<id>/<files...>"},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"adding-a-packet","dir":"Articles","previous_headings":"","what":"Adding a packet","title":"Outpack metadata","text":"order make packet available locally, need import metadata files, mark packet available. roughly creating packet (.e., first place packet ever existed) importing packet elsewhere. Making packet available allows used dependency, allows serving packet acting location (http file protocols), guarantees files actually present locally.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"adding-metadata","dir":"Articles","previous_headings":"Adding a packet","what":"Adding metadata","title":"Outpack metadata","text":"can simply copy metadata file .outpack/metadata/<packet id> yet exist. make available anything yet known location. Dangling metadata (, metadata present directory known anywhere) currently mostly ignored.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"adding-files","dir":"Articles","previous_headings":"Adding a packet","what":"Adding files","title":"Outpack metadata","text":"repository uses file store, fill first, much easier think . can easily get difference files used packet (list files packet manifest) already file store looking hash turn. request missing files insert store. may leave “dangling” files (files referred packet) problem. repository human-readable archive uses file store, files present file store easy enough check file store requested path (local relative path packet manifest). update file store first, files guaranteed present. repository uses human readable archive, simplest thing request file remote. However, might efficient check locally previously fetched copies files content, verify modified, copy place rather re-downloading.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"marking-the-packet-as-unpacked-and-known-locally","dir":"Articles","previous_headings":"Adding a packet","what":"Marking the packet as unpacked and known locally","title":"Outpack metadata","text":"local location id, write file conforming location.json schema, containing packet id, time marked unpacked hash metadata.","code":".outpack/<local location id>/<packet id>"},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"ordering-of-operations","dir":"Articles","previous_headings":"Adding a packet","what":"Ordering of operations","title":"Outpack metadata","text":"need files metadata packet marked unpacked; note configurations guarantee every packet unpacked complete tree. can import files first, metadata; lot disadvantage either. mark package unpacked known locally though components present.","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"outpack-ids","dir":"Articles","previous_headings":"Details","what":"Outpack ids","title":"Outpack metadata","text":"Outpack ids match regular expression ^[0-9]{8}-[0-9]{6}-[0-9a-f]{8}$; encode UTC date-time prefix YYYYMMDD-HHMMSS- followed 8 hexadecimal digits. R implementation, encode current second first four digits (2 bytes) append 2 bytes cryptographically random data. id tries balance reasonable degree collision resistance (65536 combinations per millisecond), lexicographic sortability reasonable degree meaningfulness.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"location-ids","dir":"Articles","previous_headings":"Details","what":"Location ids","title":"Outpack metadata","text":"Location ids meaningless 4-byte (8 character) hex strings. immutable created different different machines even point location. location id mapped (via .outpack/config.json) location name human-readable name (e.g., production staging). requirement name different machines. One directories represents local location; can find mapping within configuration.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"representing-hashes","dir":"Articles","previous_headings":"Details","what":"Representing hashes","title":"Outpack metadata","text":"Outpack typically uses sha256 hashes, want able change future. wherever hash presented, algorithm included part string. example instead used md5 algorithm written schema currently assumes hash value represented hex string.","code":"sha256:69f6cf230416cf40828da251a0dad17cbbf078587883e826f3345ff08d1aaa7d md5:bd57f7123c6bfb95c3234ff56373b7f4"},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"times","dir":"Articles","previous_headings":"Details","what":"Times","title":"Outpack metadata","text":"store information times places (e.g., times packet run, imported, etc). Rather trying deal strings, always store time seconds since 1970-01-01 00:00.00 UTC (including fractional seconds, whatever accuracy system allows).","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"file-store","dir":"Articles","previous_headings":"Details","what":"File store","title":"Outpack metadata","text":"file store designed simple, sophisticated git, whose object store similar thing. general layout looks like: hopefully fairly obvious structure. Paths format: reason second level prevent performance degradation directories containing millions files, copying git. store designed cope different hashing algorithms, though R implementation outpack supports sha256 now. Multiple hashing algorithms supported hard linking content multiple places tree, might link ","code":".outpack/files   sha256/     5d/       dfaf1f4a2e15e8fe46dbed145bf2f84bba1b3367d0a56f73de08f8585dd153       ...     77/     ... <algorithm>/<first two bytes>/<remaining bytes> sha256/5d/dfaf1f4a2e15e8fe46dbed145bf2f84bba1b3367d0a56f73de08f8585dd153 md5/84/0bc6ad3ae479dccc1c49a1910b37bd"},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"so-long-yaml-and-thanks-for-all-the-whitespace-errors","dir":"Articles","previous_headings":"Summary of changes","what":"So long YAML and thanks for all the whitespace errors","title":"Migrating from orderly (1.x)","text":"obvious user-facing change (almost) YAML, definition inputs outputs report now defined within orderly file, <reportname>.R. orderly report previously orderly.yml file looked like end within orderly file looks like: think much clearer, comes documentation autocomplete support IDEs. fact, simple reports, special functions required, though ’ll find useful (see vignette(\"orderly2\")) specific changes: longer need list packages packages:, instead just use library() ordinary script. record state session regardless get record used longer need use sources: list scripts want source, instead use source() normal global_resources become orderly2::orderly_shared_resource (aren’t really global much shared). Note directory now always shared/ orderly root, may configure .","code":"parameters:   n_min:     default: 10 script: script.R source:   - functions.R resources:   - metadata.csv depends:   raw_data:     id: latest     use:       raw_data.csv: data.csv artefacts:   data:     description: Processed data     filenames: data.rds orderly2::orderly_parameters(n_min = 10) orderly2::orderly_dependency(\"raw_data\", \"latest\",                               files = c(\"raw_data.csv\" = \"data.csv\")) orderly2::orderly_resource(\"metadata.csv\") orderly2::orderly_artefact(\"Processed data\", \"data.rds\") source(\"functions.R\")"},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"implications","dir":"Articles","previous_headings":"Summary of changes > So long YAML and thanks for all the whitespace errors","what":"Implications","title":"Migrating from orderly (1.x)","text":"change widespread implications: can program things like dependencies, creating loop series parameter values, conditionally depending reports R script can basis orderly report, can add orderly functions annotating inputs outputs progressively","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"database-support-has-been-moved-into-a-plugin","dir":"Articles","previous_headings":"Summary of changes","what":"Database support has been moved into a plugin","title":"Migrating from orderly (1.x)","text":"version 1, built-support accessing data SQL databases; moved within orderly.db plugin. major features supported.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"no-more-commit","dir":"Articles","previous_headings":"Summary of changes","what":"No more commit","title":"Migrating from orderly (1.x)","text":"orderly2 longer requires separate orderly_commit() call orderly_run(); longer make distinction local draft archive packets. Instead, added finer-grained control dependencies resolved (locally, subset servers), generalises way draft/archive used practice. See ?orderly_run details dependencies resolved. implications deleting things; draft directory always easy target deletion, now deletion need tell orderly2 deleted things. See vignette(\"introduction\") details (section “Deleting things archive”).","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"no-more-testing-or-development-mode","dir":"Articles","previous_headings":"Summary of changes","what":"No more testing or development mode","title":"Migrating from orderly (1.x)","text":"two different, unsatisfactory, mechanisms developing orderly report: orderly_test_start (version 1.1.0) orderly_develop_start (1.1.0 onwards) worked initial setup, copying dependencies etc location work (new draft orderly_test_start source directory orderly_develop_start). orderly2 can just work directly within source directory, long working directory set src/<report-name>, orderly2 commands work expected. orderly1, need careful commit (git) results running analysis, encourage per-report .gitignore files help .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"new-language-agnostic-backend","dir":"Articles","previous_headings":"Summary of changes","what":"New, language-agnostic, backend","title":"Migrating from orderly (1.x)","text":"biggest change, perhaps least visible, orderly now built open spec outpack can implemented language. develop Python implementation , possibly languages. takes control metadata. split “orderly_” “outpack_” functions package, information see last section vignette(\"introduction\") also vignette(\"outpack\").","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"other-smaller-changes","dir":"Articles","previous_headings":"Summary of changes","what":"Other, smaller changes","title":"Migrating from orderly (1.x)","text":"“Global” resources become “shared” resources, always live shared/ orderly root (.e., longer configurable). reason want reports able run fairly independently orderly2 configuration (exception plugins). practice people really vary .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"what-is-missing-compared-with-orderly1","dir":"Articles","previous_headings":"","what":"What is missing compared with orderly1","title":"Migrating from orderly (1.x)","text":"changelog feature (implement something support ) extraction secrets vault (support ) automatic handling README files (may implement support )","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"how-to-migrate","dir":"Articles","previous_headings":"","what":"How to migrate","title":"Migrating from orderly (1.x)","text":"two parts migration: updating canonical copy orderly archive (ideally one ) updating source tree. steps done via outpack.orderly package. migrate archive first. every archive want retain (might archives stored locally, production servers staging servers). Archive migration happens place; , modify anything original location. archive old used old versions orderly1 possible process hiccups. Please let us know case. result process end new directory contains new archive conforming outpack spec containing orderly2 metadata. Next, migrate source tree. done place done fresh clone source git repository. report, examine orderly.yml files script files (often script.R), delete , write new orderly file adapt report work orderly2. possible perfect might need minor tweaking hopefully reasonable. One thing preserved (probably ) comments yaml often refer yaml formatting orderly1 features hopefully much problem. probably want manually tweak generated code anyway, take advantage new orderly2 features able compute dependencies. using OrderlyWeb, probably need pause migrating, replacement yet ready.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"what-about-the-packages","dir":"Articles","previous_headings":"","what":"What about the packages?","title":"Migrating from orderly (1.x)","text":"merge orderly2 orderly package, ready release can use . However, anticipate period coexistence legacy orderly1 systems develop orderly2. help small helper package orderly.helper can smooth namespace differences; may useful interact versions.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"what-were-the-problems-in-version-1","dir":"Articles","previous_headings":"","what":"What were the problems in version 1","title":"Migrating from orderly (1.x)","text":"YAML format inflexible, error prone users, leads duplication focussed around initial needs Vaccine Impact Modelling Consortium fairly easy get archive SQLite database inconsistent state (e.g., deleting moving files archive) SQLite database behaved poorly shared file systems","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/orderly2.html","id":"collaborative-analysis","dir":"Articles","previous_headings":"","what":"Collaborative analysis","title":"orderly2","text":"Many analyses involve single person single machine; case number workflow tools make orchestrating analysis easy. workflow model graph dependencies analysis, data flows. example might update data, whole pipeline rerun. update code forecast, forecasts report rerun. experience, model works well single-user setting falls collaborative setting, especially analysis partitioned person; Alice handling data pipeline, Bob running fits, Carol organising forecasts final report. context, changes upstream affect downstream analysis, require set care around integration might used version controlling source code. example, Alice dealing change incoming data format going break analysis time Bob trying get model fits working, Bob trying integrate code changes Alice’s new data. typically deal source code using branches within git; code Bob work branch isolated Alice’s changes. contexts like () data analysis products git. needed way versioning outputs step analysis controlling integrated subsequent analyses. Another way looking problem seek way making analysis composable way functions OOP achieve programs, way docker containerisation achieved deploying software. need way putting interfaces around pieces analysis allow people refer fetch somewhere run. conceptual pieces needed : way referring unambiguously (globally) piece analysis can depended upon, everyone can agree ’re talking piece analysis system storage results running analysis can shared among group system control integration pieces analysis takes place refer transportable unit analysis “packet”. conceptually directory files created running code, atomic unit work point view orderly. packet underlying source form, anyone can run. However, time people use pre-run packets collaborators run inputs onward analyses (see vignette(\"dependencies\") vignette(\"collaboration\") details).","code":"[raw data] -> [processed data] -> [model fits] -> [forecasts] -> [report]"},{"path":"https://mrc-ide.github.io/orderly2/articles/orderly2.html","id":"reproducible-analyses","dir":"Articles","previous_headings":"","what":"Reproducible analyses","title":"orderly2","text":"degree collaboration style requires reproducibility, several aspects . system describe , even though everyone can typically run step analysis, typically don’t. differs workflow tools, users may familiar .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/orderly2.html","id":"difference-from-a-workflow-system","dir":"Articles","previous_headings":"Reproducible analyses","what":"Difference from a workflow system","title":"orderly2","text":"Workflow systems hugely influential scientific computing, people co-opting build systems like make sophisticated systems designed parallel running large complex workflows nextflow. general approach define interdependencies among parts analysis, forming graph parts analysis track inputs outputs workflow. model computation lots good points: defines interface analysis allows (encourages) breaking monolithic analysis component pieces can reasoned allows high-level parallelism making obvious parts workflow can run concurrently tracking way data flows analysis, allows minimum amount recalculation done change, downstream parts triggered shared workspace online runner, allows degree collaboration long everyone happy working constantly changing set code analysis artefacts designed orderly2 working patterns suit . motivating reasons include: nodes computational graph expensive compute require exotic hardware. Workflows upstream data never settle, need know version data ends used particular analysis Workflows upstream analyses used many downstream analyses, upstream developer may know much downstream use Nondeterministic analyses, e.g., involving stochastic simulations, rerunning node expected return exact numerical results, can’t rely different users recovering results different occasions[*] cases missing piece need way versioning nodes within computational graph, shifting emphasis automatically rerunning portions graph tracking data flowed graph. turn shifts reproducibility emphasis “everyone run code get results” “everyone run code, instead work results”. familiar docker, approach similar working pre-built docker images, whereas workflow approach similar working directly Dockerfiles; many situations end result , approaches differ guarantees, computation happens, users refer versions. [*] discourage trying force determinism manually setting seeds, potential violate statistical properties random number streams, fragile best.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/orderly2.html","id":"what-is-reproducibility-anyway","dir":"Articles","previous_headings":"Reproducible analyses","what":"What is reproducibility anyway?","title":"orderly2","text":"Reproducibility means different things different people, even within narrow sense “rerun analysis retrieve results”. last decade, idea one able rerun piece analysis retrieve results slightly morphed one must rerun piece analysis. Similarly, emphasis utility reproducibility shifted authors able rerun work (confidence rerun ) hypothetical third party wanting rerun analysis. approach flips perspective around bit, based experiences collaborative research projects, draws (overly) ambitious aim : Can prove given set inputs produced given set outputs? quickly found impossible, provided systems place one satisfied statement given level trust system. piece analysis comes server primary way people run analyses web front-end (currently OrderlyWeb, soon Packit) know analysis run end--end modification orderly2 preserves inputs alongside outputs files present final packet files went analysis, recorded R package versions full set used. system naturally involves running multiple machines (typically analysts’ laptops, server perhaps HPC environment), way orderly2 treats paths, practically little problem getting analyses working multiple places, trivially satisfying typical reproducibility aim, even though people typically focussed . shift focus proved valuable. analysis run one occasion (e.g., regular reporting, simply updating figure final submission manuscript revision), outputs may change. Understanding changes happened important. orderly2 automatically saves lot metadata run easy find things might changed. , can start interrogating graph among packets find effect change ; find previously run packets pulled old version data set, used previous release package.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"the-basic-idea","dir":"Articles","previous_headings":"","what":"The basic idea","title":"Creating plugins","text":"plugin provided package, possibly thing package provides. plugin name must (currently) package name. functions package needs call orderly2::orderly_plugin orderly2::orderly_plugin_register create register plugin, respectively. make plugin available orderly project, two new bits configuration may present orderly_config.yml - one declares plugin used, configures plugin. use plugin individual report, functions plugin used, configure use plugin. Finally, can save information back final orderly2 metadata plugin . yaml-less design orderly2 (see vignette(\"migrating\") familiar orderly1), line plugin just package code fairly blurred, reasons writing plugin typically want make something easier reports, want action reflected orderly metadata.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"an-example","dir":"Articles","previous_headings":"","what":"An example","title":"Creating plugins","text":"example, ’ll implement stripped version database plugin inspired work (see `orderly.db fuller implementation). make work need functions: …process additional fields orderly_config.yml describe find database …can called orderly file access database …can add metadata final orderly metadata done ’ll start report side things, describing want happen, work implementation. directory structure minimal project orderly_config.yml file contains information shared possible uses plugin - case connection information database: plugin called example.db listed within plugins section, along configuration; case indicating path SQLite file can loaded . example.R file contains information use database specific report; case, making results query SELECT * mtcars cyl == 4 database available R object dat Normally, imagine calculation kept minimal purpose demonstration. implement need : create package write function handle configuration orderly_config.yml write function query() used example.R query ","code":"## . ## ├── orderly_config.yml ## └── src ##     └── example ##         └── example.R minimum_orderly_version: 1.99.0 plugins:   example.db:     path: /tmp/Rtmp01esxs/file1be86b7a0ff0 dat <- example.db::query(\"SELECT * FROM mtcars WHERE cyl == 4\") orderly2::orderly_artefact(\"Summary of data\", \"data.rds\")  saveRDS(summary(dat), \"data.rds\")"},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"create-a-tiny-package","dir":"Articles","previous_headings":"An example","what":"Create a tiny package","title":"Creating plugins","text":"lots package skeleton tools , favourite, usethis::create_package() probably reasonable job. thing package needs contain Imports: orderly2 DESCRIPTION field. simple package may structure like , DESCRIPTION file contains: NAMESPACE R/plugin.R files shown .","code":"## . ## ├── DESCRIPTION ## ├── NAMESPACE ## └── R ##     └── plugin.R Package: example.db Version: 0.0.1 License: CC0 Title: Orderly Database Example Plugin Description: Simple example of an orderly plugin. Authors@R: person('Orderly Authors', role = c('aut', 'cre'),     email = 'email@example.com') Imports: orderly2"},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"handle-the-configuration","dir":"Articles","previous_headings":"An example","what":"Handle the configuration","title":"Creating plugins","text":"required function plugin needs provide one process data orderly_config.yml. probably primarily concerned validation can fairly simple first, later ’ll expand report errors nicely: arguments data: deserialised section orderly_config.yml specific plugin filename: full path orderly_config.yml return value data argument auxiliary data added validation.","code":"db_config <- function(data, filename) {   data }"},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"evaluate-the-query","dir":"Articles","previous_headings":"An example","what":"Evaluate the query","title":"Creating plugins","text":"Finally, minimal example, need function actually query; example example.db::query: arguments whatever want user provide – nothing special orderly2. important function call orderly2::orderly_plugin_context returns information can use make plugin work. explained ?orderly2::orderly_plugin_context, example use just one element, config, configuration plugin (.e., return value function db_config); see orderly2::orderly_plugin_context context can accessed . last bit package code register plugin, calling orderly2::orderly_plugin_register within .onLoad() special R function called package loaded. means whenever packages loaded (regardless whether attached) register plugin. (important name argument matches package name, orderly2 trigger loading package based name configuration; may support multiple plugins within one package later.) Note query function appear within registration, just function read process configuration. final (minimal) package code : NAMESPACE file contains","code":"query <- function(sql) {   ctx <- orderly2::orderly_plugin_context(\"example.db\")   dbname <- ctx$config$path   con <- DBI::dbConnect(RSQLite::SQLite(), dbname)   on.exit(DBI::dbDisconnect(con))   DBI::dbGetQuery(con, sql) } .onLoad <- function(...) {   orderly2::orderly_plugin_register(     name = \"example.db\",     config = db_config) } db_config <- function(data, filename) {   data }  query <- function(sql) {   ctx <- orderly2::orderly_plugin_context(\"example.db\")   dbname <- ctx$config$path   con <- DBI::dbConnect(RSQLite::SQLite(), dbname)   on.exit(DBI::dbDisconnect(con))   DBI::dbGetQuery(con, sql) }  .onLoad <- function(...) {   orderly2::orderly_plugin_register(     name = \"example.db\",     config = db_config) } export(query)"},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"trying-it-out","dir":"Articles","previous_headings":"An example","what":"Trying it out","title":"Creating plugins","text":"order test package, needs loaded. can either installing package using pkgload::load_all() (may find pkgload::load_all(export_all = FALSE) gives reliable experience. Now, can run report:","code":"pkgload::load_all() ## ℹ Loading example.db orderly2::orderly_run(\"example\", root = path_root) ## ℹ Starting packet 'example' `20240905-080238-d2ef47f8` at 2024-09-05 08:02:38.827684 ## > dat <- example.db::query(\"SELECT * FROM mtcars WHERE cyl == 4\") ## > orderly2::orderly_artefact(\"Summary of data\", \"data.rds\") ## Warning: Please use a named argument for the description in 'orderly_artefact()' ## In future versions of orderly, we will change the order of the arguments to ## 'orderly_artefact()' so that 'files' comes first. If you name your calls to ## 'description' then you will be compatible when we make this change. ## > saveRDS(summary(dat), \"data.rds\") ## ✔ Finished running example.R ## ! 1 warning found: ## • Please use a named argument for the description in 'orderly_artefact()' In ## future versions of orderly, we will change the order of the arguments to ## 'orderly_artefact()' so that 'files' comes first. If you name your calls to ## 'description' then you will be compatible when we make this change. ## ℹ Finished 20240905-080238-d2ef47f8 at 2024-09-05 08:02:39.001563 (0.1738789 secs) ## [1] \"20240905-080238-d2ef47f8\""},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"making-the-plugin-more-robust","dir":"Articles","previous_headings":"","what":"Making the plugin more robust","title":"Creating plugins","text":"plugin fairly fragile validation input data orderly_config.yml orderly.yml. fairly annoying yaml incredibly flexible reporting back information user might gone wrong hard. case, expect single key-value pair orderly_config.yml key path value path SQLite database. can easily expand configuration function report better back user misconfigure plugin: acceptable job preventing poor input suggesting user might look within configuration fix . Note return configuration data , can augment (otherwise change) data need.","code":"db_config <- function(data, filename) {   if (!is.list(data) || is.null(names(data)) || length(data) == 0) {     stop(\"Expected a named list for orderly_config.yml:example.db\")   }   if (length(data$path) != 1 || !is.character(data$path)) {     stop(\"Expected a string for orderly_config.yml:example.db:path\")   }   if (!file.exists(data$path)) {     stop(sprintf(       \"The database '%s' does not exist (orderly_config:example.db:path)\",       data$path))   }   data }"},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"saving-metadata-about-what-the-plugin-did","dir":"Articles","previous_headings":"","what":"Saving metadata about what the plugin did","title":"Creating plugins","text":"Nothing plugin saved report metadata unless save . Partly orderly.yml, saved final directory, serves sort record. However, probably want know something data returned . example might want save query string later can query without read process orderly.yml file statistics size data (e.g., number rows returned, columns) perhaps summary content hash can see content changed different versions report save metadata, use function orderly2::orderly_plugin_add_metadata; takes arguments plugin name, string like structure saved metadata (’ll use query) whatever data want save: function otherwise minimal version . also need provide serialisation function ensure metadata saved expected. saved metadata key query, get list back element query unnamed list many elements query calls given report. , ensure everything except cols length 1 (everything) gets turned scalar (1 [1]) serialise jsonlite::toJSON auto_unbox TRUE. Taking step , can also specify schema metadata conform save file inst/schema.json within package (path within inst fine). Finally, can also add deserialiation hook convert loaded metadata nice data.frame: Now, register plugin, provide path schema, along serialisation deserialisation functions: Now, orderly metadata saved (just running script part report) validate output passed orderly2::orderly_plugin_add_metadata schema, jsonvalidate installed (currently requires development version) R option outpack.schema_validate set TRUE (e.g., running options(outpack.schema_validate = TRUE)). final package structure: DESCRIPTION file NAMESPACE unchanged , schema shown just . plugin.R file contains code collected : (code .R file package, across several).","code":"query <- function(sql) {   ctx <- orderly2::orderly_plugin_context(\"example.db\")   dbname <- ctx$config$path   con <- DBI::dbConnect(RSQLite::SQLite(), dbname)   on.exit(DBI::dbDisconnect(con))   d <- DBI::dbGetQuery(con, sql)   info <- list(sql = sql, rows = nrow(d), cols = names(d))   orderly2::orderly_plugin_add_metadata(\"example.db\", \"query\", info)   d } db_serialise <- function(data) {   for (i in seq_along(data$query)) {     # Always save cols as a vector, even if length 1:     data$query[[i]]$cols <- I(data$query[[i]]$cols)   }   jsonlite::toJSON(data$query, auto_unbox = TRUE) } {     \"$schema\": \"http://json-schema.org/draft-07/schema#\",      \"type\": \"array\",     \"items\": {         \"type\": \"object\",         \"properties\": {             \"sql\": {                 \"type\": \"string\"             },             \"rows\": {                 \"type\": \"number\"             },             \"cols\": {                 \"type\": \"array\",                 \"items\": {                     \"type\": \"string\"                 }             }         },         \"required\": [\"sql\", \"rows\", \"cols\"],         \"additionalProperties\": false     } } .onLoad <- function(...) {   orderly2::orderly_plugin_register(     name = \"example.db\",     config = db_config,     serialise = db_serialise,     deserialise = db_deserialise,     schema = \"schema.json\") } ## . ## ├── archive ## │   └── example ## │       └── 20240905-080238-d2ef47f8 ## │           ├── data.rds ## │           └── example.R ## ├── draft ## │   └── example ## ├── orderly_config.yml ## └── src ##     └── example ##         └── example.R db_config <- function(data, filename) {   if (!is.list(data) || is.null(names(data)) || length(data) == 0) {     stop(\"Expected a named list for orderly_config.yml:example.db\")   }   if (length(data$path) != 1 || !is.character(data$path)) {     stop(\"Expected a string for orderly_config.yml:example.db:path\")   }   if (!file.exists(data$path)) {     stop(sprintf(       \"The database '%s' does not exist (orderly_config:example.db:path)\",       data$path))   }   data }  query <- function(sql) {   ctx <- orderly2::orderly_plugin_context(\"example.db\")   dbname <- ctx$config$path   con <- DBI::dbConnect(RSQLite::SQLite(), dbname)   on.exit(DBI::dbDisconnect(con))   d <- DBI::dbGetQuery(con, sql)   info <- list(sql = sql, rows = nrow(d), cols = names(d))   orderly2::orderly_plugin_add_metadata(\"example.db\", \"query\", info)   d }  .onLoad <- function(...) {   orderly2::orderly_plugin_register(     name = \"example.db\",     config = db_config,     serialise = db_serialise,     deserialise = db_deserialise,     schema = \"schema.json\") } id <- orderly2::orderly_run(\"example\", root = path_root) ## ℹ Starting packet 'example' `20240905-080239-da473a20` at 2024-09-05 08:02:39.855157 ## > dat <- example.db::query(\"SELECT * FROM mtcars WHERE cyl == 4\") ## > orderly2::orderly_artefact(\"Summary of data\", \"data.rds\") ## Warning: Please use a named argument for the description in 'orderly_artefact()' ## In future versions of orderly, we will change the order of the arguments to ## 'orderly_artefact()' so that 'files' comes first. If you name your calls to ## 'description' then you will be compatible when we make this change. ## > saveRDS(summary(dat), \"data.rds\") ## ✔ Finished running example.R ## ! 1 warning found: ## • Please use a named argument for the description in 'orderly_artefact()' In ## future versions of orderly, we will change the order of the arguments to ## 'orderly_artefact()' so that 'files' comes first. If you name your calls to ## 'description' then you will be compatible when we make this change. ## ℹ Finished 20240905-080239-da473a20 at 2024-09-05 08:02:39.90698 (0.0518229 secs) meta <- orderly2::orderly_metadata(id, root = path_root) meta$custom$example.db ##                                   sql rows         cols ## 1 SELECT * FROM mtcars WHERE cyl == 4   11 mpg, cyl...."},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"potential-uses","dir":"Articles","previous_headings":"","what":"Potential uses","title":"Creating plugins","text":"need functionality similar example - pulling database functionality original version orderly something independent, turns useful fraction orderly use-cases. can imagine potential uses though, : Non-DBI-based database data extraction, customised routines pulling data database Download files shared location just use (e.g., SharePoint, OneDrive, AWS). orderly_config.yml contain account connection details orderly.yml contain mapping remote data/files local files. Rather writing environment , use path argument copy files correct place. Pull data web API just running follow basic pattern requiring configuration order able connect resource service, specification resources fetched, action actually fetch resource put place.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"structure-of-queries","dir":"Articles","previous_headings":"","what":"Structure of queries","title":"Orderly query DSL","text":"simple query finds recent packet; unlikely useful without scoping - see . complex queries expressed syntax valid R (also valid Julia close valid Python). complex query composed “tests” Every “test” uses boolean operator (<, >, <=, >=, ==, !=) left right hand side can one : lookup outpack metadata (parameter:x value parameter called x, name name packet, id id packet) lookup provided data pars (:x value pars$x) literal value (e.g., \"some_name\", 1, TRUE) Tests can grouped together (, !, &&, || might expect: parameter:x == 1 || parameter:x == 2 finds packets parameter x 1 2 name == \"data\" && parameter:x > 3 finds packets called “data” parameter x greater 3 (parameter:y == 2) && !(parameter:x == 1 || parameter:x == 2) finds parameter y 2 parameter x anything 1 2 (also written (parameter:y == 2) && (parameter:x != 1 && parameter:x != 2)) four functions latest(expr) finds latest packet satisfying expr - always returns length 1 character, NA_character_ suitable packet found. expr given latest packets returned. single(expr) like latest(expr) except error expr evaluate exactly one packet id usedby(expr, FALSE) expr either literal id expression returns 1 id. finds packets used generation packet id returned expr (see dependencies section details).","code":"latest() name == \"some_name\" parameter:x > 1"},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"special-simple-queries","dir":"Articles","previous_headings":"","what":"Special simple queries","title":"Orderly query DSL","text":"two shorthand queries: latest equivalent latest() (useful applied scope) string matching regular expression id (^([0-9]{8}-[0-9]{6}-[[:xdigit:]]{8})$) equivalent single(id == \"<id>\") \"<id>\" string provided)","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"scoping-queries","dir":"Articles","previous_headings":"","what":"Scoping queries","title":"Orderly query DSL","text":"WARNING: may remove Scoping queries can used shorthand filtering returned packets. future used reduce set packets searched speed query evaluation. join together main query (scope) && (expr), except expr call latest single. case combine latest((scope) && (expr)) single((scope) && (expr)). useful want limit search particular name location perform detailed search. example, query equivalent orderly2 uses functionality resolving dependencies orderly2::orderly_dependency.","code":"orderly_query(quote(parameter:x == 1), scope = quote(name == \"data\")) orderly_query(quote(parameter:x == 1 && name == \"data\"))"},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"scoping-on-name","dir":"Articles","previous_headings":"Scoping queries","what":"Scoping on name","title":"Orderly query DSL","text":"often users want scope name instead passing scope argument shorthand name argument usability. equivalent ","code":"orderly_query(quote(parameter:x == 1), name = \"data\") orderly_query(quote(parameter:x == 1), scope = quote(name == \"data\"))"},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"dependencies","dir":"Articles","previous_headings":"","what":"Dependencies","title":"Orderly query DSL","text":"2 packets, B depends output (.e. call (id_a, ...) running packet B) can draw .  equivalently say dependency B, B depends used B , B uses parent B, B child tree dependencies among packets might want search packets used another packet. can use query function usedby(id) list packets used id. search recursively packets used id parents parents’ parents . optional second arg immediate FALSE default, set TRUE search immediate (e.g. level 1) dependencies. able search dependencies like means packet structure like  want know id used C can find using orderly_search usedby can combined groupings scope: depth usedby recurse can controlled setting depth e.g. search just immediate parents C. depth can positive integer, default depth recurse finds parents. usedby can simplified using subqueries. Subqueries denoted curly braces {} can either named passed subquery arg can anonymous. query equivalent uses subquery C. two important things note usedby: query inside usedby search entire index, ignoring scope name parameters. want find packets used latest C. subquery C scoped return results. query inside usedby must return single result. ensure must either literal id, call latest call single well searching dependency tree using usedby can search uses function. setup reports , B C want know id C uses can find using uses usedby can combined search complex arrangements dependencies. something like  want search version E depends version used latest C can via searches tree C tree find version E. Note important added name == \"\" condition , missing usedby(latest(name == \"C\")) also return B single throw error multiple packets. can also search tree find D e.g. note E packet uses D need add name == \"E\" clause. can combine usedby uses complex searches, find D C run","code":"orderly_search(quote(usedby(latest(name == \"C\"))), name = \"A\") orderly_search(quote(usedby(latest(name == \"C\")) && parameter:year == 2022),                name = \"A\") orderly_search(quote(usedby(latest(name == \"C\"), depth = 1)), name = \"A\") orderly_search(quote(usedby({C}) && parameter:year == 2022),                name = \"A\",                subquery = list(C = quote(latest(name == \"C\")))) orderly_search(quote(uses(latest(name == \"A\"))),                name = \"C\") orderly_search(   quote(latest(uses(single(usedby(latest(name == \"C\")) && name == \"A\")))),   name = \"E\") orderly_query(   quote(usedby(single(uses(name == \"D\")))),   name = \"A\") orderly_query(   quote(usedby(single(uses(single(usedby(latest(name == \"C\")) && name == \"A\"))) && name == \"E\"))),   name = \"D\")"},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"simple-things","dir":"Articles","previous_headings":"Possible future queries and interface improvements","what":"Simple things","title":"Orderly query DSL","text":"orderly1 supports .null(parameter:x) might generalise support However, Python None Julia nothing, complicates things. Alternatively use missing(parameter:x)?","code":"parameter:x == NULL"},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"explain-the-query","dir":"Articles","previous_headings":"Possible future queries and interface improvements","what":"Explain the query","title":"Orderly query DSL","text":"Often, people want know “packet match”? good show query set packets fail query excluded. definitely bit work.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/troubleshooting.html","id":"outpack-files-accidentally-committed-to-git","dir":"Articles","previous_headings":"","what":"Outpack files accidentally committed to git","title":"Troubleshooting","text":"discussed orderly introduction, want commit files .outpack/, drafts/ archive/ (used) git create sorts problems line. directed , probably ended files git want undo situation. least painful way depends situation. now put guard rails try prevent happening, still happen modify .gitignore file force-add files example. situation, orderly2 shout : may directed page. just want continue working anyway, run suggested command: things work warning first time session: subsequent calls display warning: rest section discusses might permanently fix issue.","code":"orderly2::orderly_run(\"data\") ## Error in `orderly2::orderly_run()`: ## ! Detected 6 outpack files committed to git ## ✖ Detected files were found in '.outpack/' and 'archive/' ## ℹ For tips on resolving this, please see ##   <https://mrc-ide.github.io/orderly2/articles/troubleshooting.html> ## ℹ To turn this into a warning and continue anyway set the option ##   'orderly_git_error_is_warning' to TRUE by running ##   options(orderly_git_error_is_warning = TRUE) options(orderly_git_error_is_warning = TRUE) orderly2::orderly_run(\"data\") ## Warning in orderly2::orderly_run(\"data\"): Detected 6 outpack files committed to git ## ✖ Detected files were found in '.outpack/' and 'archive/' ## ℹ For tips on resolving this, please see ##   <https://mrc-ide.github.io/orderly2/articles/troubleshooting.html> ## This warning is displayed once per session. ## ✔ Wrote '.gitignore' ## ℹ Starting packet 'data' `20240905-080246-3b8cf756` at 2024-09-05 08:02:46.235029 ## > orderly2::orderly_artefact(\"data.rds\", description = \"Final data\") ## > saveRDS(mtcars, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240905-080246-3b8cf756 at 2024-09-05 08:02:46.270178 (0.03514838 secs) ## [1] \"20240905-080246-3b8cf756\" orderly2::orderly_run(\"data\") ## ℹ Starting packet 'data' `20240905-080246-5a93f4e7` at 2024-09-05 08:02:46.356176 ## > orderly2::orderly_artefact(\"data.rds\", description = \"Final data\") ## > saveRDS(mtcars, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240905-080246-5a93f4e7 at 2024-09-05 08:02:46.376964 (0.02078772 secs) ## [1] \"20240905-080246-5a93f4e7\""},{"path":"https://mrc-ide.github.io/orderly2/articles/troubleshooting.html","id":"i-dont-care-about-my-history-at-all","dir":"Articles","previous_headings":"Outpack files accidentally committed to git","what":"I don’t care about my history at all","title":"Troubleshooting","text":"case just started project, yet collaborating anyone else (person willing re-clone sources). simplest thing : Delete .git directory entirely Run orderly2::orderly_gitignore_update(\"(root)\") set reasonable .gitignore prevent situation happening Run git add . add everything back (review git status make sure ’re happy) Run git commit -m \"Initial commit\" create single commit contains files currently repo history, also .outpack files previously pushed repo GitHub similar need set git remote add origin https://github.com/user/repo (replacing user/repo path, using git@github.com:user/repo use ssh talk GitHub) git branch -M main assuming using main default branch, now common git push --force -u origin main Note destructive require coordination collaborators changed history.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/troubleshooting.html","id":"i-just-want-this-to-go-away-and-nothing-i-have-committed-is-very-large","dir":"Articles","previous_headings":"Outpack files accidentally committed to git","what":"I just want this to go away and nothing I have committed is very large","title":"Troubleshooting","text":"care history, also committed files (e.g., committed files .outpack/ small large 100MB file archive/ preventing pushing GitHub) just delete offending files git without updating history, affecting local copies. Run git rm --cached .outpack (repeating draft archive needed) Run orderly2::orderly_gitignore_update(\"(root)\") set reasonable .gitignore prevent situation happening Run git add .gitignore also stage Run git commit -m \"Delete unwanted outpack files\" can push without issues.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/troubleshooting.html","id":"i-care-about-my-history-but-want-this-stuff-gone","dir":"Articles","previous_headings":"Outpack files accidentally committed to git","what":"I care about my history but want this stuff gone","title":"Troubleshooting","text":"working branch, unwanted files committed branch, simplest thing copy changes made somewhere safe, create new branch current main copy changes . somewhat automatically generating applying patch: unwanted files committed onto default branch, potentially gory history rewriting. See StackOverflow question, git docs currently recommended tool . Good luck!","code":"git diff -- src > changes.patch git checkout main git checkout -b changes-attempt2 git apply changes.patch git push -u origin changes-attempt2"},{"path":"https://mrc-ide.github.io/orderly2/authors.html","id":null,"dir":"","previous_headings":"","what":"Authors","title":"Authors and Citation","text":"Rich FitzJohn. Author, maintainer. Robert Ashton. Author. Alex Hill. Author. Imperial College Science, Technology Medicine. Copyright holder.","code":""},{"path":"https://mrc-ide.github.io/orderly2/authors.html","id":"citation","dir":"","previous_headings":"","what":"Citation","title":"Authors and Citation","text":"FitzJohn R, Ashton R, Hill (2024). orderly2: Orderly Next Generation. R package version 1.99.33, https://github.com/mrc-ide/orderly2.","code":"@Manual{,   title = {orderly2: Orderly Next Generation},   author = {Rich FitzJohn and Robert Ashton and Alex Hill},   year = {2024},   note = {R package version 1.99.33},   url = {https://github.com/mrc-ide/orderly2}, }"},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"orderly2","dir":"","previous_headings":"","what":"Orderly Next Generation","title":"Orderly Next Generation","text":"attendant hospital responsible non-medical care patients maintenance order cleanliness. soldier carries orders performs minor tasks officer. orderly2 package designed help make analysis reproducible. principal aim automate series basic steps process writing analyses, making easy : track inputs analysis (packages, code, data resources) store multiple versions analysis repeated track outputs analysis create analyses depend outputs previous analyses","code":""},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"get-started","dir":"","previous_headings":"","what":"Get started","title":"Orderly Next Generation","text":"Two vignettes provide overview package, depending tastes: vignette(\"orderly2\") describes problem orderly2 tries solve (read package website) vignette(\"introduction\") describes orderly2 example, walking basic features (read package website)","code":""},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"roadmap","dir":"","previous_headings":"","what":"Roadmap","title":"Orderly Next Generation","text":"package ground-rewrite orderly package, now ready use new projects adventurous users original package. next months orderly renamed orderly1, finalise API package. stable, rename back orderly2 orderly release CRAN replace existing package. migration path existing users described vignette(\"migrating\") updated based experience migrating people. web framework OrderlyWeb rewritten parallel (packit) provide migration path .","code":""},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"installation","dir":"","previous_headings":"","what":"Installation","title":"Orderly Next Generation","text":"Please install r-universe: prefer, can install GitHub remotes:","code":"install.packages(   \"orderly2\",   repos = c(\"https://mrc-ide.r-universe.dev\", \"https://cloud.r-project.org\")) remotes::install_github(\"mrc-ide/orderly2\")"},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"testing","dir":"","previous_headings":"","what":"Testing","title":"Orderly Next Generation","text":"run tests, need outpack_server available system path. One way achieve run tests also require development versions gert jsonvalidate. can installed rOpenSci r-universe, : Without , tests skipped, suite still run.","code":"cargo install --git https://github.com/mrc-ide/outpack_server install.packages(   c(\"gert\", \"jsonvalidate\"),   repos = c(\"https://ropensci.r-universe.dev\", \"https://cloud.r-project.org\"))"},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"licence","dir":"","previous_headings":"","what":"Licence","title":"Orderly Next Generation","text":"MIT © Imperial College Science, Technology Medicine","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_artefact.html","id":null,"dir":"Reference","previous_headings":"","what":"Declare orderly artefacts — orderly_artefact","title":"Declare orderly artefacts — orderly_artefact","text":"Declare artefact. turn number orderly features; see Details . can multiple calls function within orderly script.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_artefact.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Declare orderly artefacts — orderly_artefact","text":"","code":"orderly_artefact(description = NULL, files)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_artefact.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Declare orderly artefacts — orderly_artefact","text":"description name artefact files files within artefact","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_artefact.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Declare orderly artefacts — orderly_artefact","text":"Undefined","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_artefact.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Declare orderly artefacts — orderly_artefact","text":"(1) files matching copied src directory draft directory unless also listed resource orderly_resource(). feature enabled call function top level orderly script contains string literals (variables). (2) script fails produce files, orderly_run() fail, guaranteeing task really produce things need . (3) within final metadata, artefacts additional metadata; description provide grouping","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":null,"dir":"Reference","previous_headings":"","what":"Clean up source directory — orderly_cleanup","title":"Clean up source directory — orderly_cleanup","text":"Find, delete, file generated running report. comfortable , strongly recommended run orderly_cleanup_status first see deleted.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Clean up source directory — orderly_cleanup","text":"","code":"orderly_cleanup(name = NULL, dry_run = FALSE, root = NULL, locate = TRUE)  orderly_cleanup_status(name = NULL, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Clean up source directory — orderly_cleanup","text":"name Name report directory clean (.e., look src/<name> relative orderly root dry_run Logical, indicating delete anything, instead just print information root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, just outpack (see orderly_init details). locate Logical, indicating configuration searched .  TRUE config given, orderly looks working directory parents finds .outpack directory","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Clean up source directory — orderly_cleanup","text":"(currently unstable) object class orderly_cleanup_status within element delete indicates files deleted (orderly_cleanup_status) deleted (orderly_cleanup)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Clean up source directory — orderly_cleanup","text":"file deletion, look remove empty directories; orderly2 similar semantics git directories never directly tracked. recent gert (yet CRAN) ask git files ignored; ignored good candidates deletion! encourage keep per-report .gitignore lists files copy source directory, can use information clean files generation. Importantly, even file matches ignore rule committed repository, longer match ignore rule.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"notes-for-user-of-orderly-","dir":"Reference","previous_headings":"","what":"Notes for user of orderly1","title":"Clean up source directory — orderly_cleanup","text":"orderly1 function quite different semantics, full set possible files always knowable yaml file. , start point view list files compare directory.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Clean up source directory — orderly_cleanup","text":"","code":"# Create a simple example: path <- orderly2::orderly_example(\"default\") #> ✔ Created orderly root at '/tmp/RtmpTXa1jo/file170f43324a93'  # We simulate running a packet interactively by using 'source'; # you might have run this line-by-line, or with the \"Source\" # button in Rstudio. source(file.path(path, \"src/data/data.R\"), chdir = TRUE) #> Warning: cannot open file '/tmp/RtmpTXa1jo/file170f43324a93/src/data/data.R': No such file or directory #> Error in file(filename, \"r\", encoding = encoding): cannot open the connection  # Having run this, the output of the report is present in the # source directory: fs::dir_tree(path) #> Error: [ENOENT] Failed to search directory '/tmp/RtmpTXa1jo/file170f43324a93': no such file or directory  # We can detect what might want cleaning up by running # \"orderly_cleanup_status\": orderly2::orderly_cleanup_status(\"data\", root = path) #> Error: Directory does not exist: '/tmp/RtmpTXa1jo/file170f43324a93'  # Soon this will print more nicely to the screen, but for now you # can see that the status of \"data.rds\" is \"derived\", which means # that orderly knows that it is subject to being cleaned up; the # \"delete\" element shows what will be deleted.  # Do the actual deletion: orderly2::orderly_cleanup(\"data\", root = path) #> Error: Directory does not exist: '/tmp/RtmpTXa1jo/file170f43324a93'"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config.html","id":null,"dir":"Reference","previous_headings":"","what":"Read configuration — orderly_config","title":"Read configuration — orderly_config","text":"Read current orderly configuration, stored within outpack root, along orderly-specific extensions.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Read configuration — orderly_config","text":"","code":"orderly_config(root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Read configuration — orderly_config","text":"root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Read configuration — orderly_config","text":"list configuration options: core: important options outpack store, containing: path_archive: path human-readable packet archive, NULL disabled (set orderly_config_set core.path_archive) use_file_store: Indicates content-addressable file store enabled (core.use_file_store) require_complete_tree: Indicates outpack store requires dependencies fully available (core.require_complete_tree) hash_algorithm: hash algorithm used (currently modifiable) location: Information locations; see orderly_location_add, orderly_location_rename orderly_location_remove interact configuration, orderly_location_list simply list available locations. Returns data.frame columns name, id, priority, type args, args list column. orderly: list orderly-specific configuration; just minimum required version (minimum_orderly_version).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Read configuration — orderly_config","text":"","code":"# A default configuration in a new temporary directory path <- withr::local_tempdir() orderly2::orderly_init(path) #> ✔ Created orderly root at '/tmp/RtmpTXa1jo/file170f7685df08' orderly2::orderly_config(path) #> $core #> $core$path_archive #> [1] \"archive\" #>  #> $core$use_file_store #> [1] FALSE #>  #> $core$require_complete_tree #> [1] FALSE #>  #> $core$hash_algorithm #> [1] \"sha256\" #>  #>  #> $location #>    name  type args #> 1 local local      #>  #> $orderly #> $orderly$minimum_orderly_version #> [1] ‘1.99.0’ #>  #>"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":null,"dir":"Reference","previous_headings":"","what":"Set configuration options — orderly_config_set","title":"Set configuration options — orderly_config_set","text":"Set configuration options. can currently set; expanded time. See Details.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Set configuration options — orderly_config_set","text":"","code":"orderly_config_set(..., options = list(...), root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Set configuration options — orderly_config_set","text":"... Named options set (e.g., pass argument core.require_complete_tree = TRUE) options alternative ..., can pass list named options (e.g., list(core.require_complete_tree = TRUE)).  interface typically easier program . root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Set configuration options — orderly_config_set","text":"Nothing","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Set configuration options — orderly_config_set","text":"Options set order provided.  Currently, setting one option fails, options processed previous ones (rely behaviour, may change). Currently can set: core.require_complete_tree See orderly_init description options.","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Set configuration options — orderly_config_set","text":"","code":"# The default configuration does not include a file store, and # saves output within the \"archive\" directory: path <- withr::local_tempdir() orderly2::orderly_init(path) #> ✔ Created orderly root at '/tmp/RtmpTXa1jo/file170f658cd5a4' fs::dir_tree(path, all = TRUE) #> /tmp/RtmpTXa1jo/file170f658cd5a4 #> ├── .outpack #> │   ├── config.json #> │   ├── location #> │   └── metadata #> └── orderly_config.yml  # Change this after the fact: orderly2::orderly_config_set(core.use_file_store = TRUE,                              core.path_archive = NULL,                              root = path) fs::dir_tree(path, all = TRUE) #> /tmp/RtmpTXa1jo/file170f658cd5a4 #> ├── .outpack #> │   ├── config.json #> │   ├── files #> │   ├── index #> │   │   └── outpack.rds #> │   ├── location #> │   └── metadata #> └── orderly_config.yml"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_copy_files.html","id":null,"dir":"Reference","previous_headings":"","what":"Copy files from a packet — orderly_copy_files","title":"Copy files from a packet — orderly_copy_files","text":"Copy files packet anywhere. Similar orderly_dependency except used active packet context. can use function pull files outpack root directory outside control outpack, example. Note arguments need must provided name, position, exception id query.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_copy_files.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Copy files from a packet — orderly_copy_files","text":"","code":"orderly_copy_files(   ...,   files,   dest,   overwrite = TRUE,   envir = parent.frame(),   options = NULL,   root = NULL,   locate = TRUE )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_copy_files.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Copy files from a packet — orderly_copy_files","text":"... Arguments passed orderly_query, perhaps just query expression files Files copy packet. can (1) character vector, case files copied without changing names, (2) named character vector, case name used destination name, (3) data.frame (including tbl_df, data.frame objects) containing columns , case files copied names . cases, want import directory files packet, must refer source trailing slash (e.g., c(= \"/\")), create local directory /... files upstream packet directory /. omit slash error thrown suggesting add slash intended. can use limited form string interpolation names argument; using ${variable} pick values envir substitute string.  similar interpolation might familiar glue::glue similar, much simpler concatenation fancy features supported. Note unfortunate, (us) avoidable inconsistency ; interpolation values environment query done using environment:x destination filename ${x}. dest directory copy overwrite Overwrite files destination; typically want, set FALSE prefer error thrown destination file already exists. envir Optionally, environment substitute query (using environment: prefix). default use calling environment, can explicitly pass want control lookup happens. options Optionally, orderly_search_options object controlling search performed, packets considered scope. provided, default options used (.e., orderly2::orderly_search_options()) root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_copy_files.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Copy files from a packet — orderly_copy_files","text":"Nothing, invisibly. Primarily called side effect copying files packet directory dest","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_copy_files.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Copy files from a packet — orderly_copy_files","text":"can call function id string, case search packet proceed regardless whether id present.  called arguments (e.g., string match id format, named argument name, subquery parameters) interpret arguments query orderly_search find id. error query return exactly one packet id, probably want use latest(). different ways might fail (recover failure): id known metadata store (known unpacked also known present remote) fail impossible resolve files. Consider refreshing metadata orderly_location_pull_metadata refresh . id unpacked local copy files referred can found, error default (see next option). However, sometimes file refer might also present downloaded packet depended , content file unchanged packet version locally. id unpacked, local copy file allow_remote TRUE try request file whatever remote selected orderly_location_pull_packet packet. Note empty directories might created failure.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_dependency.html","id":null,"dir":"Reference","previous_headings":"","what":"Declare a dependency — orderly_dependency","title":"Declare a dependency — orderly_dependency","text":"Declare dependency another packet","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_dependency.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Declare a dependency — orderly_dependency","text":"","code":"orderly_dependency(name, query, files)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_dependency.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Declare a dependency — orderly_dependency","text":"name name packet depend query query search ; often simply string latest, indicating recent version. may want complex query though. files Files copy packet. can (1) character vector, case files copied without changing names, (2) named character vector, case name used destination name, (3) data.frame (including tbl_df, data.frame objects) containing columns , case files copied names . cases, want import directory files packet, must refer source trailing slash (e.g., c(= \"/\")), create local directory /... files upstream packet directory /. omit slash error thrown suggesting add slash intended. can use limited form string interpolation names argument; using ${variable} pick values envir substitute string.  similar interpolation might familiar glue::glue similar, much simpler concatenation fancy features supported. Note unfortunate, (us) avoidable inconsistency ; interpolation values environment query done using environment:x destination filename ${x}.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_dependency.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Declare a dependency — orderly_dependency","text":"Undefined","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_dependency.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Declare a dependency — orderly_dependency","text":"See orderly_run details search options used select locations packets found , data fetched network. running interactively, obviously work, use orderly_interactive_set_search_options() set options function respond .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_description.html","id":null,"dir":"Reference","previous_headings":"","what":"Describe the current packet — orderly_description","title":"Describe the current packet — orderly_description","text":"Describe current packet","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_description.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Describe the current packet — orderly_description","text":"","code":"orderly_description(display = NULL, long = NULL, custom = NULL)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_description.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Describe the current packet — orderly_description","text":"display friendly name report; displayed locations web interface, packit. given, must scalar character. long longer description report. given, must scalar character. custom additional metadata. given, must named list, elements scalar atomics (character, number, logical).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_description.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Describe the current packet — orderly_description","text":"Undefined","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_example.html","id":null,"dir":"Reference","previous_headings":"","what":"Copy a simple orderly example — orderly_example","title":"Copy a simple orderly example — orderly_example","text":"Copy simple orderly example use docs. function form part workflow!","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_example.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Copy a simple orderly example — orderly_example","text":"","code":"orderly_example(name, ..., dest = NULL)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_example.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Copy a simple orderly example — orderly_example","text":"name name example copy. Currently \"default\" supported. ... Arguments passed orderly_init() dest destination. default use withr::local_tempfile() create temporary directory clean . suitable use orderly examples, may prefer provide path. path must already exist.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_example.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Copy a simple orderly example — orderly_example","text":"Invisibly, path example.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_example.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Copy a simple orderly example — orderly_example","text":"","code":"path <- orderly2::orderly_example(\"default\") #> ✔ Created orderly root at '/tmp/RtmpTXa1jo/file170f8fdfa80' orderly2::orderly_list_src(root = path) #> Error: Directory does not exist: '/tmp/RtmpTXa1jo/file170f8fdfa80'"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_gitignore_update.html","id":null,"dir":"Reference","previous_headings":"","what":"Update a gitignore file — orderly_gitignore_update","title":"Update a gitignore file — orderly_gitignore_update","text":"Update gitignore, useful prevent accidentally committing files source control generated. includes artefacts, shared resources dependencies (within report directory) global level contents .outpack directory, draft folder archive directory.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_gitignore_update.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Update a gitignore file — orderly_gitignore_update","text":"","code":"orderly_gitignore_update(name, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_gitignore_update.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Update a gitignore file — orderly_gitignore_update","text":"name name gitignore file update, string \"(root)\" root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, just outpack (see orderly_init details). locate Logical, indicating configuration searched .  TRUE config given, orderly looks working directory parents finds .outpack directory","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_gitignore_update.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Update a gitignore file — orderly_gitignore_update","text":"Nothing, called side effects","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_gitignore_update.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Update a gitignore file — orderly_gitignore_update","text":"function fails message edit '.gitignore', markers corrupted, look special markers within .gitignore file.  look like   edit file : lines appears file anything first two lines order get error message, search remove lines rerun.","code":"# ---VVV--- added by orderly ---VVV---------------- # Don't manually edit content between these markers ... patterns # ---^^^--- added by orderly ---^^^----------------"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_hash.html","id":null,"dir":"Reference","previous_headings":"","what":"Compute a hash — orderly_hash_file","title":"Compute a hash — orderly_hash_file","text":"Use orderly2's hashing functions.  intended advanced users, particular want create hashes consistent orderly2 within plugins. default behaviour use algorithm used orderly root (via root argument, usual root location approach). However, string provided algorithm can use alternative algorithm.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_hash.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Compute a hash — orderly_hash_file","text":"","code":"orderly_hash_file(path, algorithm = NULL, root = NULL, locate = TRUE)  orderly_hash_data(data, algorithm = NULL, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_hash.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Compute a hash — orderly_hash_file","text":"path name file hash algorithm name algorithm use, overriding orderly root. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml data string hash","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_hash.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Compute a hash — orderly_hash_file","text":"string format <algorithm>:<digest>","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_hash.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Compute a hash — orderly_hash_file","text":"","code":"orderly2::orderly_hash_data(\"hello\", \"md5\") #> [1] \"md5:5d41402abc4b2a76b9719d911017c592\""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":null,"dir":"Reference","previous_headings":"","what":"Initialise an orderly repository — orderly_init","title":"Initialise an orderly repository — orderly_init","text":"Initialise empty orderly repository, initialise source copy orderly repository (see Details). orderly repository defined presence file orderly_config.yml root, along directory .outpack/ level.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Initialise an orderly repository — orderly_init","text":"","code":"orderly_init(   root = \".\",   path_archive = \"archive\",   use_file_store = FALSE,   require_complete_tree = FALSE,   force = FALSE )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Initialise an orderly repository — orderly_init","text":"root path initialise repository root .  repository already initialised, operation checks options passed set repository (erroring ), otherwise nothing.  default path current working directory. path_archive Path archive directory, used store human-readable copies packets.  NULL, copy made, file_store must TRUE use_file_store Logical, indicating use content-addressable file-store source truth packets.  archive non-NULL, file-store used source truth duplicated files archive exist convenience. require_complete_tree Logical, indicating require complete tree packets.  currently affects orderly_location_pull_packet, requiring always operates recursive mode.  FALSE default, set TRUE want archive behave well location; TRUE always packets hold metadata . force Logical, indicating shold initialise orderly even directory empty.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Initialise an orderly repository — orderly_init","text":"full, normalised, path root, invisibly. Typically called side effect.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Initialise an orderly repository — orderly_init","text":"expected orderly_config.yml saved version control, .outpack excluded version control; means every clone project need call orderly2::orderly_init() initialise .outpack directory. forget , error thrown reminding need . can safely call orderly2::orderly_init() already-initialised directory, however, arguments passed must exactly match configuration current root, otherwise error thrown. Please use orderly_config_set change configuration, ensures change configuration possible. configuration options given match directory already uses, nothing happens. repository call orderly2::orderly_init() already initialised .outpack directory orderly_config.yml file, write file .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Initialise an orderly repository — orderly_init","text":"","code":"# We'll use an automatically cleaned-up directory for the root: path <- withr::local_tempdir()  # Initialise a new repository, setting an option: orderly2::orderly_init(path, use_file_store = TRUE) #> ✔ Created orderly root at '/tmp/RtmpTXa1jo/file170f7409549f'  # fs::dir_tree(path, all = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_interactive_set_search_options.html","id":null,"dir":"Reference","previous_headings":"","what":"Set search options for interactive use — orderly_interactive_set_search_options","title":"Set search options for interactive use — orderly_interactive_set_search_options","text":"Set search options interactive use orderly; see orderly_dependency orderly_run details. may either orderly_search_options object, list coerced one point use (NULL). applies current session, applies interactive uses orderly functions might received copy search_options via orderly_run","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_interactive_set_search_options.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Set search options for interactive use — orderly_interactive_set_search_options","text":"","code":"orderly_interactive_set_search_options(options = NULL)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_interactive_set_search_options.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Set search options for interactive use — orderly_interactive_set_search_options","text":"options Optional control locations, used orderly_dependency; see Details section orderly_run.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_interactive_set_search_options.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Set search options for interactive use — orderly_interactive_set_search_options","text":"Nothing, called side effects","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_list_src.html","id":null,"dir":"Reference","previous_headings":"","what":"List source reports — orderly_list_src","title":"List source reports — orderly_list_src","text":"List source reports - , directories within src/ look suitable running orderly; directories contain entrypoint file - .R file name directory (e.g., src/data/data.R corresponds data).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_list_src.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"List source reports — orderly_list_src","text":"","code":"orderly_list_src(root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_list_src.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"List source reports — orderly_list_src","text":"root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, just outpack (see orderly_init details). locate Logical, indicating configuration searched .  TRUE config given, orderly looks working directory parents finds .outpack directory","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_list_src.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"List source reports — orderly_list_src","text":"character vector names source reports, suitable passing orderly_run","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_list_src.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"List source reports — orderly_list_src","text":"","code":"path <- orderly2::orderly_example(\"default\") #> ✔ Created orderly root at '/tmp/RtmpTXa1jo/file170f7c5e1acd' orderly2::orderly_list_src(root = path) #> Error: Directory does not exist: '/tmp/RtmpTXa1jo/file170f7c5e1acd'"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":null,"dir":"Reference","previous_headings":"","what":"Add a new location — orderly_location_add","title":"Add a new location — orderly_location_add","text":"Add new location - place packets might found pulled local archive.  Currently file http based locations supported, limited support custom locations. Note adding location pull metadata , need call orderly_location_pull_metadata first.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Add a new location — orderly_location_add","text":"","code":"orderly_location_add(name, type, args, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Add a new location — orderly_location_add","text":"name short name location use.  use, one local orphan type type location add. Currently supported values path (location exists elsewhere filesystem) http (location accessed outpack's http API). args Arguments location driver. arguments vary depending type used, see Details. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Add a new location — orderly_location_add","text":"Nothing","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Add a new location — orderly_location_add","text":"currently support two types locations - path, points outpack archive accessible path (e.g., computer mounted network share), http, requires outpack server running url uses HTTP API communicate, packit, uses Packit web server.  types may added later, configuration options location types definitely needed future. Configuration options different location types: Path locations: path: path archive root. generally absolute path, behaviour outpack unreliable. HTTP locations: Accessing outpack HTTP requires outpack server running. interface expected change expand API, also move support things like TLS authentication. url: location server, including protocol, example http://example.com:8080 Packit locations: Packit locations work HTTPS, include everything outpack location also provide authentication later capabilities think. url: location server token: value login token (currently GitHub token read:org scope).  Later expand authentication modes supported. Custom locations: outpack implementations expected support path http locations, standard arguments .  expect implementations support custom locations, argument lists may vary implementations. allow , can pass location type \"custom\" list arguments.  expect argument 'driver' present among list.  example action, see outpack.sharepoint package. warned may change interface future, case may need update configuration.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":"warning","dir":"Reference","previous_headings":"","what":"Warning","title":"Add a new location — orderly_location_add","text":"API may change move support different types locations.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_list.html","id":null,"dir":"Reference","previous_headings":"","what":"List known pack locations — orderly_location_list","title":"List known pack locations — orderly_location_list","text":"List known locations.  special name local always present within output function (packets known current root), though typically interested locations.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_list.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"List known pack locations — orderly_location_list","text":"","code":"orderly_location_list(verbose = FALSE, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_list.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"List known pack locations — orderly_location_list","text":"verbose Logical, indicating return data.frame includes information location. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_list.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"List known pack locations — orderly_location_list","text":"Depending value verbose: verbose = FALSE: character vector location names.  default behaviour. verbose = TRUE: data.frame columns name, type args.  args column list column, element key-value pair arguments location.","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_metadata.html","id":null,"dir":"Reference","previous_headings":"","what":"Pull metadata from a location — orderly_location_pull_metadata","title":"Pull metadata from a location — orderly_location_pull_metadata","text":"Pull metadata location, updating index.  always relatively quick updates small files contain information can found remote packets.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_metadata.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Pull metadata from a location — orderly_location_pull_metadata","text":"","code":"orderly_location_pull_metadata(location = NULL, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_metadata.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Pull metadata from a location — orderly_location_pull_metadata","text":"location name location pull (see orderly_location_list possible values).  given, pulls locations.  \"local\" \"orphan\" locations always date pulling metadata nothing. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_metadata.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Pull metadata from a location — orderly_location_pull_metadata","text":"Nothing","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_packet.html","id":null,"dir":"Reference","previous_headings":"","what":"Pull one or more packets from a location — orderly_location_pull_packet","title":"Pull one or more packets from a location — orderly_location_pull_packet","text":"Pull one packets (including files) archive one locations. make files available use dependencies (e.g., orderly_dependency).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_packet.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Pull one or more packets from a location — orderly_location_pull_packet","text":"","code":"orderly_location_pull_packet(   ...,   options = NULL,   recursive = NULL,   root = NULL,   locate = TRUE )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_packet.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Pull one or more packets from a location — orderly_location_pull_packet","text":"... Arguments passed orderly_search. special case first argument character vector ids named dot arguments, interpret argument vector ids directly. careful , query may pull lot data particular, passing NULL match everything every remote ! options Options passed orderly_search. option allow_remote must TRUE otherwise packet possibly pulled, error thrown FALSE. recursive non-NULL, logical, indicating recursively pull packets referenced packets specified id.  might copy lot data!  NULL, default value given configuration option require_complete_tree. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_packet.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Pull one or more packets from a location — orderly_location_pull_packet","text":"Invisibly, ids packets pulled","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_packet.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Pull one or more packets from a location — orderly_location_pull_packet","text":"possible take long time pull packets, moving lot data operating slow connection.  Cancelling resuming pull fairly efficient, keep track files copied even case interrupted pull.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_push.html","id":null,"dir":"Reference","previous_headings":"","what":"Push tree to location — orderly_location_push","title":"Push tree to location — orderly_location_push","text":"Push tree location. function works packets known location files required create . pushes files required build packets pushes missing metadata server. process interrupted safe resume transfer files packets missed previous call.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_push.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Push tree to location — orderly_location_push","text":"","code":"orderly_location_push(packet_id, location, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_push.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Push tree to location — orderly_location_push","text":"packet_id One packets push server location name location push (see orderly_location_list possible values). root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_push.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Push tree to location — orderly_location_push","text":"Invisibly, details information actually moved (might less requested, depending dependencies packets already known location).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_remove.html","id":null,"dir":"Reference","previous_headings":"","what":"Remove a location — orderly_location_remove","title":"Remove a location — orderly_location_remove","text":"Remove existing location. packets location known elsewhere now associated 'orphan' location instead.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_remove.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Remove a location — orderly_location_remove","text":"","code":"orderly_location_remove(name, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_remove.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Remove a location — orderly_location_remove","text":"name short name location. remove local orphan root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_remove.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Remove a location — orderly_location_remove","text":"Nothing","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_rename.html","id":null,"dir":"Reference","previous_headings":"","what":"Rename a location — orderly_location_rename","title":"Rename a location — orderly_location_rename","text":"Rename existing location","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_rename.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Rename a location — orderly_location_rename","text":"","code":"orderly_location_rename(old, new, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_rename.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Rename a location — orderly_location_rename","text":"old current short name location. rename local orphan new desired short name location. one local orphan root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_rename.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Rename a location — orderly_location_rename","text":"Nothing","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata.html","id":null,"dir":"Reference","previous_headings":"","what":"Read outpack metadata — orderly_metadata","title":"Read outpack metadata — orderly_metadata","text":"Read metadata particular id. may want use orderly_search find id corresponding particular query.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Read outpack metadata — orderly_metadata","text":"","code":"orderly_metadata(id, root = NULL, locate = FALSE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Read outpack metadata — orderly_metadata","text":"id id fetch metadata . error thrown id known root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Read outpack metadata — orderly_metadata","text":"list metadata. See outpack schema details (https://github.com/mrc-ide/outpack)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":null,"dir":"Reference","previous_headings":"","what":"Extract metadata from orderly2 packets — orderly_metadata_extract","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"Extract metadata group packets.  experimental high-level function interacting metadata way hope useful. expand bit time goes , based feedback get let us know think.  See Details use .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"","code":"orderly_metadata_extract(..., extract = NULL, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"... Arguments passed orderly_search. special case first argument character vector ids named dot arguments, interpret argument vector ids directly. extract character vector columns extract, possibly named. See Details format. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"data.frame, columns vary based names extract; see Details information.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"Extracting data outpack metadata challenging way works data structures familiar R users, naturally tree structured, metadata may present packets (e.g., packet depend another dependency section, one run context without git git metadata). just want raw tree-structured data, can always use orderly_metadata load full metadata packet (even one currently available computer, just known ) structure data remain fairly constant across orderly2 versions. However, sometimes want extract data order ask specific questions like: parameter combinations available across range packets? particular set packets used? files packets produce? Later like ask even complex questions like: version file graph.png change? inputs changed versions? ...able answer questions requires similar approach interrogating metadata across range packets. orderly_metadata_extract function aims simplify process pulling bits metadata arranging data.frame (sorts) .  little mini-language extract argument simple rewriting results, can always . order use function need know metadata available; expand vignette worked examples make easier understand. function works top-level keys, : id: packet id (always returned) name: packet name parameters: key-value pair values, string keys atomic values. guarantee presence keys packets, types. time: key-value pair times, string keys time values (see DateTimeClasses; stored seconds since 1970 actual metadata). present start end always present. files: files present packet. data.frame (per packet), columns path (relative), size (bytes) hash. depends: dependencies used packet. data.frame (per packet), columns packet (id), query (string, used find packet) files (another data.frame columns corresponding filenames upstream packet, respectively) git: either metadata state git null. given sha branch strings, url array strings/character vector (can zero, one elements). session: information session packet run (unstandardised, even orderly version may change) custom: additional metadata added respective engine.  packets run orderly2, orderly field , list: artefacts: data.frame artefact information, containing columns description (string) paths (list column paths). shared: data.frame copied shared resources original name () name copied packet (). role: data.frame identified roles files, columns path role. description: list information orderly_description human-readable descriptions tags. session: list information session run, list platform containing information platform (R version version, operating system os system name system) packages containing columns package , version attached. nesting makes providing universally useful data format difficult; considering files data.frame files column, list data.frames; similar nestedness applies depends orderly custom data. However, able fairly easily process data format need . simplest extraction uses names top-level keys:   creates data.frame columns corresponding keys, one row per packet. name always string, character vector, parameters files complex, list columns. must provide id; always returned always first character vector column. can rename columns providing name entries within extract, example:   , except parameters column renamed pars. interestingly, can index structure like parameters; suppose want value parameter x, write:   allowed packet parameters element list. However, know type x (might vary packets). can add information though write:   create numeric column. packet value x non-integer, call orderly_metadata_extract fail error, packet lacks value x, missing value appropriate type added. Note coercion number, error non-NULL non-numeric value found.  Valid types use <type> boolean, number string (note differ slightly R's names want emphasise scalar quantities; also note integer may produce unexpected errors integer-like numeric values). can also use list default.  Things schema known scalar atomics (name) automatically simplified. can index array-valued elements (files depends) way object-valued elements:   get list column file names per packet another hashes, probably less useful data.frame get extracting just files longer hash information aligned. can index fairly deeply; possible get orderly \"display name\" :   path need extract dot (likely package name plugin, custom.orderly.db) need escape dot backslash (, custom.orderly\\.db). probably need two slashes use raw string (recent versions R).","code":"extract = c(\"name\", \"parameters\", \"files\") extract = c(\"name\", pars = \"parameters\", \"files\") extract = c(x = \"parameters.x\") extract = c(x = \"parameters.x is number\") extract = c(file_path = \"files.path\", file_hash = \"files.hash\") extract = c(display = \"custom.orderly.description.display is string\")"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":"custom-orderly-metadata","dir":"Reference","previous_headings":"","what":"Custom 'orderly' metadata","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"Within custom.orderly, additional fields can extracted. format subject change, stored metadata schema (short term) way deserialise . probably best rely right now, expand section can.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_read.html","id":null,"dir":"Reference","previous_headings":"","what":"Read outpack metadata json file — orderly_metadata_read","title":"Read outpack metadata json file — orderly_metadata_read","text":"Low-level function reading metadata deserialising . function can used directly read metadata json file without reference root contains . may useful context reading metadata file written part failed run.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_read.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Read outpack metadata json file — orderly_metadata_read","text":"","code":"orderly_metadata_read(path, plugins = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_read.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Read outpack metadata json file — orderly_metadata_read","text":"path Path json file plugins Try deserialise data loaded plugins (see Details).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_read.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Read outpack metadata json file — orderly_metadata_read","text":"list outpack metadata; see schema details. contrast reading json file directly jsonlite::fromJSON, function take care convert scalar length-one vectors expected types.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_read.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Read outpack metadata json file — orderly_metadata_read","text":"Custom metadata saved plugins may deserialised expected called function, designed operate separately valid orderly root (.e., load data file regardless came ). plugins TRUE (default) deserialise data matches loaded plugin.  means behaviour function depends loaded plugin packages. can force running orderly2::orderly_config() within orderly directory, load declared plugins.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_new.html","id":null,"dir":"Reference","previous_headings":"","what":"Create a new report — orderly_new","title":"Create a new report — orderly_new","text":"Create new empty report.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_new.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Create a new report — orderly_new","text":"","code":"orderly_new(name, template = NULL, force = FALSE, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_new.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Create a new report — orderly_new","text":"name name report template template use. acceptable values now NULL (uses built-default) FALSE suppresses default content.  may support customisable templates future - let us know useful. force Create orderly file - <name>.R within existing directory src/<name>; may useful already created directory files first want help creating orderly file. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, just outpack (see orderly_init details). locate Logical, indicating configuration searched .  TRUE config given, orderly looks working directory parents finds .outpack directory","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_new.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Create a new report — orderly_new","text":"Nothing, called side effects ","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parameters.html","id":null,"dir":"Reference","previous_headings":"","what":"Declare orderly parameters — orderly_parameters","title":"Declare orderly parameters — orderly_parameters","text":"Declare orderly parameters. one call within file, though enforced! Typically put close top, though order really matter. Parameters scalar atomic values (e.g. string, number boolean) defaults must present literally (.e., may come variable ). Provide NULL default, case parameter required.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parameters.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Declare orderly parameters — orderly_parameters","text":"","code":"orderly_parameters(...)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parameters.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Declare orderly parameters — orderly_parameters","text":"... number parameters","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parameters.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Declare orderly parameters — orderly_parameters","text":"Undefined","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parameters.html","id":"behaviour-in-interactive-sessions","dir":"Reference","previous_headings":"","what":"Behaviour in interactive sessions","title":"Declare orderly parameters — orderly_parameters","text":"running interactively (.e., via source() running orderly file session copy/paste Rstudio), orderly_parameters() function different behaviour. First, look current environment (likely global environment) values parameters - , variables bound names parameters.  parameters found look default values use possible, possible either error prompt based global option orderly_interactive_parameters_missing_error. TRUE, ask enter value parameters (strings need entered quotes).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parse_file.html","id":null,"dir":"Reference","previous_headings":"","what":"Parse the orderly entrypoint script — orderly_parse_file","title":"Parse the orderly entrypoint script — orderly_parse_file","text":"expert use .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parse_file.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Parse the orderly entrypoint script — orderly_parse_file","text":"","code":"orderly_parse_file(path)  orderly_parse_expr(exprs, filename)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parse_file.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Parse the orderly entrypoint script — orderly_parse_file","text":"path Path orderly_* script exprs Parsed AST orderly_* script filename Name orderly_* file include metadata","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parse_file.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Parse the orderly entrypoint script — orderly_parse_file","text":"Parsed orderly entrypoint script","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parse_file.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Parse the orderly entrypoint script — orderly_parse_file","text":"Parses details calls orderly_ -script functions intermediate representation downstream use. Also validates calls orderly_* -script functions well-formed.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_add_metadata.html","id":null,"dir":"Reference","previous_headings":"","what":"Add metadata from plugin — orderly_plugin_add_metadata","title":"Add metadata from plugin — orderly_plugin_add_metadata","text":"Add plugin-specific metadata running packet. take describing. accumulate number bits metadata arbitrary fields, later serialise json.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_add_metadata.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Add metadata from plugin — orderly_plugin_add_metadata","text":"","code":"orderly_plugin_add_metadata(name, field, data)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_add_metadata.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Add metadata from plugin — orderly_plugin_add_metadata","text":"name name plugin; must used orderly_plugin_register orderly_plugin_context field name field add data . required even plugin produces one sort data, case can remove later within serialisation function. data Arbitrary data added currently running packet","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_add_metadata.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Add metadata from plugin — orderly_plugin_add_metadata","text":"Nothing, called side effects","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_context.html","id":null,"dir":"Reference","previous_headings":"","what":"Fetch plugin context — orderly_plugin_context","title":"Fetch plugin context — orderly_plugin_context","text":"Fetch running context, use within plugin.  intention within free functions plugin makes available, call function get information state packet.  typically call orderly_plugin_add_metadata() afterwards.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_context.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Fetch plugin context — orderly_plugin_context","text":"","code":"orderly_plugin_context(name, envir)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_context.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Fetch plugin context — orderly_plugin_context","text":"name Name plugin envir environment calling function. can typically pass parent.frame() (rlang::caller_env()) function calling orderly_plugin_context() function called user. argument effect interactive use (envir almost certainly global environment).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_context.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Fetch plugin context — orderly_plugin_context","text":"list elements: is_active: logical, indicating running orderly_run(); may need change behaviour depending value. path: path running packet. almost always working directory, unless packet contains calls setwd() similar. may create files . config: configuration plugin, processing plugin's read function (see orderly_plugin_register) envir: environment packet running . Often global environment, assume ! may read write environment. src: path packet source directory. different current directory packet running, user interactively working report. may read directory must write parameters: parameters passed run report.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_context.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Fetch plugin context — orderly_plugin_context","text":"plugin function called, orderly2 running one two modes; (1) within orderly_run(), case part way creating packet brand new directory, possibly using special environment evaluation, (2) interactively, user developing report. plugin needs able support modes, function return information state help cope either case.","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_register.html","id":null,"dir":"Reference","previous_headings":"","what":"Register an orderly plugin — orderly_plugin_register","title":"Register an orderly plugin — orderly_plugin_register","text":"Create orderly plugin. plugin typically defined package used extend orderly enabling new functionality, declared orderly_config.yml orderly file, affecting running reports primarily creating new objects report environment.  system discussed detail vignette(\"plugins\"), expanded (likely breaking ways) soon.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_register.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Register an orderly plugin — orderly_plugin_register","text":"","code":"orderly_plugin_register(   name,   config,   serialise = NULL,   deserialise = NULL,   cleanup = NULL,   schema = NULL )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_register.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Register an orderly plugin — orderly_plugin_register","text":"name name plugin, typically package name config function read, check process configuration section orderly_config.yml. function passed deserialised data plugin's section orderly_config.yml, full path file.  order loading plugins defined, plugin must standalone try interact plugins load. return processed copy configuration data, passed second argument read. serialise function serialise metadata added plugin's functions outpack metadata. passed list entries pushed via orderly_plugin_add_metadata(); named list names corresponding field argument orderly_plugin_add_metadata list element unnamed list values corresponding data. NULL, serialisation done, metadata plugin added. deserialise function deserialise metadata serialised serialise function. intended help deal issues disambiguating unserialising objects json (scalars vs arrays lenth 1, data.frames vs lists--lists etc), make plugin nicer work orderly_metadata_extract(). function given single argument data data jsonlite::fromJSON(..., simplifyVector = FALSE) apply required simplifications , returning modified copy argument. cleanup Optionally, function clean state plugin uses. can call orderly_plugin_context within function access anything need . given, cleanup done. schema Optionally path, within package, schema metadata created plugin; omit .json extension.  file contains sources file inst/plugin/myschema.json pass plugin/myschema.  See vignette(\"plugins\") details.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_register.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Register an orderly plugin — orderly_plugin_register","text":"Nothing, function called side effect registering plugin.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_prune_orphans.html","id":null,"dir":"Reference","previous_headings":"","what":"Prune orphan packet metadata — orderly_prune_orphans","title":"Prune orphan packet metadata — orderly_prune_orphans","text":"Prune orphan packets metadata store.  function can used remove references packets longer reachable; happened deleted packet manually archive ran orderly_validate_archive removed location.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_prune_orphans.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Prune orphan packet metadata — orderly_prune_orphans","text":"","code":"orderly_prune_orphans(root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_prune_orphans.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Prune orphan packet metadata — orderly_prune_orphans","text":"root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_prune_orphans.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Prune orphan packet metadata — orderly_prune_orphans","text":"Invisibly, character vector orphaned packet ids","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_prune_orphans.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Prune orphan packet metadata — orderly_prune_orphans","text":"orphan packet used anywhere, can easily drop - never existed.  referenced metadata know elsewhere locally, problem upstream location (one happen). referenced packet run locally, metadata deleted. expose function mostly users want expunge permanently reference previously run packets. hope never need really reason run .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query.html","id":null,"dir":"Reference","previous_headings":"","what":"Construct outpack query — orderly_query","title":"Construct outpack query — orderly_query","text":"Construct outpack query, typically passed orderly_search","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Construct outpack query — orderly_query","text":"","code":"orderly_query(expr, name = NULL, scope = NULL, subquery = NULL)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Construct outpack query — orderly_query","text":"expr query expression. NULL expression matches everything. name Optionally, name packet scope query . intersected scope arg shorthand way running scope = list(name = \"name\") scope Optionally, scope query limit packets searched pars subquery Optionally, named list subqueries can referenced name expr.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Construct outpack query — orderly_query","text":"orderly_query object, modified, can passed orderly_search()","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query_explain.html","id":null,"dir":"Reference","previous_headings":"","what":"Explain a query — orderly_query_explain","title":"Explain a query — orderly_query_explain","text":"Explain query matched. experimental output change. moment, can tell query matches, fails match based one number &&-ed together clauses.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query_explain.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Explain a query — orderly_query_explain","text":"","code":"orderly_query_explain(   ...,   parameters = NULL,   envir = parent.frame(),   options = NULL,   root = NULL )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query_explain.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Explain a query — orderly_query_explain","text":"... Arguments passed orderly_query, perhaps just query expression parameters Optionally, named list parameters substitute query (using : prefix) envir Optionally, environment substitute query (using environment: prefix). default use calling environment, can explicitly pass want control lookup happens. options Optionally, orderly_search_options object controlling search performed, packets considered scope. provided, default options used (.e., orderly2::orderly_search_options()) root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query_explain.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Explain a query — orderly_query_explain","text":"object class orderly_query_explain, can inspected (contents subject change) print method show user-friendly summary query result.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_resource.html","id":null,"dir":"Reference","previous_headings":"","what":"Declare orderly resources — orderly_resource","title":"Declare orderly resources — orderly_resource","text":"Declare file, group files, orderly resource. explicitly declaring files resources orderly mark files immutable inputs validate analysis modify run orderly_run()","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_resource.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Declare orderly resources — orderly_resource","text":"","code":"orderly_resource(files)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_resource.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Declare orderly resources — orderly_resource","text":"files number names files","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_resource.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Declare orderly resources — orderly_resource","text":"Invisibly, character vector resources included call. rely order files expanded directories, likely platform dependent.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":null,"dir":"Reference","previous_headings":"","what":"Run a report — orderly_run","title":"Run a report — orderly_run","text":"Run report.  create new directory drafts/<reportname>, copy declared resources , run script check expected artefacts created.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Run a report — orderly_run","text":"","code":"orderly_run(   name,   parameters = NULL,   envir = NULL,   echo = TRUE,   search_options = NULL,   root = NULL,   locate = TRUE )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Run a report — orderly_run","text":"name Name report run. leading ./ src/ trailing / path parts removed (e.g., added autocomplete). parameters Parameters passed report. named list parameters declared orderly.yml.  parameter must scalar character, numeric, integer logical. envir environment used evaluate report script; default use global environment, may always wanted. echo Optional logical control printing output source() console. search_options Optional control locations, used orderly_dependency; converted orderly_search_options object, see Details. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, just outpack (see orderly_init details). locate Logical, indicating configuration searched .  TRUE config given, orderly looks working directory parents finds .outpack directory","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Run a report — orderly_run","text":"id created report (string)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"locations-used-in-dependency-resolution","dir":"Reference","previous_headings":"","what":"Locations used in dependency resolution","title":"Run a report — orderly_run","text":"packet depends packets, want control locations used find appropriate packets. control passed function argument orderly_dependency property way packet created packet ; importantly different users may different names locations makes little sense encode location name source code. Alternatively, want use different locations different contexts (initial development want include local copies packets possible dependencies vs resolving dependencies resolved one locations! Similarly, might want include packets known locations currently downloaded onto machine - pulling packets take anything seconds hours depending size speed network connection (pulling packets mean packet fails run). allow control can pass argument search_options, orderly_search_options object, allows control names locations use, whether metadata refreshed pull anything packets currently downloaded considered candidates. effect running interactively, case can specify search options (root specific) orderly_interactive_set_search_options","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"which-packets-might-be-selected-from-locations-","dir":"Reference","previous_headings":"","what":"Which packets might be selected from locations?","title":"Run a report — orderly_run","text":"search_options argument controls outpack searches packets given query anything might moved network (one outpack archive another). default everything resolved locally ; can depend packets unpacked within current archive.  pass search_options argument contains allow_remote = TRUE (see orderly_search_options packets known anywhere candidates using dependencies needed pull resolved files remote location. Note even packet locally present might needed - content anywhere else unpacked packet reuse content without re-fetching. pull_metadata = TRUE, refresh location metadata pulling, location argument controls locations pulled .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"equivalence-to-the-old-use-draft-option","dir":"Reference","previous_headings":"","what":"Equivalence to the old use_draft option","title":"Run a report — orderly_run","text":"location handling generalises orderly (v1)'s old use_draft option, terms location argument orderly2::orderly_search_options`: use_draft = TRUE location = \"local\" use_draft = FALSE location = c(...) provide locations except local (setdiff(orderly2::orderly_location_list(), \"local\")) use_draft = \"newer\" location = NULL (last option one people preferred new default behaviour). addition, resolve dependencies currently exist production right now options:   updates current metadata production, runs queries packets known remote, depends even (yet) locally.  functionality never available orderly version 1, though intended support .","code":"location = \"production\", pull_metadata = TRUE, require_unpacked = FALSE"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"running-with-a-source-tree-separate-from-outpack-root","dir":"Reference","previous_headings":"","what":"Running with a source tree separate from outpack root","title":"Run a report — orderly_run","text":"Sometimes useful run things different place disk outpack root. know two cases come : running reports within runner server, make clean clone source tree particular git reference new temporary directory run report , insert orderly repo fixed non-temporary location. user convenient torun report hard drive store archive metadata (larger) shared drive. first instance, source path <src> contains file orderly_config.yml directory src/ source reports, separate path <root> contains directory .outpack/ metadata - may also unpacked archive, .git/ directory depending configuration. (Later make sense support \"bare\" outpack layout.)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"manually-setting-report-source-directory","dir":"Reference","previous_headings":"","what":"Manually setting report source directory","title":"Run a report — orderly_run","text":"manually set report source directory, need set path directory ORDERLY_REPORT_SRC environment variable.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Run a report — orderly_run","text":"","code":"# Create a simple example: path <- orderly2::orderly_example(\"default\") #> ✔ Created orderly root at '/tmp/RtmpTXa1jo/file170f1f0afcb1'  # Run the 'data' task: orderly2::orderly_run(\"data\", root = path) #> Error: Directory does not exist: '/tmp/RtmpTXa1jo/file170f1f0afcb1'  # After running, a finished packet appears in the archive: fs::dir_tree(path) #> Error: [ENOENT] Failed to search directory '/tmp/RtmpTXa1jo/file170f1f0afcb1': no such file or directory  # and we can query the metadata: orderly2::orderly_metadata_extract(name = \"data\", root = path) #> Error: Directory does not exist: '/tmp/RtmpTXa1jo/file170f1f0afcb1'"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run_info.html","id":null,"dir":"Reference","previous_headings":"","what":"Information about currently running report — orderly_run_info","title":"Information about currently running report — orderly_run_info","text":"Fetch information actively running report.  allows reflect information report back part report, example embedding current report id, information computed dependencies. information slightly different format orderly version 1.x (currently) include information dependencies run outside orderly_run, never reliable previously.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run_info.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Information about currently running report — orderly_run_info","text":"","code":"orderly_run_info()"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run_info.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Information about currently running report — orderly_run_info","text":"list elements name: name current report id: id current report, NA running interactively root: orderly root path depends: data frame information dependencies (available interactively) index: integer sequence along calls orderly_dependency name: name dependency query: query used find dependency id: computed id included packet filename: file used packet : filename used locally","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search.html","id":null,"dir":"Reference","previous_headings":"","what":"Query outpack's database — orderly_search","title":"Query outpack's database — orderly_search","text":"Evaluate query outpack database, returning vector matching packet ids.  Note default searches packets unpacked available direct use computer; search within packets known locations (might know via metadata) need use options argument.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Query outpack's database — orderly_search","text":"","code":"orderly_search(   ...,   parameters = NULL,   envir = parent.frame(),   options = NULL,   root = NULL )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Query outpack's database — orderly_search","text":"... Arguments passed orderly_query, perhaps just query expression parameters Optionally, named list parameters substitute query (using : prefix) envir Optionally, environment substitute query (using environment: prefix). default use calling environment, can explicitly pass want control lookup happens. options Optionally, orderly_search_options object controlling search performed, packets considered scope. provided, default options used (.e., orderly2::orderly_search_options()) root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Query outpack's database — orderly_search","text":"character vector matching ids. case match query returning single value (e.g., latest(...) single(...)) character missing value (NA_character_)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search_options.html","id":null,"dir":"Reference","previous_headings":"","what":"Packet search options — orderly_search_options","title":"Packet search options — orderly_search_options","text":"Options controlling packet searches carried , example via orderly_search orderly_run. details never included metadata alongside query (, part query even though affect ).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search_options.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Packet search options — orderly_search_options","text":"","code":"orderly_search_options(   location = NULL,   allow_remote = FALSE,   pull_metadata = FALSE )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search_options.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Packet search options — orderly_search_options","text":"location Optional vector locations pull . might future expand allow wildcards exceptions. allow_remote Logical, indicating allow packets found currently unpacked (.e., known location metadata ). TRUE, conjunction orderly_dependency might pull large quantity data. pull_metadata Logical, indicating pull metadata immediately search. location given, pass orderly_location_pull_metadata filter locations update.  pulling many packets sequence, want update option FALSE first pull.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search_options.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Packet search options — orderly_search_options","text":"object class orderly_search_options modified creation (see note pull_metadata)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_shared_resource.html","id":null,"dir":"Reference","previous_headings":"","what":"Copy shared resources into a packet directory — orderly_shared_resource","title":"Copy shared resources into a packet directory — orderly_shared_resource","text":"Copy shared resources packet directory. can use share common resources (data code) multiple packets. Additional metadata added keep track files came .  Using function requires shared resources directory shared/ exists orderly root; error raised configured attempt fetch files.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_shared_resource.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Copy shared resources into a packet directory — orderly_shared_resource","text":"","code":"orderly_shared_resource(...)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_shared_resource.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Copy shared resources into a packet directory — orderly_shared_resource","text":"... shared resources copy. arguments named, name destination file value filename within shared resource directory. can use limited form string interpolation names argument; using ${variable} pick values envir substitute string.  similar interpolation might familiar glue::glue similar, much simpler concatenation fancy features supported.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_shared_resource.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Copy shared resources into a packet directory — orderly_shared_resource","text":"Invisibly, data.frame columns (fileames copied running packet) (filenames within shared/).  orderly_resource, rely ordering directory expansion performed.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_strict_mode.html","id":null,"dir":"Reference","previous_headings":"","what":"Enable orderly strict mode — orderly_strict_mode","title":"Enable orderly strict mode — orderly_strict_mode","text":"Put orderly2 \"strict mode\", closer defaults orderly 1.0.0; mode explicitly included files (via orderly_resource orderly_shared_resource) copied running packet, warn unexpected files end run.  Using strict mode allows orderly2 aggressive deletes files within source directory, accurate reports , faster start packets developing interactively.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_strict_mode.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Enable orderly strict mode — orderly_strict_mode","text":"","code":"orderly_strict_mode()"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_strict_mode.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Enable orderly strict mode — orderly_strict_mode","text":"Undefined","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_strict_mode.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Enable orderly strict mode — orderly_strict_mode","text":"future, may extend strict mode allow requiring computation occurs within orderly functions (.e., requirements run packet fully known actually running ).  likely default behaviour orderly_strict_mode gain argument. allow server processes either override value (enabling even explicitly given) /require .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_validate_archive.html","id":null,"dir":"Reference","previous_headings":"","what":"Validate unpacked packets. — orderly_validate_archive","title":"Validate unpacked packets. — orderly_validate_archive","text":"Validate unpacked packets. time, expect function become fully featured, validating .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_validate_archive.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Validate unpacked packets. — orderly_validate_archive","text":"","code":"orderly_validate_archive(..., action = \"inform\", root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_validate_archive.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Validate unpacked packets. — orderly_validate_archive","text":"... Either arguments search can constructed (useful options include name = \"something\"), character vector ids, leave empty validate everything. action action take finding invalid packet. See Details. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_validate_archive.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Validate unpacked packets. — orderly_validate_archive","text":"Invisibly, character vector repaired (invalid) packets.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_validate_archive.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Validate unpacked packets. — orderly_validate_archive","text":"actions can take finding invalid packet : inform (default): just print information problem orphan: mark packet orphaned within metadata, touch files archive (default directory archive/) - safe option leave consistent state without deleting anything. delete: addition marking packet orphan, also delete files archive. Later, add \"repair\" option try fix broken packets. validation interacts option core.require_complete_tree; option TRUE, packet valid (recursive) dependencies also valid, action apply packets also upstream dependencies invalidated.  validation happen even query implied ... include packets complete tree required. validation also interact core.use_file_store repair supported, becomes trivial.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":null,"dir":"Reference","previous_headings":"","what":"Query index — query_index","title":"Query index — query_index","text":"Class managing active index whilst evaluating query","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"public-fields","dir":"Reference","previous_headings":"","what":"Public fields","title":"Query index — query_index","text":"index packet index depends Named list data frames. Names packet ids, values packets depended packet id (.e. parents). uses Named list data frames. Names packet ids, values packets used packet id (.e. children). root outpack root object","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"public-methods","dir":"Reference","previous_headings":"","what":"Public methods","title":"Query index — query_index","text":"query_index$new() query_index$get_packet_depends() query_index$get_packet_uses()","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"method-new-","dir":"Reference","previous_headings":"","what":"Method new()","title":"Query index — query_index","text":"Create new query_index object","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Query index — query_index","text":"","code":"query_index$new(root, index, depends, uses)"},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Query index — query_index","text":"root outpack root object index packet index data frame depends Named list data frames. Names packet ids, values packets depended packet id (.e. parents). uses Named list data frames. Names packet ids, values packets used packet id (.e. children). data depends relationships flow direction.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"method-get-packet-depends-","dir":"Reference","previous_headings":"","what":"Method get_packet_depends()","title":"Query index — query_index","text":"Get ids packets packet depends specified level","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"usage-1","dir":"Reference","previous_headings":"","what":"Usage","title":"Query index — query_index","text":"","code":"query_index$get_packet_depends(id, depth)"},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"arguments-1","dir":"Reference","previous_headings":"","what":"Arguments","title":"Query index — query_index","text":"id id packet get parents depth Depth parents get, depth 1 gets immediate parents depth 2 gets parents parents parents, depth Inf recurse whole tree get parents","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"returns","dir":"Reference","previous_headings":"","what":"Returns","title":"Query index — query_index","text":"ids parents packet","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"method-get-packet-uses-","dir":"Reference","previous_headings":"","what":"Method get_packet_uses()","title":"Query index — query_index","text":"Get ids packets used packet","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"usage-2","dir":"Reference","previous_headings":"","what":"Usage","title":"Query index — query_index","text":"","code":"query_index$get_packet_uses(id, depth)"},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"arguments-2","dir":"Reference","previous_headings":"","what":"Arguments","title":"Query index — query_index","text":"id id packet get children depth Depth children get, depth 1 gets immediate children depth 2 gets children children children, depth Inf recurse whole tree get children","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"returns-1","dir":"Reference","previous_headings":"","what":"Returns","title":"Query index — query_index","text":"ids children packet","code":""}]