diff --git a/404.html b/404.html
index 7106f18d..9227836e 100644
--- a/404.html
+++ b/404.html
@@ -20,7 +20,7 @@
 
     <a class="navbar-brand me-2" href="https://mrc-ide.github.io/orderly2/index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/LICENSE-text.html b/LICENSE-text.html
index 402528ec..2abf38ce 100644
--- a/LICENSE-text.html
+++ b/LICENSE-text.html
@@ -7,7 +7,7 @@
 
     <a class="navbar-brand me-2" href="index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/articles/collaboration.html b/articles/collaboration.html
index 1db049a5..5ebef96e 100644
--- a/articles/collaboration.html
+++ b/articles/collaboration.html
@@ -20,7 +20,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -193,7 +193,7 @@ <h2 id="an-example">An example<a class="anchor" aria-label="anchor" href="#an-ex
 <p>Alice needs to run <code>orderly::orderly_init()</code> first:</p>
 <div class="sourceCode" id="cb3"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_init.html">orderly_init</a></span><span class="op">(</span><span class="st">"."</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpYA62oL/file1aaa1c2bcfff/alice'</span></span>
+<span><span class="co">## <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/Rtmp4tLdTj/file1a7d6ec6f83/alice'</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Wrote '.gitignore'</span></span>
 <span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_list_src.html">orderly_list_src</a></span><span class="op">(</span><span class="op">)</span></span>
 <span><span class="co">## [1] "data"</span></span></code></pre></div>
@@ -206,11 +206,11 @@ <h2 id="an-example">An example<a class="anchor" aria-label="anchor" href="#an-ex
 <p>Once done, she can run any analysis:</p>
 <div class="sourceCode" id="cb5"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="va">id</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"data"</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240708-173319-f5e76d59` at 2024-07-08 17:33:19.96726</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240710-102605-babfcf9e` at 2024-07-10 10:26:05.736024</span></span>
 <span><span class="co">## &gt; orderly2::orderly_artefact("Final data", "data.rds")</span></span>
 <span><span class="co">## &gt; saveRDS(mtcars, "data.rds")</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">data.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240708-173319-f5e76d59 at 2024-07-08 17:33:20.026166 (0.05890656 secs)</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240710-102605-babfcf9e at 2024-07-10 10:26:05.793755 (0.05773139 secs)</span></span></code></pre></div>
 <p>Perhaps it takes several goes for Alice to be happy with the
 analysis, but at some point she has something ready to share. She can
 then “push” the final packet up onto their server:</p>
@@ -220,7 +220,7 @@ <h2 id="an-example">An example<a class="anchor" aria-label="anchor" href="#an-ex
 initialised, and the server added:</p>
 <div class="sourceCode" id="cb7"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_init.html">orderly_init</a></span><span class="op">(</span><span class="st">"."</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpYA62oL/file1aaa1c2bcfff/bob'</span></span>
+<span><span class="co">## <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/Rtmp4tLdTj/file1a7d6ec6f83/bob'</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Wrote '.gitignore'</span></span></code></pre></div>
 <div class="sourceCode" id="cb8"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_location_add.html">orderly_location_add</a></span><span class="op">(</span></span>
@@ -234,7 +234,7 @@ <h2 id="an-example">An example<a class="anchor" aria-label="anchor" href="#an-ex
 <span>  name <span class="op">=</span> <span class="st">"data"</span>,</span>
 <span>  options <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/list.html" class="external-link">list</a></span><span class="op">(</span>allow_remote <span class="op">=</span> <span class="cn">TRUE</span>, pull_metadata <span class="op">=</span> <span class="cn">TRUE</span><span class="op">)</span><span class="op">)</span></span>
 <span><span class="co">##                         id name parameters</span></span>
-<span><span class="co">## 1 20240708-173319-f5e76d59 data</span></span></code></pre></div>
+<span><span class="co">## 1 20240710-102605-babfcf9e data</span></span></code></pre></div>
 <p>Having seen there is a new “data” packet here, he can pull this down
 locally (TODO: mrc-4414 makes this nicer):</p>
 <div class="sourceCode" id="cb10"><pre class="downlit sourceCode r">
@@ -242,10 +242,10 @@ <h2 id="an-example">An example<a class="anchor" aria-label="anchor" href="#an-ex
 <span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Looking for suitable files already on disk</span></span>
 <span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Need to fetch 2 files (1.3 kB) from 1 location</span></span>
 <span><span class="co">## ⠙ Fetching file 1/2 (81 B) from 'server' | ETA:  0s [3ms]</span></span>
-<span><span class="co">## <span style="color: #00BB00;">✔</span> Fetched 2 files (1.3 kB) from 'server' in 21ms.</span></span>
+<span><span class="co">## <span style="color: #00BB00;">✔</span> Fetched 2 files (1.3 kB) from 'server' in 20ms.</span></span>
 <span><span class="co">## </span></span></code></pre></div>
 <p>Now Bob is in a position to develop against the same packet that
-Alice ran (20240708-173319-f5e76d59)</p>
+Alice ran (20240710-102605-babfcf9e)</p>
 </div>
 <div class="section level2">
 <h2 id="possible-working-patterns">Possible working patterns<a class="anchor" aria-label="anchor" href="#possible-working-patterns"></a>
@@ -402,7 +402,7 @@ <h2 id="sharing-packets-with-collaborators-using-a-shared-file-system">Sharing p
 <span>  use_file_store <span class="op">=</span> <span class="cn">TRUE</span>, </span>
 <span>  require_complete_tree <span class="op">=</span> <span class="cn">TRUE</span></span>
 <span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpYA62oL/file1aaa1c2bcfff/sharepoint'</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/Rtmp4tLdTj/file1a7d6ec6f83/sharepoint'</span></span></code></pre></div>
 <p>Create an orderly store with a file store and a complete tree. See
 <code><a href="../reference/orderly_init.html">orderly2::orderly_init()</a></code> for more details.</p>
 </li>
diff --git a/articles/dependencies.html b/articles/dependencies.html
index 2d57edbe..bec3fcdf 100644
--- a/articles/dependencies.html
+++ b/articles/dependencies.html
@@ -20,7 +20,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -138,16 +138,16 @@ <h3 id="basic-use">Basic use<a class="anchor" aria-label="anchor" href="#basic-u
 <code><a href="../articles/introduction.html">vignette("introduction")</a></code>, to get us started).</p>
 <div class="sourceCode" id="cb3"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="va">id1</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"data"</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240708-173323-6fc634f2` at 2024-07-08 17:33:23.443321</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240710-102609-362a30fc` at 2024-07-10 10:26:09.219197</span></span>
 <span><span class="co">## &gt; d &lt;- read.csv("data.csv")</span></span>
 <span><span class="co">## &gt; d$z &lt;- resid(lm(y ~ x, d))</span></span>
 <span><span class="co">## &gt; saveRDS(d, "data.rds")</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">data.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240708-173323-6fc634f2 at 2024-07-08 17:33:23.479339 (0.03601766 secs)</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240710-102609-362a30fc at 2024-07-10 10:26:09.254927 (0.03572965 secs)</span></span>
 <span><span class="va">id2</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"analysis"</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">analysis</span>' `20240708-173323-81eaf8d3` at 2024-07-08 17:33:23.513865</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">analysis</span>' `20240710-102609-483418bb` at 2024-07-10 10:26:09.287808</span></span>
 <span><span class="co">## &gt; orderly2::orderly_dependency("data", "latest()", "data.rds")</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Depending on <span style="color: #0000BB;">data</span> @ `20240708-173323-6fc634f2` (via latest(name == "data"))</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Depending on <span style="color: #0000BB;">data</span> @ `20240710-102609-362a30fc` (via latest(name == "data"))</span></span>
 <span><span class="co">## &gt; d &lt;- readRDS("data.rds")</span></span>
 <span><span class="co">## &gt; png("analysis.png")</span></span>
 <span><span class="co">## &gt; plot(y ~ x, d)</span></span>
@@ -155,14 +155,14 @@ <h3 id="basic-use">Basic use<a class="anchor" aria-label="anchor" href="#basic-u
 <span><span class="co">## agg_png </span></span>
 <span><span class="co">##       2</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">analysis.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240708-173323-81eaf8d3 at 2024-07-08 17:33:23.700104 (0.1862397 secs)</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240710-102609-483418bb at 2024-07-10 10:26:09.380892 (0.09308434 secs)</span></span></code></pre></div>
 <p>When we look at the metadata for the packet created from the
 <code>analysis</code> report, we can see it has used
-<code>20240708-173323-6fc634f2</code> as its dependency:</p>
+<code>20240710-102609-362a30fc</code> as its dependency:</p>
 <div class="sourceCode" id="cb4"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_metadata.html">orderly_metadata</a></span><span class="op">(</span><span class="va">id2</span><span class="op">)</span><span class="op">$</span><span class="va">depends</span></span>
 <span><span class="co">##                     packet                  query        files</span></span>
-<span><span class="co">## 1 20240708-173323-6fc634f2 latest(name == "data") data.rds....</span></span></code></pre></div>
+<span><span class="co">## 1 20240710-102609-362a30fc latest(name == "data") data.rds....</span></span></code></pre></div>
 <p>(indeed it had to, there is only one copy of the <code>data</code>
 packet to pick from).</p>
 </div>
@@ -187,35 +187,35 @@ <h3 id="filtering-candidates-by-parameters">Filtering candidates by parameters<a
 <p>We can run this for several values of <code>cyl</code>:</p>
 <div class="sourceCode" id="cb7"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"data"</span>, <span class="fu"><a href="https://rdrr.io/r/base/list.html" class="external-link">list</a></span><span class="op">(</span>cyl <span class="op">=</span> <span class="fl">4</span><span class="op">)</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240708-173324-17521339` at 2024-07-08 17:33:24.094335</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240710-102609-c0708a99` at 2024-07-10 10:26:09.754879</span></span>
 <span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Parameters:</span></span>
 <span><span class="co">## • cyl: 4</span></span>
 <span><span class="co">## &gt; orderly2::orderly_parameters(cyl = NULL)</span></span>
 <span><span class="co">## &gt; d &lt;- mtcars[mtcars$cyl == cyl, ]</span></span>
 <span><span class="co">## &gt; saveRDS(d, "data.rds")</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">data.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240708-173324-17521339 at 2024-07-08 17:33:24.126508 (0.0321734 secs)</span></span>
-<span><span class="co">## [1] "20240708-173324-17521339"</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240710-102609-c0708a99 at 2024-07-10 10:26:09.786579 (0.03170037 secs)</span></span>
+<span><span class="co">## [1] "20240710-102609-c0708a99"</span></span>
 <span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"data"</span>, <span class="fu"><a href="https://rdrr.io/r/base/list.html" class="external-link">list</a></span><span class="op">(</span>cyl <span class="op">=</span> <span class="fl">6</span><span class="op">)</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240708-173324-2529100c` at 2024-07-08 17:33:24.148738</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240710-102609-cde98f1c` at 2024-07-10 10:26:09.807431</span></span>
 <span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Parameters:</span></span>
 <span><span class="co">## • cyl: 6</span></span>
 <span><span class="co">## &gt; orderly2::orderly_parameters(cyl = NULL)</span></span>
 <span><span class="co">## &gt; d &lt;- mtcars[mtcars$cyl == cyl, ]</span></span>
 <span><span class="co">## &gt; saveRDS(d, "data.rds")</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">data.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240708-173324-2529100c at 2024-07-08 17:33:24.179934 (0.03119636 secs)</span></span>
-<span><span class="co">## [1] "20240708-173324-2529100c"</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240710-102609-cde98f1c at 2024-07-10 10:26:09.835665 (0.02823305 secs)</span></span>
+<span><span class="co">## [1] "20240710-102609-cde98f1c"</span></span>
 <span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"data"</span>, <span class="fu"><a href="https://rdrr.io/r/base/list.html" class="external-link">list</a></span><span class="op">(</span>cyl <span class="op">=</span> <span class="fl">8</span><span class="op">)</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240708-173324-336c93ff` at 2024-07-08 17:33:24.203909</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240710-102609-db26b192` at 2024-07-10 10:26:09.859044</span></span>
 <span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Parameters:</span></span>
 <span><span class="co">## • cyl: 8</span></span>
 <span><span class="co">## &gt; orderly2::orderly_parameters(cyl = NULL)</span></span>
 <span><span class="co">## &gt; d &lt;- mtcars[mtcars$cyl == cyl, ]</span></span>
 <span><span class="co">## &gt; saveRDS(d, "data.rds")</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">data.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240708-173324-336c93ff at 2024-07-08 17:33:24.230692 (0.02678275 secs)</span></span>
-<span><span class="co">## [1] "20240708-173324-336c93ff"</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240710-102609-db26b192 at 2024-07-10 10:26:09.885346 (0.02630186 secs)</span></span>
+<span><span class="co">## [1] "20240710-102609-db26b192"</span></span></code></pre></div>
 <p>Our follow-on analysis contains:</p>
 <div class="sourceCode" id="cb8"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_parameters.html">orderly_parameters</a></span><span class="op">(</span>cyl <span class="op">=</span> <span class="cn">NULL</span><span class="op">)</span></span>
@@ -233,7 +233,7 @@ <h3 id="filtering-candidates-by-parameters">Filtering candidates by parameters<a
 currently running report (<code>this:cyl</code>).</p>
 <div class="sourceCode" id="cb9"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"analysis"</span>, <span class="fu"><a href="https://rdrr.io/r/base/list.html" class="external-link">list</a></span><span class="op">(</span>cyl <span class="op">=</span> <span class="fl">4</span><span class="op">)</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">analysis</span>' `20240708-173324-62818858` at 2024-07-08 17:33:24.387952</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">analysis</span>' `20240710-102610-0981d7cb` at 2024-07-10 10:26:10.04024</span></span>
 <span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Parameters:</span></span>
 <span><span class="co">## • cyl: 4</span></span>
 <span><span class="co">## &gt; orderly2::orderly_parameters(cyl = NULL)</span></span>
@@ -241,7 +241,7 @@ <h3 id="filtering-candidates-by-parameters">Filtering candidates by parameters<a
 <span><span class="co">## +   "data",</span></span>
 <span><span class="co">## +   "latest(parameter:cyl == this:cyl)",</span></span>
 <span><span class="co">## +   "data.rds")</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Depending on <span style="color: #0000BB;">data</span> @ `20240708-173324-17521339` (via latest(parameter:cyl == this:cyl &amp;&amp; name == "data"))</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Depending on <span style="color: #0000BB;">data</span> @ `20240710-102609-c0708a99` (via latest(parameter:cyl == this:cyl &amp;&amp; name == "data"))</span></span>
 <span><span class="co">## &gt; d &lt;- readRDS("data.rds")</span></span>
 <span><span class="co">## &gt; png("analysis.png")</span></span>
 <span><span class="co">## &gt; plot(mpg ~ disp, d)</span></span>
@@ -249,8 +249,8 @@ <h3 id="filtering-candidates-by-parameters">Filtering candidates by parameters<a
 <span><span class="co">## agg_png </span></span>
 <span><span class="co">##       2</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">analysis.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240708-173324-62818858 at 2024-07-08 17:33:24.44188 (0.05392814 secs)</span></span>
-<span><span class="co">## [1] "20240708-173324-62818858"</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240710-102610-0981d7cb at 2024-07-10 10:26:10.092804 (0.05256438 secs)</span></span>
+<span><span class="co">## [1] "20240710-102610-0981d7cb"</span></span></code></pre></div>
 </div>
 <div class="section level3">
 <h3 id="interpreting-errors">Interpreting errors<a class="anchor" aria-label="anchor" href="#interpreting-errors"></a>
@@ -258,7 +258,7 @@ <h3 id="interpreting-errors">Interpreting errors<a class="anchor" aria-label="an
 <p>If your query fails to resolve a candidate it will error:</p>
 <div class="sourceCode" id="cb10"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"analysis"</span>, <span class="fu"><a href="https://rdrr.io/r/base/list.html" class="external-link">list</a></span><span class="op">(</span>cyl <span class="op">=</span> <span class="fl">9000</span><span class="op">)</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">analysis</span>' `20240708-173324-8673176b` at 2024-07-08 17:33:24.528374</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">analysis</span>' `20240710-102610-2c69ef18` at 2024-07-10 10:26:10.17657</span></span>
 <span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Parameters:</span></span>
 <span><span class="co">## • cyl: 9000</span></span>
 <span><span class="co">## &gt; orderly2::orderly_parameters(cyl = NULL)</span></span>
@@ -267,7 +267,7 @@ <h3 id="interpreting-errors">Interpreting errors<a class="anchor" aria-label="an
 <span><span class="co">## +   "latest(parameter:cyl == this:cyl)",</span></span>
 <span><span class="co">## +   "data.rds")</span></span>
 <span><span class="co">## <span style="color: #BB0000;">✖</span> Error running <span style="color: #0000BB;">analysis.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240708-173324-8673176b at 2024-07-08 17:33:24.59256 (0.06418586 secs)</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240710-102610-2c69ef18 at 2024-07-10 10:26:10.238994 (0.06242466 secs)</span></span>
 <span><span class="co">## <span style="color: #BBBB00; font-weight: bold;">Error</span><span style="font-weight: bold;"> in `orderly2::orderly_run()`:</span></span></span>
 <span><span class="co">## <span style="color: #BBBB00;">!</span> Failed to run report</span></span>
 <span><span class="co">## <span style="font-weight: bold;">Caused by error in `outpack_packet_use_dependency()`:</span></span></span>
@@ -324,8 +324,8 @@ <h3 id="interpreting-errors">Interpreting errors<a class="anchor" aria-label="an
 <span><span class="co">## [1] 3</span></span>
 <span><span class="co">## </span></span>
 <span><span class="co">## $found</span></span>
-<span><span class="co">## [1] "20240708-173324-17521339" "20240708-173324-2529100c"</span></span>
-<span><span class="co">## [3] "20240708-173324-336c93ff"</span></span></code></pre></div>
+<span><span class="co">## [1] "20240710-102609-c0708a99" "20240710-102609-cde98f1c"</span></span>
+<span><span class="co">## [3] "20240710-102609-db26b192"</span></span></code></pre></div>
 <p>(this would have worked with
 <code>rlang::last_error()$explanation$parts$A</code> too).</p>
 <p>You can also use <code><a href="../reference/orderly_metadata_extract.html">orderly2::orderly_metadata_extract</a></code> to
@@ -335,9 +335,9 @@ <h3 id="interpreting-errors">Interpreting errors<a class="anchor" aria-label="an
 <span>  name <span class="op">=</span> <span class="st">"data"</span>,</span>
 <span>  extract <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html" class="external-link">c</a></span><span class="op">(</span>cyl <span class="op">=</span> <span class="st">"parameters.cyl is number"</span><span class="op">)</span><span class="op">)</span></span>
 <span><span class="co">##                         id cyl</span></span>
-<span><span class="co">## 1 20240708-173324-17521339   4</span></span>
-<span><span class="co">## 2 20240708-173324-2529100c   6</span></span>
-<span><span class="co">## 3 20240708-173324-336c93ff   8</span></span></code></pre></div>
+<span><span class="co">## 1 20240710-102609-c0708a99   4</span></span>
+<span><span class="co">## 2 20240710-102609-cde98f1c   6</span></span>
+<span><span class="co">## 3 20240710-102609-db26b192   8</span></span></code></pre></div>
 </div>
 <div class="section level3">
 <h3 id="filtering-candidates-in-other-ways">Filtering candidates in other ways<a class="anchor" aria-label="anchor" href="#filtering-candidates-in-other-ways"></a>
@@ -351,7 +351,7 @@ <h3 id="filtering-candidates-in-other-ways">Filtering candidates in other ways<a
 <code>latest(parameter:cyl == environment:cyl)</code> to match against
 whatever value <code>cyl</code> took in the evaluating environment.</p>
 <p>Instead of a query, you can provide a single id (e.g,
-<code>20240708-173324-62818858</code>), which would mean that even as
+<code>20240710-102610-0981d7cb</code>), which would mean that even as
 new copies of the <code>data</code> packet are created, this dependency
 will always resolve to the same value.</p>
 <p>You can chain together logical operations with
diff --git a/articles/index.html b/articles/index.html
index cd37b638..c571cb89 100644
--- a/articles/index.html
+++ b/articles/index.html
@@ -7,7 +7,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/articles/introduction.html b/articles/introduction.html
index 39c7ca62..f842f9be 100644
--- a/articles/introduction.html
+++ b/articles/introduction.html
@@ -20,7 +20,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -107,7 +107,7 @@ <h2 id="creating-an-empty-orderly-repository">Creating an empty orderly reposito
 <div class="sourceCode" id="cb2"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="va">path</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/tempfile.html" class="external-link">tempfile</a></span><span class="op">(</span><span class="op">)</span> <span class="co"># we'll use a temporary directory here - see note below</span></span>
 <span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_init.html">orderly_init</a></span><span class="op">(</span><span class="va">path</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmplJ4AWR/file1b3524c63db5'</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/Rtmpv6E3fl/file1b083976469a'</span></span></code></pre></div>
 <p>which creates a few files:</p>
 <pre><code><span><span class="co">## <span style="color: #0000BB; font-weight: bold;">.</span></span></span>
 <span><span class="co">## ├── <span style="color: #0000BB; font-weight: bold;">.outpack</span></span></span>
@@ -169,14 +169,14 @@ <h2 id="creating-your-first-orderly-report">Creating your first orderly report<a
 <code><a href="../reference/orderly_run.html">orderly2::orderly_run()</a></code>:</p>
 <div class="sourceCode" id="cb8"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="va">id</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"incoming_data"</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">incoming_data</span>' `20240708-173327-dd6e3856` at 2024-07-08 17:33:27.869865</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">incoming_data</span>' `20240710-102613-8bebab5b` at 2024-07-10 10:26:13.55152</span></span>
 <span><span class="co">## &gt; d &lt;- read.csv("data.csv")</span></span>
 <span><span class="co">## &gt; d$z &lt;- resid(lm(y ~ x, d))</span></span>
 <span><span class="co">## &gt; saveRDS(d, "data.rds")</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">incoming_data.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240708-173327-dd6e3856 at 2024-07-08 17:33:27.932913 (0.06304812 secs)</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240710-102613-8bebab5b at 2024-07-10 10:26:13.616706 (0.06518626 secs)</span></span>
 <span><span class="va">id</span></span>
-<span><span class="co">## [1] "20240708-173327-dd6e3856"</span></span></code></pre></div>
+<span><span class="co">## [1] "20240710-102613-8bebab5b"</span></span></code></pre></div>
 <p>The <code>id</code> that is created is a new identifier for the
 packet that will be both unique among all packets (within reason) and
 chronologically sortable. A packet that has an id that sorts after
@@ -185,7 +185,7 @@ <h2 id="creating-your-first-orderly-report">Creating your first orderly report<a
 <pre><code><span><span class="co">## <span style="color: #0000BB; font-weight: bold;">.</span></span></span>
 <span><span class="co">## ├── <span style="color: #0000BB; font-weight: bold;">archive</span></span></span>
 <span><span class="co">## │   └── <span style="color: #0000BB; font-weight: bold;">incoming_data</span></span></span>
-<span><span class="co">## │       └── <span style="color: #0000BB; font-weight: bold;">20240708-173327-dd6e3856</span></span></span>
+<span><span class="co">## │       └── <span style="color: #0000BB; font-weight: bold;">20240710-102613-8bebab5b</span></span></span>
 <span><span class="co">## │           ├── data.csv</span></span>
 <span><span class="co">## │           ├── data.rds</span></span>
 <span><span class="co">## │           └── <span style="color: #00BB00;">incoming_data.R</span></span></span>
@@ -198,7 +198,7 @@ <h2 id="creating-your-first-orderly-report">Creating your first orderly report<a
 <span><span class="co">##         └── <span style="color: #00BB00;">incoming_data.R</span></span></span></code></pre>
 <p>A few things have changed here:</p>
 <ul>
-<li>we have a directory archive/incoming_data/20240708-173327-dd6e3856;
+<li>we have a directory archive/incoming_data/20240710-102613-8bebab5b;
 this directory contains
 <ul>
 <li>the file that was created when we ran the report
@@ -270,10 +270,10 @@ <h2 id="depending-on-packets-from-another-report">Depending on packets from anot
 <code><a href="../reference/orderly_run.html">orderly2::orderly_run()</a></code>:</p>
 <div class="sourceCode" id="cb13"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="va">id</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"analysis"</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">analysis</span>' `20240708-173328-62b7c059` at 2024-07-08 17:33:28.388776</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">analysis</span>' `20240710-102614-134183cb` at 2024-07-10 10:26:14.078362</span></span>
 <span><span class="co">## &gt; orderly2::orderly_dependency("incoming_data", "latest()",</span></span>
 <span><span class="co">## +                              c("incoming.rds" = "data.rds"))</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Depending on <span style="color: #0000BB;">incoming_data</span> @ `20240708-173327-dd6e3856` (via latest(name == "incoming_data"))</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Depending on <span style="color: #0000BB;">incoming_data</span> @ `20240710-102613-8bebab5b` (via latest(name == "incoming_data"))</span></span>
 <span><span class="co">## &gt; d &lt;- readRDS("incoming.rds")</span></span>
 <span><span class="co">## &gt; png("analysis.png")</span></span>
 <span><span class="co">## &gt; plot(y ~ x, d)</span></span>
@@ -281,7 +281,7 @@ <h2 id="depending-on-packets-from-another-report">Depending on packets from anot
 <span><span class="co">## agg_png </span></span>
 <span><span class="co">##       2</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">analysis.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240708-173328-62b7c059 at 2024-07-08 17:33:28.469544 (0.08076811 secs)</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240710-102614-134183cb at 2024-07-10 10:26:14.161607 (0.08324504 secs)</span></span></code></pre></div>
 <p>For more information on dependencies, see
 <code><a href="../articles/dependencies.html">vignette("dependencies")</a></code>.</p>
 </div>
@@ -355,7 +355,7 @@ <h2 id="available-in-report-orderly-commands">Available in-report orderly comman
 available at that point (see below).</p>
 <div class="sourceCode" id="cb15"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="va">id</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"incoming_data"</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">incoming_data</span>' `20240708-173328-ada1f7ef` at 2024-07-08 17:33:28.683199</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">incoming_data</span>' `20240710-102614-5fcf9915` at 2024-07-10 10:26:14.379319</span></span>
 <span><span class="co">## &gt; orderly2::orderly_strict_mode()</span></span>
 <span><span class="co">## &gt; orderly2::orderly_resource("data.csv")</span></span>
 <span><span class="co">## &gt; orderly2::orderly_artefact("Processed data", "data.rds")</span></span>
@@ -363,7 +363,7 @@ <h2 id="available-in-report-orderly-commands">Available in-report orderly comman
 <span><span class="co">## &gt; d$z &lt;- resid(lm(y ~ x, d))</span></span>
 <span><span class="co">## &gt; saveRDS(d, "data.rds")</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">incoming_data.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240708-173328-ada1f7ef at 2024-07-08 17:33:28.708968 (0.02576971 secs)</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240710-102614-5fcf9915 at 2024-07-10 10:26:14.405794 (0.02647424 secs)</span></span></code></pre></div>
 </div>
 <div class="section level2">
 <h2 id="parameterised-reports">Parameterised reports<a class="anchor" aria-label="anchor" href="#parameterised-reports"></a>
@@ -405,7 +405,7 @@ <h2 id="parameterised-reports">Parameterised reports<a class="anchor" aria-label
 set.</p>
 <div class="sourceCode" id="cb19"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="va">id</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"random"</span>, <span class="fu"><a href="https://rdrr.io/r/base/list.html" class="external-link">list</a></span><span class="op">(</span>n_samples <span class="op">=</span> <span class="fl">15</span><span class="op">)</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">random</span>' `20240708-173328-eb7818c4` at 2024-07-08 17:33:28.922947</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">random</span>' `20240710-102614-9f6109cc` at 2024-07-10 10:26:14.625971</span></span>
 <span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Parameters:</span></span>
 <span><span class="co">## • n_samples: 15</span></span>
 <span><span class="co">## &gt; orderly2::orderly_parameters(n_samples = 10)</span></span>
@@ -413,7 +413,7 @@ <h2 id="parameterised-reports">Parameterised reports<a class="anchor" aria-label
 <span><span class="co">## &gt; d &lt;- data.frame(x = x, y = x + rnorm(n_samples))</span></span>
 <span><span class="co">## &gt; saveRDS(d, "data.rds")</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">random.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240708-173328-eb7818c4 at 2024-07-08 17:33:28.954906 (0.0319581 secs)</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240710-102614-9f6109cc at 2024-07-10 10:26:14.657642 (0.03167081 secs)</span></span></code></pre></div>
 <p>Our resulting file has 15 rows, as the parameter we passed in
 affected the report:</p>
 <div class="sourceCode" id="cb20"><pre class="downlit sourceCode r">
@@ -440,11 +440,11 @@ <h2 id="parameterised-reports">Parameterised reports<a class="anchor" aria-label
 example we can find the most recent version of a packet by running:</p>
 <div class="sourceCode" id="cb21"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_search.html">orderly_search</a></span><span class="op">(</span><span class="st">'latest(name == "random")'</span><span class="op">)</span></span>
-<span><span class="co">## [1] "20240708-173328-eb7818c4"</span></span></code></pre></div>
+<span><span class="co">## [1] "20240710-102614-9f6109cc"</span></span></code></pre></div>
 <p>But we can also pass in parameter queries here:</p>
 <div class="sourceCode" id="cb22"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_search.html">orderly_search</a></span><span class="op">(</span><span class="st">'latest(name == "random" &amp;&amp; parameter:n_samples &gt; 10)'</span><span class="op">)</span></span>
-<span><span class="co">## [1] "20240708-173328-eb7818c4"</span></span></code></pre></div>
+<span><span class="co">## [1] "20240710-102614-9f6109cc"</span></span></code></pre></div>
 <p>These can be used within <code><a href="../reference/orderly_dependency.html">orderly2::orderly_dependency()</a></code>
 (the <code>name == "random"</code> part is implied by the first
 <code>name</code> argument), for example</p>
@@ -476,21 +476,21 @@ <h2 id="shared-resources">Shared resources<a class="anchor" aria-label="anchor"
 <pre><code><span><span class="co">## <span style="color: #0000BB; font-weight: bold;">.</span></span></span>
 <span><span class="co">## ├── <span style="color: #0000BB; font-weight: bold;">archive</span></span></span>
 <span><span class="co">## │   ├── <span style="color: #0000BB; font-weight: bold;">analysis</span></span></span>
-<span><span class="co">## │   │   └── <span style="color: #0000BB; font-weight: bold;">20240708-173328-62b7c059</span></span></span>
+<span><span class="co">## │   │   └── <span style="color: #0000BB; font-weight: bold;">20240710-102614-134183cb</span></span></span>
 <span><span class="co">## │   │       ├── <span style="color: #00BB00;">analysis.R</span></span></span>
 <span><span class="co">## │   │       ├── <span style="color: #BB00BB; font-weight: bold;">analysis.png</span></span></span>
 <span><span class="co">## │   │       └── incoming.rds</span></span>
 <span><span class="co">## │   ├── <span style="color: #0000BB; font-weight: bold;">incoming_data</span></span></span>
-<span><span class="co">## │   │   ├── <span style="color: #0000BB; font-weight: bold;">20240708-173327-dd6e3856</span></span></span>
+<span><span class="co">## │   │   ├── <span style="color: #0000BB; font-weight: bold;">20240710-102613-8bebab5b</span></span></span>
 <span><span class="co">## │   │   │   ├── data.csv</span></span>
 <span><span class="co">## │   │   │   ├── data.rds</span></span>
 <span><span class="co">## │   │   │   └── <span style="color: #00BB00;">incoming_data.R</span></span></span>
-<span><span class="co">## │   │   └── <span style="color: #0000BB; font-weight: bold;">20240708-173328-ada1f7ef</span></span></span>
+<span><span class="co">## │   │   └── <span style="color: #0000BB; font-weight: bold;">20240710-102614-5fcf9915</span></span></span>
 <span><span class="co">## │   │       ├── data.csv</span></span>
 <span><span class="co">## │   │       ├── data.rds</span></span>
 <span><span class="co">## │   │       └── <span style="color: #00BB00;">incoming_data.R</span></span></span>
 <span><span class="co">## │   └── <span style="color: #0000BB; font-weight: bold;">random</span></span></span>
-<span><span class="co">## │       └── <span style="color: #0000BB; font-weight: bold;">20240708-173328-eb7818c4</span></span></span>
+<span><span class="co">## │       └── <span style="color: #0000BB; font-weight: bold;">20240710-102614-9f6109cc</span></span></span>
 <span><span class="co">## │           ├── data.rds</span></span>
 <span><span class="co">## │           └── <span style="color: #00BB00;">random.R</span></span></span>
 <span><span class="co">## ├── <span style="color: #0000BB; font-weight: bold;">draft</span></span></span>
@@ -521,7 +521,7 @@ <h2 id="shared-resources">Shared resources<a class="anchor" aria-label="anchor"
 <p>We can run this:</p>
 <div class="sourceCode" id="cb27"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="va">id</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"use_shared"</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">use_shared</span>' `20240708-173329-6b8c373f` at 2024-07-08 17:33:29.4233</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">use_shared</span>' `20240710-102615-2421c3c7` at 2024-07-10 10:26:15.144337</span></span>
 <span><span class="co">## &gt; orderly2::orderly_shared_resource("data.csv")</span></span>
 <span><span class="co">## &gt; orderly2::orderly_artefact("analysis", "analysis.png")</span></span>
 <span><span class="co">## &gt; d &lt;- read.csv("data.csv")</span></span>
@@ -531,11 +531,11 @@ <h2 id="shared-resources">Shared resources<a class="anchor" aria-label="anchor"
 <span><span class="co">## agg_png </span></span>
 <span><span class="co">##       2</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">use_shared.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240708-173329-6b8c373f at 2024-07-08 17:33:29.460283 (0.03698277 secs)</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240710-102615-2421c3c7 at 2024-07-10 10:26:15.181531 (0.03719354 secs)</span></span></code></pre></div>
 <p>In the resulting archive, the file that was used from the shared
 directory is present:</p>
 <pre><code><span><span class="co">## <span style="color: #0000BB; font-weight: bold;">archive/use_shared</span></span></span>
-<span><span class="co">## └── <span style="color: #0000BB; font-weight: bold;">20240708-173329-6b8c373f</span></span></span>
+<span><span class="co">## └── <span style="color: #0000BB; font-weight: bold;">20240710-102615-2421c3c7</span></span></span>
 <span><span class="co">##     ├── <span style="color: #BB00BB; font-weight: bold;">analysis.png</span></span></span>
 <span><span class="co">##     ├── data.csv</span></span>
 <span><span class="co">##     └── <span style="color: #00BB00;">use_shared.R</span></span></span></code></pre>
@@ -629,49 +629,49 @@ <h2 id="deleting-things-from-the-archive">Deleting things from the archive<a cla
 <span>  name <span class="op">=</span> <span class="st">"incoming_data"</span>,</span>
 <span>  extract <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html" class="external-link">c</a></span><span class="op">(</span>time <span class="op">=</span> <span class="st">"time.start"</span><span class="op">)</span><span class="op">)</span></span>
 <span><span class="co">##                         id                time</span></span>
-<span><span class="co">## 1 20240708-173327-dd6e3856 2024-07-08 17:33:27</span></span>
-<span><span class="co">## 2 20240708-173328-ada1f7ef 2024-07-08 17:33:28</span></span></code></pre></div>
+<span><span class="co">## 1 20240710-102613-8bebab5b 2024-07-10 10:26:13</span></span>
+<span><span class="co">## 2 20240710-102614-5fcf9915 2024-07-10 10:26:14</span></span></code></pre></div>
 <p>When we run the <code>analysis</code> task, it will pull in the most
-recent version (<code>20240708-173328-ada1f7ef</code>). However, if you
+recent version (<code>20240710-102614-5fcf9915</code>). However, if you
 had deleted this manually (e.g., to save space or accidentally) or
 corrupted it (e.g., by opening some output in Excel and letting it save
 changes) it will not be able to be included, and running
 <code>analysis</code> will fail:</p>
 <div class="sourceCode" id="cb34"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"analysis"</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">analysis</span>' `20240708-173330-0b2519a5` at 2024-07-08 17:33:30.046688</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">analysis</span>' `20240710-102615-c4877123` at 2024-07-10 10:26:15.770787</span></span>
 <span><span class="co">## &gt; orderly2::orderly_dependency("incoming_data", "latest()",</span></span>
 <span><span class="co">## +                              c("incoming.rds" = "data.rds"))</span></span>
 <span><span class="co">## <span style="color: #BB0000;">✖</span> Error running <span style="color: #0000BB;">analysis.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240708-173330-0b2519a5 at 2024-07-08 17:33:30.128869 (0.08218169 secs)</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240710-102615-c4877123 at 2024-07-10 10:26:15.853003 (0.08221602 secs)</span></span>
 <span><span class="co">## <span style="color: #BBBB00; font-weight: bold;">Error</span><span style="font-weight: bold;"> in `orderly2::orderly_run()`:</span></span></span>
 <span><span class="co">## <span style="color: #BBBB00;">!</span> Failed to run report</span></span>
 <span><span class="co">## <span style="font-weight: bold;">Caused by error in `orderly_copy_files()`:</span></span></span>
-<span><span class="co">## <span style="color: #BBBB00;">!</span> Unable to copy files, due to deleted packet 20240708-173328-ada1f7ef</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Consider 'orderly2::orderly_validate_archive("20240708-173328-ada1f7ef",</span></span>
+<span><span class="co">## <span style="color: #BBBB00;">!</span> Unable to copy files, due to deleted packet 20240710-102614-5fcf9915</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Consider 'orderly2::orderly_validate_archive("20240710-102614-5fcf9915",</span></span>
 <span><span class="co">##   action = "orphan")' to remove this packet from consideration</span></span>
 <span><span class="co">## <span style="font-weight: bold;">Caused by error:</span></span></span>
 <span><span class="co">## <span style="color: #BBBB00;">!</span> File not found in archive</span></span>
 <span><span class="co">## <span style="color: #BB0000;">✖</span> data.rds</span></span></code></pre></div>
 <p>The error here tries to be fairly informative, telling us that we
 failed because when copying files from
-<code>20240708-173328-ada1f7ef</code> we found that the packet was
+<code>20240710-102614-5fcf9915</code> we found that the packet was
 corrupt, because the file <code>data.rds</code> was not found in the
 archive. It also suggests a fix; we can tell <code>orderly2</code> that
-<code>20240708-173328-ada1f7ef</code> is “orphaned” and should not be
+<code>20240710-102614-5fcf9915</code> is “orphaned” and should not be
 considered for inclusion when we look for dependencies.</p>
 <p>We can carry out the suggestion and just validate this packet by
 running</p>
 <div class="sourceCode" id="cb35"><pre class="downlit sourceCode r">
-<code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_validate_archive.html">orderly_validate_archive</a></span><span class="op">(</span><span class="st">"20240708-173328-ada1f7ef"</span>, action <span class="op">=</span> <span class="st">"orphan"</span><span class="op">)</span></span></code></pre></div>
+<code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_validate_archive.html">orderly_validate_archive</a></span><span class="op">(</span><span class="st">"20240710-102614-5fcf9915"</span>, action <span class="op">=</span> <span class="st">"orphan"</span><span class="op">)</span></span></code></pre></div>
 <p>or we can validate <em>all</em> the packets we have:</p>
 <div class="sourceCode" id="cb36"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_validate_archive.html">orderly_validate_archive</a></span><span class="op">(</span>action <span class="op">=</span> <span class="st">"orphan"</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BB00;">✔</span> 20240708-173327-dd6e3856 (incoming_data) is valid</span></span>
-<span><span class="co">## <span style="color: #00BB00;">✔</span> 20240708-173328-62b7c059 (analysis) is valid</span></span>
-<span><span class="co">## <span style="color: #BB0000;">✖</span> 20240708-173328-ada1f7ef (incoming_data) is invalid due to its files</span></span>
-<span><span class="co">## <span style="color: #00BB00;">✔</span> 20240708-173328-eb7818c4 (random) is valid</span></span>
-<span><span class="co">## <span style="color: #00BB00;">✔</span> 20240708-173329-6b8c373f (use_shared) is valid</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BB00;">✔</span> 20240710-102613-8bebab5b (incoming_data) is valid</span></span>
+<span><span class="co">## <span style="color: #00BB00;">✔</span> 20240710-102614-134183cb (analysis) is valid</span></span>
+<span><span class="co">## <span style="color: #BB0000;">✖</span> 20240710-102614-5fcf9915 (incoming_data) is invalid due to its files</span></span>
+<span><span class="co">## <span style="color: #00BB00;">✔</span> 20240710-102614-9f6109cc (random) is valid</span></span>
+<span><span class="co">## <span style="color: #00BB00;">✔</span> 20240710-102615-2421c3c7 (use_shared) is valid</span></span></code></pre></div>
 <p>If we had the option <code>core.require_complete_tree</code> enabled,
 then this process would also look for any packets that used our
 now-deleted packet and orphan those too, as we no longer have a complete
@@ -758,16 +758,16 @@ <h2 id="interaction-with-the-outpack-store">Interaction with the outpack store<a
 <span><span class="co">## │   └── outpack.rds</span></span>
 <span><span class="co">## ├── <span style="color: #0000BB; font-weight: bold;">location</span></span></span>
 <span><span class="co">## │   ├── <span style="color: #0000BB; font-weight: bold;">local</span></span></span>
-<span><span class="co">## │   │   ├── 20240708-173327-dd6e3856</span></span>
-<span><span class="co">## │   │   ├── 20240708-173328-62b7c059</span></span>
-<span><span class="co">## │   │   ├── 20240708-173328-eb7818c4</span></span>
-<span><span class="co">## │   │   └── 20240708-173329-6b8c373f</span></span>
+<span><span class="co">## │   │   ├── 20240710-102613-8bebab5b</span></span>
+<span><span class="co">## │   │   ├── 20240710-102614-134183cb</span></span>
+<span><span class="co">## │   │   ├── 20240710-102614-9f6109cc</span></span>
+<span><span class="co">## │   │   └── 20240710-102615-2421c3c7</span></span>
 <span><span class="co">## │   └── <span style="color: #0000BB; font-weight: bold;">orphan</span></span></span>
 <span><span class="co">## └── <span style="color: #0000BB; font-weight: bold;">metadata</span></span></span>
-<span><span class="co">##     ├── 20240708-173327-dd6e3856</span></span>
-<span><span class="co">##     ├── 20240708-173328-62b7c059</span></span>
-<span><span class="co">##     ├── 20240708-173328-eb7818c4</span></span>
-<span><span class="co">##     └── 20240708-173329-6b8c373f</span></span></code></pre>
+<span><span class="co">##     ├── 20240710-102613-8bebab5b</span></span>
+<span><span class="co">##     ├── 20240710-102614-134183cb</span></span>
+<span><span class="co">##     ├── 20240710-102614-9f6109cc</span></span>
+<span><span class="co">##     └── 20240710-102615-2421c3c7</span></span></code></pre>
 <p>As can be perhaps inferred from the filenames, the files
 <code>.outpack/metadata/&lt;packet-id&gt;</code> are the metadata for
 each packet as it has been run. The files
@@ -807,16 +807,16 @@ <h2 id="interaction-with-the-outpack-store">Interaction with the outpack store<a
 <span><span class="co">## │   └── outpack.rds</span></span>
 <span><span class="co">## ├── <span style="color: #0000BB; font-weight: bold;">location</span></span></span>
 <span><span class="co">## │   ├── <span style="color: #0000BB; font-weight: bold;">local</span></span></span>
-<span><span class="co">## │   │   ├── 20240708-173327-dd6e3856</span></span>
-<span><span class="co">## │   │   ├── 20240708-173328-62b7c059</span></span>
-<span><span class="co">## │   │   ├── 20240708-173328-eb7818c4</span></span>
-<span><span class="co">## │   │   └── 20240708-173329-6b8c373f</span></span>
+<span><span class="co">## │   │   ├── 20240710-102613-8bebab5b</span></span>
+<span><span class="co">## │   │   ├── 20240710-102614-134183cb</span></span>
+<span><span class="co">## │   │   ├── 20240710-102614-9f6109cc</span></span>
+<span><span class="co">## │   │   └── 20240710-102615-2421c3c7</span></span>
 <span><span class="co">## │   └── <span style="color: #0000BB; font-weight: bold;">orphan</span></span></span>
 <span><span class="co">## └── <span style="color: #0000BB; font-weight: bold;">metadata</span></span></span>
-<span><span class="co">##     ├── 20240708-173327-dd6e3856</span></span>
-<span><span class="co">##     ├── 20240708-173328-62b7c059</span></span>
-<span><span class="co">##     ├── 20240708-173328-eb7818c4</span></span>
-<span><span class="co">##     └── 20240708-173329-6b8c373f</span></span></code></pre>
+<span><span class="co">##     ├── 20240710-102613-8bebab5b</span></span>
+<span><span class="co">##     ├── 20240710-102614-134183cb</span></span>
+<span><span class="co">##     ├── 20240710-102614-9f6109cc</span></span>
+<span><span class="co">##     └── 20240710-102615-2421c3c7</span></span></code></pre>
 <p>The files under <code>.outpack/files/</code> should never be modified
 or deleted. This approach to storage naturally deduplicates the file
 archive, so that a large file used in many places is only ever stored
diff --git a/articles/metadata.html b/articles/metadata.html
index eab51276..025270e4 100644
--- a/articles/metadata.html
+++ b/articles/metadata.html
@@ -20,7 +20,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/articles/migrating.html b/articles/migrating.html
index c604fde1..69d47b41 100644
--- a/articles/migrating.html
+++ b/articles/migrating.html
@@ -20,7 +20,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/articles/orderly2.html b/articles/orderly2.html
index 05550c69..78c353a1 100644
--- a/articles/orderly2.html
+++ b/articles/orderly2.html
@@ -20,7 +20,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/articles/plugins.html b/articles/plugins.html
index 67343512..2b35ae1f 100644
--- a/articles/plugins.html
+++ b/articles/plugins.html
@@ -20,7 +20,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -137,7 +137,7 @@ <h3 id="an-example">An example<a class="anchor" aria-label="anchor" href="#an-ex
 <div class="sourceCode" id="cb2"><pre class="sourceCode yaml"><code class="sourceCode yaml"><span id="cb2-1"><a href="#cb2-1" tabindex="-1"></a><span class="fu">minimum_orderly_version</span><span class="kw">:</span><span class="at"> </span><span class="fl">1.99.0</span></span>
 <span id="cb2-2"><a href="#cb2-2" tabindex="-1"></a><span class="fu">plugins</span><span class="kw">:</span></span>
 <span id="cb2-3"><a href="#cb2-3" tabindex="-1"></a><span class="at">  </span><span class="fu">example.db</span><span class="kw">:</span></span>
-<span id="cb2-4"><a href="#cb2-4" tabindex="-1"></a><span class="at">    </span><span class="fu">path</span><span class="kw">:</span><span class="at"> /tmp/RtmpFoww4i/file1c2637c713b9</span></span></code></pre></div>
+<span id="cb2-4"><a href="#cb2-4" tabindex="-1"></a><span class="at">    </span><span class="fu">path</span><span class="kw">:</span><span class="at"> /tmp/RtmplIsFe8/file1bfa3d5e9c71</span></span></code></pre></div>
 <p>Our plugin is called <code>example.db</code> and is listed within the
 <code>plugins</code> section, along with its configuration; in this case
 indicating the path where the SQLite file can be loaded from.</p>
@@ -289,13 +289,13 @@ <h4 id="trying-it-out">Trying it out<a class="anchor" aria-label="anchor" href="
 <p>Now, we can run the report:</p>
 <div class="sourceCode" id="cb13"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"example"</span>, root <span class="op">=</span> <span class="va">path_root</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">example</span>' `20240708-173338-4f7ea6f4` at 2024-07-08 17:33:38.315095</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">example</span>' `20240710-102624-0adf8529` at 2024-07-10 10:26:24.046976</span></span>
 <span><span class="co">## &gt; dat &lt;- example.db::query("SELECT * FROM mtcars WHERE cyl == 4")</span></span>
 <span><span class="co">## &gt; orderly2::orderly_artefact("Summary of data", "data.rds")</span></span>
 <span><span class="co">## &gt; saveRDS(summary(dat), "data.rds")</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">example.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240708-173338-4f7ea6f4 at 2024-07-08 17:33:38.496032 (0.1809373 secs)</span></span>
-<span><span class="co">## [1] "20240708-173338-4f7ea6f4"</span></span></code></pre></div>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240710-102624-0adf8529 at 2024-07-10 10:26:24.226441 (0.1794653 secs)</span></span>
+<span><span class="co">## [1] "20240710-102624-0adf8529"</span></span></code></pre></div>
 </div>
 </div>
 <div class="section level2">
@@ -435,7 +435,7 @@ <h2 id="saving-metadata-about-what-the-plugin-did">Saving metadata about what th
 <pre><code><span><span class="co">## <span style="color: #0000BB; font-weight: bold;">.</span></span></span>
 <span><span class="co">## ├── <span style="color: #0000BB; font-weight: bold;">archive</span></span></span>
 <span><span class="co">## │   └── <span style="color: #0000BB; font-weight: bold;">example</span></span></span>
-<span><span class="co">## │       └── <span style="color: #0000BB; font-weight: bold;">20240708-173338-4f7ea6f4</span></span></span>
+<span><span class="co">## │       └── <span style="color: #0000BB; font-weight: bold;">20240710-102624-0adf8529</span></span></span>
 <span><span class="co">## │           ├── data.rds</span></span>
 <span><span class="co">## │           └── <span style="color: #00BB00;">example.R</span></span></span>
 <span><span class="co">## ├── <span style="color: #0000BB; font-weight: bold;">draft</span></span></span>
@@ -487,12 +487,12 @@ <h2 id="saving-metadata-about-what-the-plugin-did">Saving metadata about what th
 several).</p>
 <div class="sourceCode" id="cb21"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="va">id</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"example"</span>, root <span class="op">=</span> <span class="va">path_root</span><span class="op">)</span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">example</span>' `20240708-173339-60aa2af8` at 2024-07-08 17:33:39.380817</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">example</span>' `20240710-102625-1d2e2744` at 2024-07-10 10:26:25.117168</span></span>
 <span><span class="co">## &gt; dat &lt;- example.db::query("SELECT * FROM mtcars WHERE cyl == 4")</span></span>
 <span><span class="co">## &gt; orderly2::orderly_artefact("Summary of data", "data.rds")</span></span>
 <span><span class="co">## &gt; saveRDS(summary(dat), "data.rds")</span></span>
 <span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">example.R</span></span></span>
-<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240708-173339-60aa2af8 at 2024-07-08 17:33:39.422584 (0.04176736 secs)</span></span>
+<span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240710-102625-1d2e2744 at 2024-07-10 10:26:25.160065 (0.04289699 secs)</span></span>
 <span><span class="va">meta</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_metadata.html">orderly_metadata</a></span><span class="op">(</span><span class="va">id</span>, root <span class="op">=</span> <span class="va">path_root</span><span class="op">)</span></span>
 <span><span class="va">meta</span><span class="op">$</span><span class="va">custom</span><span class="op">$</span><span class="va">example.db</span></span>
 <span><span class="co">##                                   sql rows         cols</span></span>
diff --git a/articles/query.html b/articles/query.html
index ccc87a9a..2650afd7 100644
--- a/articles/query.html
+++ b/articles/query.html
@@ -20,7 +20,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/articles/troubleshooting.html b/articles/troubleshooting.html
index eefc4a5c..44a435ef 100644
--- a/articles/troubleshooting.html
+++ b/articles/troubleshooting.html
@@ -20,7 +20,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -116,21 +116,21 @@ <h2 id="outpack-files-accidentally-committed-to-git">Outpack files accidentally
 <span><span class="co">##   <span style="color: #0000BB; font-style: italic;">&lt;https://mrc-ide.github.io/orderly2/articles/troubleshooting.html&gt;</span></span></span>
 <span><span class="co">## <span style="color: #555555;">This warning is displayed once per session.</span></span></span></code></pre>
 <pre><code><span><span class="co">## <span style="color: #00BB00;">✔</span> Wrote '.gitignore'</span></span></code></pre>
-<pre><code><span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240708-173345-ec5aeb3c` at 2024-07-08 17:33:45.926435</span></span></code></pre>
+<pre><code><span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240710-102631-d2ef2801` at 2024-07-10 10:26:31.827093</span></span></code></pre>
 <pre><code><span><span class="co">## &gt; orderly2::orderly_artefact("Final data", "data.rds")</span></span>
 <span><span class="co">## &gt; saveRDS(mtcars, "data.rds")</span></span></code></pre>
 <pre><code><span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">data.R</span></span></span></code></pre>
-<pre><code><span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240708-173345-ec5aeb3c at 2024-07-08 17:33:45.964067 (0.03763103 secs)</span></span></code></pre>
-<pre><code><span><span class="co">## [1] "20240708-173345-ec5aeb3c"</span></span></code></pre>
+<pre><code><span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240710-102631-d2ef2801 at 2024-07-10 10:26:31.863847 (0.03675437 secs)</span></span></code></pre>
+<pre><code><span><span class="co">## [1] "20240710-102631-d2ef2801"</span></span></code></pre>
 <p>subsequent calls will not display the warning:</p>
 <div class="sourceCode" id="cb12"><pre class="downlit sourceCode r">
 <code class="sourceCode R"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="../reference/orderly_run.html">orderly_run</a></span><span class="op">(</span><span class="st">"data"</span><span class="op">)</span></span></code></pre></div>
-<pre><code><span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240708-173346-0ce7688a` at 2024-07-08 17:33:46.053698</span></span></code></pre>
+<pre><code><span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Starting packet '<span style="color: #0000BB;">data</span>' `20240710-102631-f313e685` at 2024-07-10 10:26:31.95268</span></span></code></pre>
 <pre><code><span><span class="co">## &gt; orderly2::orderly_artefact("Final data", "data.rds")</span></span>
 <span><span class="co">## &gt; saveRDS(mtcars, "data.rds")</span></span></code></pre>
 <pre><code><span><span class="co">## <span style="color: #00BB00;">✔</span> Finished running <span style="color: #0000BB;">data.R</span></span></span></code></pre>
-<pre><code><span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240708-173346-0ce7688a at 2024-07-08 17:33:46.076102 (0.02240443 secs)</span></span></code></pre>
-<pre><code><span><span class="co">## [1] "20240708-173346-0ce7688a"</span></span></code></pre>
+<pre><code><span><span class="co">## <span style="color: #00BBBB;">ℹ</span> Finished 20240710-102631-f313e685 at 2024-07-10 10:26:31.97453 (0.02185059 secs)</span></span></code></pre>
+<pre><code><span><span class="co">## [1] "20240710-102631-f313e685"</span></span></code></pre>
 <p>The rest of this section discusses how you might permanently fix the
 issue.</p>
 <div class="section level3">
diff --git a/authors.html b/authors.html
index e00915d1..0427c2b2 100644
--- a/authors.html
+++ b/authors.html
@@ -7,7 +7,7 @@
 
     <a class="navbar-brand me-2" href="index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -73,13 +73,13 @@ <h2 id="citation">Citation</h2>
 
       <p>FitzJohn R, Ashton R, Hill A (2024).
 <em>orderly2: Orderly Next Generation</em>.
-R package version 1.99.20, <a href="https://github.com/mrc-ide/orderly2" class="external-link">https://github.com/mrc-ide/orderly2</a>.
+R package version 1.99.21, <a href="https://github.com/mrc-ide/orderly2" class="external-link">https://github.com/mrc-ide/orderly2</a>.
 </p>
       <pre>@Manual{,
   title = {orderly2: Orderly Next Generation},
   author = {Rich FitzJohn and Robert Ashton and Alex Hill},
   year = {2024},
-  note = {R package version 1.99.20},
+  note = {R package version 1.99.21},
   url = {https://github.com/mrc-ide/orderly2},
 }</pre>
     </div>
diff --git a/index.html b/index.html
index 1a5b6067..c38e3125 100644
--- a/index.html
+++ b/index.html
@@ -22,7 +22,7 @@
 
     <a class="navbar-brand me-2" href="index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/pkgdown.yml b/pkgdown.yml
index b07f6a39..a7842ec4 100644
--- a/pkgdown.yml
+++ b/pkgdown.yml
@@ -11,7 +11,7 @@ articles:
   plugins: plugins.html
   query: query.html
   troubleshooting: troubleshooting.html
-last_built: 2024-07-08T17:33Z
+last_built: 2024-07-10T10:25Z
 urls:
   reference: https://mrc-ide.github.io/orderly2/reference
   article: https://mrc-ide.github.io/orderly2/articles
diff --git a/reference/index.html b/reference/index.html
index f1fcf667..8cfb2ba5 100644
--- a/reference/index.html
+++ b/reference/index.html
@@ -7,7 +7,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_artefact.html b/reference/orderly_artefact.html
index fe0c8166..92ac0fa7 100644
--- a/reference/orderly_artefact.html
+++ b/reference/orderly_artefact.html
@@ -11,7 +11,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_cleanup.html b/reference/orderly_cleanup.html
index e6531013..1a9386ff 100644
--- a/reference/orderly_cleanup.html
+++ b/reference/orderly_cleanup.html
@@ -13,7 +13,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -135,24 +135,24 @@ <h2 id="notes-for-user-of-orderly-">Notes for user of orderly1<a class="anchor"
     <h2 id="ref-examples">Examples<a class="anchor" aria-label="anchor" href="#ref-examples"></a></h2>
     <div class="sourceCode"><pre class="sourceCode r"><code><span class="r-in"><span><span class="co"># Create a simple example:</span></span></span>
 <span class="r-in"><span><span class="va">path</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="orderly_example.html">orderly_example</a></span><span class="op">(</span><span class="st">"default"</span><span class="op">)</span></span></span>
-<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpfBuUfo/file174c1393fe1b'</span>
+<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpO3RBoN/file17206a1e0276'</span>
 <span class="r-in"><span></span></span>
 <span class="r-in"><span><span class="co"># We simulate running a packet interactively by using 'source';</span></span></span>
 <span class="r-in"><span><span class="co"># you might have run this line-by-line, or with the "Source"</span></span></span>
 <span class="r-in"><span><span class="co"># button in Rstudio.</span></span></span>
 <span class="r-in"><span><span class="kw"><a href="https://rdrr.io/r/base/source.html" class="external-link">source</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/r/base/file.path.html" class="external-link">file.path</a></span><span class="op">(</span><span class="va">path</span>, <span class="st">"src/data/data.R"</span><span class="op">)</span>, chdir <span class="op">=</span> <span class="cn">TRUE</span><span class="op">)</span></span></span>
-<span class="r-wrn co"><span class="r-pr">#&gt;</span> <span class="warning">Warning: </span>cannot open file '/tmp/RtmpfBuUfo/file174c1393fe1b/src/data/data.R': No such file or directory</span>
+<span class="r-wrn co"><span class="r-pr">#&gt;</span> <span class="warning">Warning: </span>cannot open file '/tmp/RtmpO3RBoN/file17206a1e0276/src/data/data.R': No such file or directory</span>
 <span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error in file(filename, "r", encoding = encoding):</span> cannot open the connection</span>
 <span class="r-in"><span></span></span>
 <span class="r-in"><span><span class="co"># Having run this, the output of the report is present in the</span></span></span>
 <span class="r-in"><span><span class="co"># source directory:</span></span></span>
 <span class="r-in"><span><span class="fu">fs</span><span class="fu">::</span><span class="fu"><a href="https://fs.r-lib.org/reference/dir_tree.html" class="external-link">dir_tree</a></span><span class="op">(</span><span class="va">path</span><span class="op">)</span></span></span>
-<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> [ENOENT] Failed to search directory '/tmp/RtmpfBuUfo/file174c1393fe1b': no such file or directory</span>
+<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> [ENOENT] Failed to search directory '/tmp/RtmpO3RBoN/file17206a1e0276': no such file or directory</span>
 <span class="r-in"><span></span></span>
 <span class="r-in"><span><span class="co"># We can detect what might want cleaning up by running</span></span></span>
 <span class="r-in"><span><span class="co"># "orderly_cleanup_status":</span></span></span>
 <span class="r-in"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu">orderly_cleanup_status</span><span class="op">(</span><span class="st">"data"</span>, root <span class="op">=</span> <span class="va">path</span><span class="op">)</span></span></span>
-<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpfBuUfo/file174c1393fe1b'</span>
+<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpO3RBoN/file17206a1e0276'</span>
 <span class="r-in"><span></span></span>
 <span class="r-in"><span><span class="co"># Soon this will print more nicely to the screen, but for now you</span></span></span>
 <span class="r-in"><span><span class="co"># can see that the status of "data.rds" is "derived", which means</span></span></span>
@@ -161,7 +161,7 @@ <h2 id="ref-examples">Examples<a class="anchor" aria-label="anchor" href="#ref-e
 <span class="r-in"><span></span></span>
 <span class="r-in"><span><span class="co"># Do the actual deletion:</span></span></span>
 <span class="r-in"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu">orderly_cleanup</span><span class="op">(</span><span class="st">"data"</span>, root <span class="op">=</span> <span class="va">path</span><span class="op">)</span></span></span>
-<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpfBuUfo/file174c1393fe1b'</span>
+<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpO3RBoN/file17206a1e0276'</span>
 </code></pre></div>
     </div>
   </main><aside class="col-md-3"><nav id="toc" aria-label="Table of contents"><h2>On this page</h2>
diff --git a/reference/orderly_config.html b/reference/orderly_config.html
index ca5fe77d..92c440c2 100644
--- a/reference/orderly_config.html
+++ b/reference/orderly_config.html
@@ -9,7 +9,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -110,7 +110,7 @@ <h2 id="ref-examples">Examples<a class="anchor" aria-label="anchor" href="#ref-e
 <span class="r-in"><span><span class="co"># A default configuration in a new temporary directory</span></span></span>
 <span class="r-in"><span><span class="va">path</span> <span class="op">&lt;-</span> <span class="fu">withr</span><span class="fu">::</span><span class="fu"><a href="https://withr.r-lib.org/reference/with_tempfile.html" class="external-link">local_tempdir</a></span><span class="op">(</span><span class="op">)</span></span></span>
 <span class="r-in"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="orderly_init.html">orderly_init</a></span><span class="op">(</span><span class="va">path</span><span class="op">)</span></span></span>
-<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpfBuUfo/file174c66475d05'</span>
+<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpO3RBoN/file172041aa773c'</span>
 <span class="r-in"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu">orderly_config</span><span class="op">(</span><span class="va">path</span><span class="op">)</span></span></span>
 <span class="r-out co"><span class="r-pr">#&gt;</span> $core</span>
 <span class="r-out co"><span class="r-pr">#&gt;</span> $core$path_archive</span>
diff --git a/reference/orderly_config_set.html b/reference/orderly_config_set.html
index 6fed375d..9d799ec4 100644
--- a/reference/orderly_config_set.html
+++ b/reference/orderly_config_set.html
@@ -9,7 +9,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -115,9 +115,9 @@ <h2 id="ref-examples">Examples<a class="anchor" aria-label="anchor" href="#ref-e
 <span class="r-in"><span><span class="co"># saves output within the "archive" directory:</span></span></span>
 <span class="r-in"><span><span class="va">path</span> <span class="op">&lt;-</span> <span class="fu">withr</span><span class="fu">::</span><span class="fu"><a href="https://withr.r-lib.org/reference/with_tempfile.html" class="external-link">local_tempdir</a></span><span class="op">(</span><span class="op">)</span></span></span>
 <span class="r-in"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="orderly_init.html">orderly_init</a></span><span class="op">(</span><span class="va">path</span><span class="op">)</span></span></span>
-<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpfBuUfo/file174c69243531'</span>
+<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpO3RBoN/file17204247565d'</span>
 <span class="r-in"><span><span class="fu">fs</span><span class="fu">::</span><span class="fu"><a href="https://fs.r-lib.org/reference/dir_tree.html" class="external-link">dir_tree</a></span><span class="op">(</span><span class="va">path</span>, all <span class="op">=</span> <span class="cn">TRUE</span><span class="op">)</span></span></span>
-<span class="r-out co"><span class="r-pr">#&gt;</span> <span style="color: #0000BB; font-weight: bold;">/tmp/RtmpfBuUfo/file174c69243531</span></span>
+<span class="r-out co"><span class="r-pr">#&gt;</span> <span style="color: #0000BB; font-weight: bold;">/tmp/RtmpO3RBoN/file17204247565d</span></span>
 <span class="r-out co"><span class="r-pr">#&gt;</span> ├── <span style="color: #0000BB; font-weight: bold;">.outpack</span></span>
 <span class="r-out co"><span class="r-pr">#&gt;</span> │   ├── config.json</span>
 <span class="r-out co"><span class="r-pr">#&gt;</span> │   ├── <span style="color: #0000BB; font-weight: bold;">location</span></span>
@@ -129,7 +129,7 @@ <h2 id="ref-examples">Examples<a class="anchor" aria-label="anchor" href="#ref-e
 <span class="r-in"><span>                             core.path_archive <span class="op">=</span> <span class="cn">NULL</span>,</span></span>
 <span class="r-in"><span>                             root <span class="op">=</span> <span class="va">path</span><span class="op">)</span></span></span>
 <span class="r-in"><span><span class="fu">fs</span><span class="fu">::</span><span class="fu"><a href="https://fs.r-lib.org/reference/dir_tree.html" class="external-link">dir_tree</a></span><span class="op">(</span><span class="va">path</span>, all <span class="op">=</span> <span class="cn">TRUE</span><span class="op">)</span></span></span>
-<span class="r-out co"><span class="r-pr">#&gt;</span> <span style="color: #0000BB; font-weight: bold;">/tmp/RtmpfBuUfo/file174c69243531</span></span>
+<span class="r-out co"><span class="r-pr">#&gt;</span> <span style="color: #0000BB; font-weight: bold;">/tmp/RtmpO3RBoN/file17204247565d</span></span>
 <span class="r-out co"><span class="r-pr">#&gt;</span> ├── <span style="color: #0000BB; font-weight: bold;">.outpack</span></span>
 <span class="r-out co"><span class="r-pr">#&gt;</span> │   ├── config.json</span>
 <span class="r-out co"><span class="r-pr">#&gt;</span> │   ├── <span style="color: #0000BB; font-weight: bold;">files</span></span>
diff --git a/reference/orderly_copy_files.html b/reference/orderly_copy_files.html
index 3b6a484d..88595f4a 100644
--- a/reference/orderly_copy_files.html
+++ b/reference/orderly_copy_files.html
@@ -19,7 +19,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_dependency.html b/reference/orderly_dependency.html
index 489bf2d3..9127813a 100644
--- a/reference/orderly_dependency.html
+++ b/reference/orderly_dependency.html
@@ -7,7 +7,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_description.html b/reference/orderly_description.html
index 37a998c5..b243aa70 100644
--- a/reference/orderly_description.html
+++ b/reference/orderly_description.html
@@ -7,7 +7,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_example.html b/reference/orderly_example.html
index c53eda26..d903184a 100644
--- a/reference/orderly_example.html
+++ b/reference/orderly_example.html
@@ -9,7 +9,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -89,9 +89,9 @@ <h2 id="value">Value<a class="anchor" aria-label="anchor" href="#value"></a></h2
     <div class="section level2">
     <h2 id="ref-examples">Examples<a class="anchor" aria-label="anchor" href="#ref-examples"></a></h2>
     <div class="sourceCode"><pre class="sourceCode r"><code><span class="r-in"><span><span class="va">path</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu">orderly_example</span><span class="op">(</span><span class="st">"default"</span><span class="op">)</span></span></span>
-<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpfBuUfo/file174c5e8a3cc6'</span>
+<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpO3RBoN/file172057385d9'</span>
 <span class="r-in"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="orderly_list_src.html">orderly_list_src</a></span><span class="op">(</span>root <span class="op">=</span> <span class="va">path</span><span class="op">)</span></span></span>
-<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpfBuUfo/file174c5e8a3cc6'</span>
+<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpO3RBoN/file172057385d9'</span>
 </code></pre></div>
     </div>
   </main><aside class="col-md-3"><nav id="toc" aria-label="Table of contents"><h2>On this page</h2>
diff --git a/reference/orderly_gitignore_update.html b/reference/orderly_gitignore_update.html
index dd7ad71b..7fc7fadd 100644
--- a/reference/orderly_gitignore_update.html
+++ b/reference/orderly_gitignore_update.html
@@ -15,7 +15,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_hash.html b/reference/orderly_hash.html
index d6b4b86e..b5740c71 100644
--- a/reference/orderly_hash.html
+++ b/reference/orderly_hash.html
@@ -19,7 +19,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_init.html b/reference/orderly_init.html
index 278db477..03da0f10 100644
--- a/reference/orderly_init.html
+++ b/reference/orderly_init.html
@@ -13,7 +13,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -147,7 +147,7 @@ <h2 id="ref-examples">Examples<a class="anchor" aria-label="anchor" href="#ref-e
 <span class="r-in"><span></span></span>
 <span class="r-in"><span><span class="co"># Initialise a new repository, setting an option:</span></span></span>
 <span class="r-in"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu">orderly_init</span><span class="op">(</span><span class="va">path</span>, use_file_store <span class="op">=</span> <span class="cn">TRUE</span><span class="op">)</span></span></span>
-<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpfBuUfo/file174c46e6b11f'</span>
+<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpO3RBoN/file17204d71c6f8'</span>
 <span class="r-in"><span></span></span>
 <span class="r-in"><span><span class="co"># fs::dir_tree(path, all = TRUE)</span></span></span>
 </code></pre></div>
diff --git a/reference/orderly_interactive_set_search_options.html b/reference/orderly_interactive_set_search_options.html
index efb5a554..bb49989a 100644
--- a/reference/orderly_interactive_set_search_options.html
+++ b/reference/orderly_interactive_set_search_options.html
@@ -21,7 +21,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_list_src.html b/reference/orderly_list_src.html
index fa8c2b32..3ef30dd2 100644
--- a/reference/orderly_list_src.html
+++ b/reference/orderly_list_src.html
@@ -13,7 +13,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -99,9 +99,9 @@ <h2 id="see-also">See also<a class="anchor" aria-label="anchor" href="#see-also"
     <div class="section level2">
     <h2 id="ref-examples">Examples<a class="anchor" aria-label="anchor" href="#ref-examples"></a></h2>
     <div class="sourceCode"><pre class="sourceCode r"><code><span class="r-in"><span><span class="va">path</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="orderly_example.html">orderly_example</a></span><span class="op">(</span><span class="st">"default"</span><span class="op">)</span></span></span>
-<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpfBuUfo/file174c42072cbd'</span>
+<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpO3RBoN/file172037dab125'</span>
 <span class="r-in"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu">orderly_list_src</span><span class="op">(</span>root <span class="op">=</span> <span class="va">path</span><span class="op">)</span></span></span>
-<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpfBuUfo/file174c42072cbd'</span>
+<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpO3RBoN/file172037dab125'</span>
 </code></pre></div>
     </div>
   </main><aside class="col-md-3"><nav id="toc" aria-label="Table of contents"><h2>On this page</h2>
diff --git a/reference/orderly_location_add.html b/reference/orderly_location_add.html
index 9c5275f0..d9e9e8f5 100644
--- a/reference/orderly_location_add.html
+++ b/reference/orderly_location_add.html
@@ -17,7 +17,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_location_list.html b/reference/orderly_location_list.html
index fff83b90..ab1e4303 100644
--- a/reference/orderly_location_list.html
+++ b/reference/orderly_location_list.html
@@ -7,7 +7,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_location_pull_metadata.html b/reference/orderly_location_pull_metadata.html
index 3f7af7a5..2f208cfe 100644
--- a/reference/orderly_location_pull_metadata.html
+++ b/reference/orderly_location_pull_metadata.html
@@ -11,7 +11,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_location_pull_packet.html b/reference/orderly_location_pull_packet.html
index 3c6fc989..07ce4108 100644
--- a/reference/orderly_location_pull_packet.html
+++ b/reference/orderly_location_pull_packet.html
@@ -13,7 +13,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_location_push.html b/reference/orderly_location_push.html
index 7d1397ee..85e0e631 100644
--- a/reference/orderly_location_push.html
+++ b/reference/orderly_location_push.html
@@ -17,7 +17,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_location_remove.html b/reference/orderly_location_remove.html
index ff04be95..29798896 100644
--- a/reference/orderly_location_remove.html
+++ b/reference/orderly_location_remove.html
@@ -11,7 +11,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_location_rename.html b/reference/orderly_location_rename.html
index c394d0c0..906c0234 100644
--- a/reference/orderly_location_rename.html
+++ b/reference/orderly_location_rename.html
@@ -7,7 +7,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_metadata.html b/reference/orderly_metadata.html
index df7b797c..469b3060 100644
--- a/reference/orderly_metadata.html
+++ b/reference/orderly_metadata.html
@@ -11,7 +11,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_metadata_extract.html b/reference/orderly_metadata_extract.html
index d3b601a8..ee14c908 100644
--- a/reference/orderly_metadata_extract.html
+++ b/reference/orderly_metadata_extract.html
@@ -15,7 +15,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_metadata_read.html b/reference/orderly_metadata_read.html
index 3829d677..37127e39 100644
--- a/reference/orderly_metadata_read.html
+++ b/reference/orderly_metadata_read.html
@@ -15,7 +15,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_new.html b/reference/orderly_new.html
index 07b5139a..f682ba64 100644
--- a/reference/orderly_new.html
+++ b/reference/orderly_new.html
@@ -7,7 +7,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_parameters.html b/reference/orderly_parameters.html
index f7f5d688..acdd689c 100644
--- a/reference/orderly_parameters.html
+++ b/reference/orderly_parameters.html
@@ -19,7 +19,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_parse_file.html b/reference/orderly_parse_file.html
index 2e1f62fb..3e66cb60 100644
--- a/reference/orderly_parse_file.html
+++ b/reference/orderly_parse_file.html
@@ -7,7 +7,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_plugin_add_metadata.html b/reference/orderly_plugin_add_metadata.html
index bfd66866..f410f5ec 100644
--- a/reference/orderly_plugin_add_metadata.html
+++ b/reference/orderly_plugin_add_metadata.html
@@ -11,7 +11,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_plugin_context.html b/reference/orderly_plugin_context.html
index 419feef0..f6fe09a2 100644
--- a/reference/orderly_plugin_context.html
+++ b/reference/orderly_plugin_context.html
@@ -15,7 +15,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_plugin_register.html b/reference/orderly_plugin_register.html
index 0ac5040e..8ceff7ee 100644
--- a/reference/orderly_plugin_register.html
+++ b/reference/orderly_plugin_register.html
@@ -19,7 +19,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_prune_orphans.html b/reference/orderly_prune_orphans.html
index 888917b6..cc531601 100644
--- a/reference/orderly_prune_orphans.html
+++ b/reference/orderly_prune_orphans.html
@@ -17,7 +17,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_query.html b/reference/orderly_query.html
index 0080b300..cbeea12b 100644
--- a/reference/orderly_query.html
+++ b/reference/orderly_query.html
@@ -9,7 +9,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_query_explain.html b/reference/orderly_query_explain.html
index e78dda38..e27657b4 100644
--- a/reference/orderly_query_explain.html
+++ b/reference/orderly_query_explain.html
@@ -13,7 +13,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_resource.html b/reference/orderly_resource.html
index aefa8b6e..08e08f8d 100644
--- a/reference/orderly_resource.html
+++ b/reference/orderly_resource.html
@@ -13,7 +13,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_run.html b/reference/orderly_run.html
index 5c8074a9..de0c8977 100644
--- a/reference/orderly_run.html
+++ b/reference/orderly_run.html
@@ -11,7 +11,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
@@ -236,19 +236,19 @@ <h2 id="manually-setting-report-source-directory">Manually setting report source
     <h2 id="ref-examples">Examples<a class="anchor" aria-label="anchor" href="#ref-examples"></a></h2>
     <div class="sourceCode"><pre class="sourceCode r"><code><span class="r-in"><span><span class="co"># Create a simple example:</span></span></span>
 <span class="r-in"><span><span class="va">path</span> <span class="op">&lt;-</span> <span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="orderly_example.html">orderly_example</a></span><span class="op">(</span><span class="st">"default"</span><span class="op">)</span></span></span>
-<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpfBuUfo/file174c1399bc00'</span>
+<span class="r-msg co"><span class="r-pr">#&gt;</span> <span style="color: #00BB00;">✔</span> Created orderly root at '/tmp/RtmpO3RBoN/file1720100cfa3f'</span>
 <span class="r-in"><span></span></span>
 <span class="r-in"><span><span class="co"># Run the 'data' task:</span></span></span>
 <span class="r-in"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu">orderly_run</span><span class="op">(</span><span class="st">"data"</span>, root <span class="op">=</span> <span class="va">path</span><span class="op">)</span></span></span>
-<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpfBuUfo/file174c1399bc00'</span>
+<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpO3RBoN/file1720100cfa3f'</span>
 <span class="r-in"><span></span></span>
 <span class="r-in"><span><span class="co"># After running, a finished packet appears in the archive:</span></span></span>
 <span class="r-in"><span><span class="fu">fs</span><span class="fu">::</span><span class="fu"><a href="https://fs.r-lib.org/reference/dir_tree.html" class="external-link">dir_tree</a></span><span class="op">(</span><span class="va">path</span><span class="op">)</span></span></span>
-<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> [ENOENT] Failed to search directory '/tmp/RtmpfBuUfo/file174c1399bc00': no such file or directory</span>
+<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> [ENOENT] Failed to search directory '/tmp/RtmpO3RBoN/file1720100cfa3f': no such file or directory</span>
 <span class="r-in"><span></span></span>
 <span class="r-in"><span><span class="co"># and we can query the metadata:</span></span></span>
 <span class="r-in"><span><span class="fu">orderly2</span><span class="fu">::</span><span class="fu"><a href="orderly_metadata_extract.html">orderly_metadata_extract</a></span><span class="op">(</span>name <span class="op">=</span> <span class="st">"data"</span>, root <span class="op">=</span> <span class="va">path</span><span class="op">)</span></span></span>
-<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpfBuUfo/file174c1399bc00'</span>
+<span class="r-err co"><span class="r-pr">#&gt;</span> <span class="error">Error:</span> Directory does not exist: '/tmp/RtmpO3RBoN/file1720100cfa3f'</span>
 </code></pre></div>
     </div>
   </main><aside class="col-md-3"><nav id="toc" aria-label="Table of contents"><h2>On this page</h2>
diff --git a/reference/orderly_run_info.html b/reference/orderly_run_info.html
index c4e7c4a9..2a83e13d 100644
--- a/reference/orderly_run_info.html
+++ b/reference/orderly_run_info.html
@@ -21,7 +21,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_search.html b/reference/orderly_search.html
index 1de1b584..868f1ed2 100644
--- a/reference/orderly_search.html
+++ b/reference/orderly_search.html
@@ -17,7 +17,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_search_options.html b/reference/orderly_search_options.html
index 84e36bf1..e37491c1 100644
--- a/reference/orderly_search_options.html
+++ b/reference/orderly_search_options.html
@@ -15,7 +15,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_shared_resource.html b/reference/orderly_shared_resource.html
index 00f8856d..62f42ff5 100644
--- a/reference/orderly_shared_resource.html
+++ b/reference/orderly_shared_resource.html
@@ -17,7 +17,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_strict_mode.html b/reference/orderly_strict_mode.html
index 9757c1cb..4d6fc2c9 100644
--- a/reference/orderly_strict_mode.html
+++ b/reference/orderly_strict_mode.html
@@ -23,7 +23,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/orderly_validate_archive.html b/reference/orderly_validate_archive.html
index b44c20e9..a2ea2469 100644
--- a/reference/orderly_validate_archive.html
+++ b/reference/orderly_validate_archive.html
@@ -9,7 +9,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/reference/query_index.html b/reference/query_index.html
index 0c7c54a4..f51051a3 100644
--- a/reference/query_index.html
+++ b/reference/query_index.html
@@ -7,7 +7,7 @@
 
     <a class="navbar-brand me-2" href="../index.html">orderly2</a>
 
-    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.20</small>
+    <small class="nav-text text-muted me-auto" data-bs-toggle="tooltip" data-bs-placement="bottom" title="">1.99.21</small>
 
 
     <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbar" aria-controls="navbar" aria-expanded="false" aria-label="Toggle navigation">
diff --git a/search.json b/search.json
index 90c4cbf5..138ae1c8 100644
--- a/search.json
+++ b/search.json
@@ -1 +1 @@
-[{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"type-of-location","dir":"Articles","previous_headings":"","what":"Type of location","title":"Collaborative analysis","text":"orderly2 supports two types locations default: path: orderly2 working copy can find filesystem. Note copy people can see (example network share, cloud-synced file system Dropbox OneDrive) see Sharing packets collaborators using shared file system details http: require running HTTP API, either via outpack_server packit location system somewhat extensible, details subject change. documentation behave regardless location stored mechanism transport, can focus instead workflows.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"an-example","dir":"Articles","previous_headings":"","what":"An example","title":"Collaborative analysis","text":"Suppose two researchers, Alice Bob, collaborating piece analysis. purposes example (find common situation practice) share source tree analysis git. Alice Bob clone repo onto machines, using favourite git client (terminal git, via GitHub desktop, gert) moment, including hidden files (except .git), Alice sees: point, cloning Alice .outpack directory (see ) Alice needs run orderly::orderly_init() first: plan work Bob, sharing results shared server, Alice adds orderly2 configuration orderly2::orderly_location_add(): done, can run analysis: Perhaps takes several goes Alice happy analysis, point something ready share. can “push” final packet onto server: Now, consider Bob. also needs source code cloned, orderly initialised, server added: (, Bob also used name server refer shared server, used anything; similar git’s use origin.) Bob can now query packets available server: seen new “data” packet , can pull locally (TODO: mrc-4414 makes nicer): Now Bob position develop packet Alice ran (20240708-173319-f5e76d59)","code":"fs::dir_tree(\".\", all = TRUE, glob = \".git\", invert = TRUE) ## . ## ├── orderly_config.yml ## └── src ##     └── data ##         └── data.R orderly2::orderly_list_src() ## [1] \"data\" orderly2::orderly_init(\".\") ## ✔ Created orderly root at '/tmp/RtmpYA62oL/file1aaa1c2bcfff/alice' ## ✔ Wrote '.gitignore' orderly2::orderly_list_src() ## [1] \"data\" orderly2::orderly_location_add(   \"server\", \"http\", list(url = \"http://orderly.example.com\")) id <- orderly2::orderly_run(\"data\") ## ℹ Starting packet 'data' `20240708-173319-f5e76d59` at 2024-07-08 17:33:19.96726 ## > orderly2::orderly_artefact(\"Final data\", \"data.rds\") ## > saveRDS(mtcars, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240708-173319-f5e76d59 at 2024-07-08 17:33:20.026166 (0.05890656 secs) orderly2::orderly_location_push(id, \"server\") orderly2::orderly_init(\".\") ## ✔ Created orderly root at '/tmp/RtmpYA62oL/file1aaa1c2bcfff/bob' ## ✔ Wrote '.gitignore' orderly2::orderly_location_add(   \"server\", \"http\", list(url = \"http://orderly.example.com\")) orderly2::orderly_metadata_extract(   name = \"data\",   options = list(allow_remote = TRUE, pull_metadata = TRUE)) ##                         id name parameters ## 1 20240708-173319-f5e76d59 data orderly2::orderly_location_pull_packet(id) ## ℹ Looking for suitable files already on disk ## ℹ Need to fetch 2 files (1.3 kB) from 1 location ## ⠙ Fetching file 1/2 (81 B) from 'server' | ETA:  0s [3ms] ## ✔ Fetched 2 files (1.3 kB) from 'server' in 21ms. ##"},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"possible-working-patterns","dir":"Articles","previous_headings":"","what":"Possible working patterns","title":"Collaborative analysis","text":"seen several broad patterns distributing packets. Central server, push allowed: case ; flexible requires everyone relaxed packets created ultimately trusting colleagues exercising good environment hygiene. Just rules enforced computer though, doesn’t mean might ideas within group can push. may useful push HPC system running computationally demanding tasks . Central server, final copy run : case, Alice Bob can run things server, “canonical” copies, nothing pushed server. Reports can run server ssh-ing onto server running packets manually (TODO: mrc-4412) using web interface (possible OrderlyWeb possible packit). approach works well primary goal confidence packets everyone works dependencies created “clean” environment unexpected global dependencies. web version, can also enforce things like running default branch. Alice Bob end collection packets local archives mix canonical ones (server) locally-created ones, local ones always dead end never get shared anyone. result, Alice Bob may delete archive directories without great concern. running packets, make sure packet run similar way packet run server, Bob may want ensure dependencies found server considered. , can create search options pass orderly2::orderly_run running report. developing, Bob may find passing orderly2::orderly_interactive_set_search_options() helpful mean calls orderly2::orderly_description() use packets server. Staging production servers: might set multiple servers, one staging, one production. staging servers never source truth might (might ) allow people push , less strict rules gets run (perhaps allowing running packets feature branch). setting, users end packets created locally, staging machines production server local archive. staging server probably end mix packets packets production. approach specifying search options useful choosing included final packet.","code":"search_options <- orderly2::orderly_search_options(location = \"server\")"},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"pulling-complete-trees-or-not","dir":"Articles","previous_headings":"","what":"Pulling complete trees, or not","title":"Collaborative analysis","text":"pull packets server (orderly2::orderly_location_pull_packet()), depending settings either end just packet request, packet plus recursive dependencies. controlled configuration option core.require_complete_tree, FALSE default; suitable users, servers set core.require_complete_tree = TRUE. situations , users never holding canonical source packets - matter don’t dependencies. hold metadata packets (, recursively) download dependencies later wanted inspect . Typically though, just extra space used disk. push packet server, always pushes complete tree.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"moving-as-little-data-as-possible","dir":"Articles","previous_headings":"","what":"Moving as little data as possible","title":"Collaborative analysis","text":"pulling packets, first user pulls metadata server. orderly2::orderly_location_metadata_pull run, metadata server updated client (mrc-4444 make lazier). now know everything server holds, hashes files contained packet. likely already know files within packet requested, client first looks see exist within archive, verifying changed. requests server files resolve locally. recursive pull (core.require_complete_tree = TRUE) look union files set packets currently missing archive. frequently much less full set files within packets. algorithm similar push: look full dependency chain target packet find file ids ask server yet know unknown subset get union file hashes ask server subset files yet know push files - done one time present done simultaneously async requests future files sever upload missing metadata, chronological order server verifies packet metadata dependencies referenced files multi-step process means avoid copying data already known , well avoiding moving data multiple times. also means process pushing (pulling) interrupted can safely resumed state always consistent.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"interaction-between-source-tree-and--outpack","dir":"Articles","previous_headings":"","what":"Interaction between source tree and .outpack","title":"Collaborative analysis","text":"important .outpack directory shared via git; warn now, can use orderly2::ordery_gitignore_update() automatically create suitable .gitignore file prevent accidentally committed. However, Alice Bob starting new machine : clone source repository run orderly2::orderly_init() within directory run orderly2 commands normal Without running orderly2::orderly_init(), get error prompting initialise repo. requirement different repositories share configuration options passed orderly2::orderly_init(); can enable disable file store different sets locations enabled, depending workflows. means , practice, association convention set related orderly2 locations source tree can orderly2 repository points locations refer different source trees!","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"sharing-packets-with-collaborators-using-a-shared-file-system","dir":"Articles","previous_headings":"","what":"Sharing packets with collaborators using a shared file system","title":"Collaborative analysis","text":"One simplest ways share packets collaborator shared file system, example network share, cloud-synced file system Dropbox OneDrive. can follows. Create new folder orderly remote shared file system. Alice synced path_sharepoint_alice. Initialise orderly location shared file system Create orderly store file store complete tree. See orderly2::orderly_init() details. Add location Push packets want share available collaborator pull. Note data pushed file store .outpack directory. .dot files considered hidden operating system, “show hidden files” file browser see pushed packet. collaborator can now pull shared file. , : Sync drive location machine. Bob synced path_sharepoint_bob Add location Pull metadata use packets desired","code":"orderly2::orderly_init(   root = path_sharepoint_alice,    path_archive = NULL,    use_file_store = TRUE,    require_complete_tree = TRUE ) ## ✔ Created orderly root at '/tmp/RtmpYA62oL/file1aaa1c2bcfff/sharepoint' orderly2::orderly_location_add(   \"sharepoint\", \"path\", list(path = path_sharepoint_alice)) orderly2::orderly_location_push(id, \"sharepoint\") orderly2::orderly_location_add(   \"alices_orderly\", \"path\", list(path = path_sharepoint_bob)) orderly2::orderly_location_pull_metadata(   \"alices_orderly\" )"},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"using-dependencies","dir":"Articles","previous_headings":"","what":"Using dependencies","title":"Dependencies between packets","text":", show practically use dependencies common scenarios increasing complexity. code examples purposefully -simple order keep presentation straightforward, see end document discussion complex pieces code might “optimally” .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"basic-use","dir":"Articles","previous_headings":"Using dependencies","what":"Basic use","title":"Dependencies between packets","text":"primary mechanism using dependencies call orderly2::orderly_dependency() within orderly file; finds suitable completed packet copies files found within packet current report. src/analysis/analysis.R contains: , ’ve used orderly2::orderly_dependency() pull file data.rds recent version (latest()) data packet, ’ve used file normal make plot, ’ve saved analysis.png (similar example vignette(\"introduction\"), get us started). look metadata packet created analysis report, can see used 20240708-173323-6fc634f2 dependency: (indeed , one copy data packet pick ).","code":"## src ## ├── analysis ## │   └── analysis.R ## └── data ##     ├── data.R ##     └── data.csv orderly2::orderly_dependency(\"data\", \"latest()\", \"data.rds\") d <- readRDS(\"data.rds\") png(\"analysis.png\") plot(y ~ x, d) dev.off() id1 <- orderly2::orderly_run(\"data\") ## ℹ Starting packet 'data' `20240708-173323-6fc634f2` at 2024-07-08 17:33:23.443321 ## > d <- read.csv(\"data.csv\") ## > d$z <- resid(lm(y ~ x, d)) ## > saveRDS(d, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240708-173323-6fc634f2 at 2024-07-08 17:33:23.479339 (0.03601766 secs) id2 <- orderly2::orderly_run(\"analysis\") ## ℹ Starting packet 'analysis' `20240708-173323-81eaf8d3` at 2024-07-08 17:33:23.513865 ## > orderly2::orderly_dependency(\"data\", \"latest()\", \"data.rds\") ## ℹ Depending on data @ `20240708-173323-6fc634f2` (via latest(name == \"data\")) ## > d <- readRDS(\"data.rds\") ## > png(\"analysis.png\") ## > plot(y ~ x, d) ## > dev.off() ## agg_png  ##       2 ## ✔ Finished running analysis.R ## ℹ Finished 20240708-173323-81eaf8d3 at 2024-07-08 17:33:23.700104 (0.1862397 secs) orderly2::orderly_metadata(id2)$depends ##                     packet                  query        files ## 1 20240708-173323-6fc634f2 latest(name == \"data\") data.rds...."},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"filtering-candidates-by-parameters","dir":"Articles","previous_headings":"Using dependencies","what":"Filtering candidates by parameters","title":"Dependencies between packets","text":"example, query simple — recently created packet name data. One common pattern see analysis might parameter (example country name) downstream analysis might share parameter want pull data country. src/data/data.R containing: can run several values cyl: follow-analysis contains: query latest(parameter:cyl == :cyl) says “find recent packet ’s parameter”cyl” (parameter:cyl) parameter currently running report (:cyl).","code":"## src ## ├── analysis ## │   └── analysis.R ## └── data ##     └── data.R orderly2::orderly_parameters(cyl = NULL) d <- mtcars[mtcars$cyl == cyl, ] saveRDS(d, \"data.rds\") orderly2::orderly_run(\"data\", list(cyl = 4)) ## ℹ Starting packet 'data' `20240708-173324-17521339` at 2024-07-08 17:33:24.094335 ## ℹ Parameters: ## • cyl: 4 ## > orderly2::orderly_parameters(cyl = NULL) ## > d <- mtcars[mtcars$cyl == cyl, ] ## > saveRDS(d, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240708-173324-17521339 at 2024-07-08 17:33:24.126508 (0.0321734 secs) ## [1] \"20240708-173324-17521339\" orderly2::orderly_run(\"data\", list(cyl = 6)) ## ℹ Starting packet 'data' `20240708-173324-2529100c` at 2024-07-08 17:33:24.148738 ## ℹ Parameters: ## • cyl: 6 ## > orderly2::orderly_parameters(cyl = NULL) ## > d <- mtcars[mtcars$cyl == cyl, ] ## > saveRDS(d, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240708-173324-2529100c at 2024-07-08 17:33:24.179934 (0.03119636 secs) ## [1] \"20240708-173324-2529100c\" orderly2::orderly_run(\"data\", list(cyl = 8)) ## ℹ Starting packet 'data' `20240708-173324-336c93ff` at 2024-07-08 17:33:24.203909 ## ℹ Parameters: ## • cyl: 8 ## > orderly2::orderly_parameters(cyl = NULL) ## > d <- mtcars[mtcars$cyl == cyl, ] ## > saveRDS(d, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240708-173324-336c93ff at 2024-07-08 17:33:24.230692 (0.02678275 secs) ## [1] \"20240708-173324-336c93ff\" orderly2::orderly_parameters(cyl = NULL) orderly2::orderly_dependency(   \"data\",   \"latest(parameter:cyl == this:cyl)\",   \"data.rds\") d <- readRDS(\"data.rds\") png(\"analysis.png\") plot(mpg ~ disp, d) dev.off() orderly2::orderly_run(\"analysis\", list(cyl = 4)) ## ℹ Starting packet 'analysis' `20240708-173324-62818858` at 2024-07-08 17:33:24.387952 ## ℹ Parameters: ## • cyl: 4 ## > orderly2::orderly_parameters(cyl = NULL) ## > orderly2::orderly_dependency( ## +   \"data\", ## +   \"latest(parameter:cyl == this:cyl)\", ## +   \"data.rds\") ## ℹ Depending on data @ `20240708-173324-17521339` (via latest(parameter:cyl == this:cyl && name == \"data\")) ## > d <- readRDS(\"data.rds\") ## > png(\"analysis.png\") ## > plot(mpg ~ disp, d) ## > dev.off() ## agg_png  ##       2 ## ✔ Finished running analysis.R ## ℹ Finished 20240708-173324-62818858 at 2024-07-08 17:33:24.44188 (0.05392814 secs) ## [1] \"20240708-173324-62818858\""},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"interpreting-errors","dir":"Articles","previous_headings":"Using dependencies","what":"Interpreting errors","title":"Dependencies between packets","text":"query fails resolve candidate error: error message tries fairly self explanatory; failed find packet satisfies querylatest(parameter:cyl == :cyl && name == \"data\"); note report name data become part query, two conditions matched . error suggests running rlang::last_error()$explanation information, can : orderly_query_explain object, tries come reasons query might matched; ’ll expand future let us know might like see. tells query can decomposed two subqueries (match parameter cyl 9000), matched packets B (match packet name data), matched 3 packets. subquery matched packets pairs don’t try guide towards problematic pairs. can also ask orderly2 explain query : save object can explore detail: (worked rlang::last_error()$explanation$parts$). can also use orderly2::orderly_metadata_extract work values might looked :","code":"orderly2::orderly_run(\"analysis\", list(cyl = 9000)) ## ℹ Starting packet 'analysis' `20240708-173324-8673176b` at 2024-07-08 17:33:24.528374 ## ℹ Parameters: ## • cyl: 9000 ## > orderly2::orderly_parameters(cyl = NULL) ## > orderly2::orderly_dependency( ## +   \"data\", ## +   \"latest(parameter:cyl == this:cyl)\", ## +   \"data.rds\") ## ✖ Error running analysis.R ## ℹ Finished 20240708-173324-8673176b at 2024-07-08 17:33:24.59256 (0.06418586 secs) ## Error in `orderly2::orderly_run()`: ## ! Failed to run report ## Caused by error in `outpack_packet_use_dependency()`: ## ! Failed to find packet for query 'latest(parameter:cyl == this:cyl && ##   name == \"data\")' ## ℹ See 'rlang::last_error()$explanation' for details rlang::last_error()$explanation ## Evaluated query: 'latest(A && B)' and found 0 packets ## • A (parameter:cyl == this:cyl): 0 packets ##  ## • B (name == \"data\"): 3 packets orderly2::orderly_query_explain(   quote(latest(parameter:cyl == 9000)), name = \"data\") ## Evaluated query: 'latest(A && B)' and found 0 packets ## • A (parameter:cyl == 9000): 0 packets ##  ## • B (name == \"data\"): 3 packets explanation <- orderly2::orderly_query_explain(   quote(latest(parameter:cyl == 9000)), name = \"data\") explanation$parts$B ## $name ## [1] \"B\" ##  ## $str ## [1] \"name == \\\"data\\\"\" ##  ## $expr ## name == \"data\" ##  ## $n ## [1] 3 ##  ## $found ## [1] \"20240708-173324-17521339\" \"20240708-173324-2529100c\" ## [3] \"20240708-173324-336c93ff\" orderly2::orderly_metadata_extract(   name = \"data\",   extract = c(cyl = \"parameters.cyl is number\")) ##                         id cyl ## 1 20240708-173324-17521339   4 ## 2 20240708-173324-2529100c   6 ## 3 20240708-173324-336c93ff   8"},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"filtering-candidates-in-other-ways","dir":"Articles","previous_headings":"Using dependencies","what":"Filtering candidates in other ways","title":"Dependencies between packets","text":"saw two types filtering candidates: latest() selected recent packet latest(parameter:cyl == :cyl) found packet whose parameter matched one parameters. used latest(parameter:cyl == 4) hard code specific parameter value, used latest(parameter:cyl == environment:cyl) match whatever value cyl took evaluating environment. Instead query, can provide single id (e.g, 20240708-173324-62818858), mean even new copies data packet created, dependency always resolve value. can chain together logical operations && (sides must true) || (either side must true), group conditions parentheses. addition ==, usual complement comparison operators work. might complex queries like practice people queries series restrictions &&.","code":"latest((parameter:x == 1 || parameter:x == 2) && parameter:y > 10)"},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"computing-dependencies-and-using-many-dependencies-at-once","dir":"Articles","previous_headings":"Using dependencies","what":"Computing dependencies and using many dependencies at once","title":"Dependencies between packets","text":"One common pattern map-reduce pattern set orderly reports. , set packets created vector parameters, perhaps chain different reports parameter, combined together. parameter p takes values “x”, “y” “z”, might look like: , D want combine three copies C packet, one p “x”, “y” “z”. Especially three values hard coded, might just write Note call vary second argument select different parameter value, third argument naming destination file different name (end three files data/). can write loop: , second argument use environment:p fetch value p calling environment - looping value take three values. name third argument, use special interpolation format ${p} substitute value p build filename.","code":"B(p = \"x\") -- C(p = \"x\")   /                          \\ A - B(p = \"y\") -- C(p = \"y\") - D   \\                          /     B(p = \"z\") -- C(p = \"z\") orderly2::orderly_dependency(\"C\", quote(latest(parameter:p == \"x\")),                              c(\"data/x.rds\" = \"result.rds\")) orderly2::orderly_dependency(\"C\", quote(latest(parameter:p == \"y\")),                              c(\"data/y.rds\" = \"result.rds\")) orderly2::orderly_dependency(\"C\", quote(latest(parameter:p == \"z\")),                              c(\"data/z.rds\" = \"result.rds\")) for (p in c(\"x\", \"y\", \"z\")) {   orderly2::orderly_dependency(\"C\", quote(latest(parameter:p == environment:p)),                                c(\"data/${p}.rds\" = \"result.rds\")) }"},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"how-dependencies-interact-with-locations","dir":"Articles","previous_headings":"","what":"How dependencies interact with locations","title":"Dependencies between packets","text":"default, packet unpacked local archive considered candidate inclusion orderly_dependency(). always want. locations selected, packets within considered candidates can controlled search_options argument orderly2::orderly_run (note argument orderly_run(), orderly_dependency() effect controlled runner report, writer report). three components affect packets selected location: character vector locations, matching location names. packets can found locations considered. mix locally created packets well ones people can see, specifying location = \"server\" limit packets available server, means end dependencies colleagues also get. allow_remote: controls willing download files location order satisfy dependency. TRUE, run report, might download files recent packets available location locally. pull_metadata: effect allow_remote also TRUE; causes metadata refreshed dependency resolution. discussion details ?orderly_run","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"other-points","dir":"Articles","previous_headings":"","what":"Other points","title":"Dependencies between packets","text":"used systems like targets, easy make reports smaller need . ’s real need make small, picking right size challenge. small, ’ll end writing lot code orchestrate running different reports pulling things together. ’ll end spending lot time whether things “date” one another really group things always wants run together. ’re big might end work want , order make change part piece analysis must run whole thing .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"installation","dir":"Articles","previous_headings":"","what":"Installation","title":"Introduction to orderly2","text":"","code":"install.packages(   \"orderly2\",   repos = c(\"https://mrc-ide.r-universe.dev\", \"https://cloud.r-project.org\"))"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"creating-an-empty-orderly-repository","dir":"Articles","previous_headings":"","what":"Creating an empty orderly repository","title":"Introduction to orderly2","text":"first step initialise empty orderly2 repository. orderly2 repository directory file orderly_config.yml within , since version 2 also directory .outpack/. Files within .outpack/ directory never directly modified users directory excluded version control (see orderly2::orderly_gitignore_update). Create orderly2 repository calling orderly2::orderly_init(): creates files: step performed completely empty directory, otherwise error thrown. Later, re-initialise orderly2 repository cloning new machine, working others; discussed vignette(\"collaboration\"). orderly_config.yml file contains little default: vignette, created orderly root R’s per-session temporary directory, deleted R exits. want use directory persist across restarting R (certainly want using orderly2 real project!) replace path within home directory, location control. rest vignette evaluate commands within directory, changing directory path ’ve created:","code":"path <- tempfile() # we'll use a temporary directory here - see note below orderly2::orderly_init(path) ## ✔ Created orderly root at '/tmp/RtmplJ4AWR/file1b3524c63db5' ## . ## ├── .outpack ## │   ├── config.json ## │   ├── location ## │   └── metadata ## └── orderly_config.yml minimum_orderly_version: \"1.99.0\" setwd(path)"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"creating-your-first-orderly-report","dir":"Articles","previous_headings":"","what":"Creating your first orderly report","title":"Introduction to orderly2","text":"orderly report directory src/<name> containing orderly file <name>.R. file may special commands , now ’ll create one simple possible; ’ll create random data save disk. seems silly, imagine standing something like: downloading file external site resource running simulation saving output fitting model data merging set files together create final data set directory structure (ignoring .outpack) looks like: src/incoming_data/incoming_data.R contains: run report create new packet, use orderly2::orderly_run(): id created new identifier packet unique among packets (within reason) chronologically sortable. packet id sorts another packet’s id started packet. run report, directory structure looks like: things changed : file created ran report (data.rds; see script ) log happened report run packet created incoming_data.R data.csv, original input come source tree empty directory draft/incoming_data created orderly ran report first place addition, quite files changed within .outpack directory, covered . ’s ! Notice initial script just plain R script, can develop interactively within src/incoming_data directory. Note however, paths referred within relative src/incoming_data orderly repository root. important reports see world relative incoming_data.R file. created, can refer report id pull files wherever need , context another orderly report just copy desktop email someone. example, copy file data.rds created location outside orderly’s control copies data.rds new temporary directory dest name final.rds. uses orderly2’s outpack_ functions, designed interact outpack archives regardless created (orderly2 program creates outpack archives). Typically lower-level orderly_ functions.","code":"## . ## ├── orderly_config.yml ## └── src ##     └── incoming_data ##         ├── data.csv ##         └── incoming_data.R d <- read.csv(\"data.csv\") d$z <- resid(lm(y ~ x, d)) saveRDS(d, \"data.rds\") id <- orderly2::orderly_run(\"incoming_data\") ## ℹ Starting packet 'incoming_data' `20240708-173327-dd6e3856` at 2024-07-08 17:33:27.869865 ## > d <- read.csv(\"data.csv\") ## > d$z <- resid(lm(y ~ x, d)) ## > saveRDS(d, \"data.rds\") ## ✔ Finished running incoming_data.R ## ℹ Finished 20240708-173327-dd6e3856 at 2024-07-08 17:33:27.932913 (0.06304812 secs) id ## [1] \"20240708-173327-dd6e3856\" ## . ## ├── archive ## │   └── incoming_data ## │       └── 20240708-173327-dd6e3856 ## │           ├── data.csv ## │           ├── data.rds ## │           └── incoming_data.R ## ├── draft ## │   └── incoming_data ## ├── orderly_config.yml ## └── src ##     └── incoming_data ##         ├── data.csv ##         └── incoming_data.R dest <- tempfile() fs::dir_create(dest) orderly2::orderly_copy_files(id, files = c(\"final.rds\" = \"data.rds\"),                              dest = dest)"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"depending-on-packets-from-another-report","dir":"Articles","previous_headings":"","what":"Depending on packets from another report","title":"Introduction to orderly2","text":"Creating new dataset mostly useful someone else can use . introduce first special orderly commands can use orderly file src/ directory now looks like: src/analysis/analysis.R contains: , ’ve used orderly2::orderly_dependency() pull file data.rds recent version (latest()) data packet filename incoming.rds, ’ve used file normal make plot, ’ve saved analysis.png. can run just , using orderly2::orderly_run(): information dependencies, see vignette(\"dependencies\").","code":"## src ## ├── analysis ## │   └── analysis.R ## └── incoming_data ##     ├── data.csv ##     └── incoming_data.R orderly2::orderly_dependency(\"incoming_data\", \"latest()\",                              c(\"incoming.rds\" = \"data.rds\")) d <- readRDS(\"incoming.rds\") png(\"analysis.png\") plot(y ~ x, d) dev.off() id <- orderly2::orderly_run(\"analysis\") ## ℹ Starting packet 'analysis' `20240708-173328-62b7c059` at 2024-07-08 17:33:28.388776 ## > orderly2::orderly_dependency(\"incoming_data\", \"latest()\", ## +                              c(\"incoming.rds\" = \"data.rds\")) ## ℹ Depending on incoming_data @ `20240708-173327-dd6e3856` (via latest(name == \"incoming_data\")) ## > d <- readRDS(\"incoming.rds\") ## > png(\"analysis.png\") ## > plot(y ~ x, d) ## > dev.off() ## agg_png  ##       2 ## ✔ Finished running analysis.R ## ℹ Finished 20240708-173328-62b7c059 at 2024-07-08 17:33:28.469544 (0.08076811 secs)"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"available-in-report-orderly-commands","dir":"Articles","previous_headings":"","what":"Available in-report orderly commands","title":"Introduction to orderly2","text":"function orderly2::orderly_dependency() designed operate packet runs. functions act adding metadata final packet, perhaps copying files directory. orderly2::orderly_description(): Provide longer name description report; can reflected tooling uses orderly metadata much informative short name. orderly2::orderly_parameters(): Declares parameters can passed control behaviour report. Parameters key-value pairs simple data (booleans, numbers, strings) report can respond . can also used queries orderly2::orderly_dependency() find packets satisfy criteria. orderly2::orderly_resource(): Declares file resource; file input report, comes source directory. default, orderly treats files directory resource, can useful mark explicitly, necessary “strict mode” (see ). Files marked resource immutable may deleted modified. orderly2::orderly_shared_resource(): Copies file “shared resources” directory shared/, can data files source code located root orderly repository. can reasonable way sharing data commonly used code among several reports. orderly2::orderly_artefact(): Declares file (set files) created report, even run. makes easier check report behaves expected can allow reasoning related set reports without running . declaring something artefact (especially conjunction “strict mode”) also easier clean src directories used interactive development (see ). orderly2::orderly_dependency(): Copy files one packet packet runs, seen . orderly2::orderly_strict_mode(): Declares report run “strict mode” (see ). addition, also function orderly::orderly_run_info() can used running report returns information currently running report (id, resolved dependencies etc). Let’s add additional annotations previous reports: , ’ve added block special orderly commands; go anywhere, example files refer . strict mode enabled (see ) orderly2::orderly_resource calls must go files used made available point (see ).","code":"orderly2::orderly_strict_mode() orderly2::orderly_resource(\"data.csv\") orderly2::orderly_artefact(\"Processed data\", \"data.rds\")  d <- read.csv(\"data.csv\") d$z <- resid(lm(y ~ x, d)) saveRDS(d, \"data.rds\") id <- orderly2::orderly_run(\"incoming_data\") ## ℹ Starting packet 'incoming_data' `20240708-173328-ada1f7ef` at 2024-07-08 17:33:28.683199 ## > orderly2::orderly_strict_mode() ## > orderly2::orderly_resource(\"data.csv\") ## > orderly2::orderly_artefact(\"Processed data\", \"data.rds\") ## > d <- read.csv(\"data.csv\") ## > d$z <- resid(lm(y ~ x, d)) ## > saveRDS(d, \"data.rds\") ## ✔ Finished running incoming_data.R ## ℹ Finished 20240708-173328-ada1f7ef at 2024-07-08 17:33:28.708968 (0.02576971 secs)"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"parameterised-reports","dir":"Articles","previous_headings":"","what":"Parameterised reports","title":"Introduction to orderly2","text":"Much flexibility comes orderly graph comes using parameterised reports; reports take set parameters change behaviour based parameters. Downstream reports can depend parameterised report filter based suitable parameters. example, consider simple report generate samples based parameter: creates report single parameter n_samples default value 10. used define parameter default, defined multiple parameters can anything report switches value parameter: might read different URLs fetch different underlying data might fit different analysis might read different shared resources (see ) might depend different dependencies might produce different artefacts However, see parameters relatively heavyweight things try consistent set packets created report. report use control size generated data set. resulting file 15 rows, parameter passed affected report: can use parameters orderly’s search functions. example can find recent version packet running: can also pass parameter queries : can used within orderly2::orderly_dependency() (name == \"random\" part implied first name argument), example case report querying also parameters can use within query, using prefix. suppose downstream report simply uses n number samples might write: depend recent packet called random parameter n_samples value current report’s parameter n. See outpack query documentation much detail .","code":"orderly2::orderly_parameters(n_samples = 10) x <- seq_len(n_samples) d <- data.frame(x = x, y = x + rnorm(n_samples)) saveRDS(d, \"data.rds\") orderly2::orderly_parameters(n_samples = NULL) orderly2::orderly_parameters(n_samples = 10, distribution = \"normal\") id <- orderly2::orderly_run(\"random\", list(n_samples = 15)) ## ℹ Starting packet 'random' `20240708-173328-eb7818c4` at 2024-07-08 17:33:28.922947 ## ℹ Parameters: ## • n_samples: 15 ## > orderly2::orderly_parameters(n_samples = 10) ## > x <- seq_len(n_samples) ## > d <- data.frame(x = x, y = x + rnorm(n_samples)) ## > saveRDS(d, \"data.rds\") ## ✔ Finished running random.R ## ℹ Finished 20240708-173328-eb7818c4 at 2024-07-08 17:33:28.954906 (0.0319581 secs) orderly2::orderly_copy_files(id, files = c(\"random.rds\" = \"data.rds\"),                              dest = dest) readRDS(file.path(dest, \"random.rds\")) ##     x          y ## 1   1  0.4463006 ## 2   2  2.6289820 ## 3   3  5.0650249 ## 4   4  2.3690106 ## 5   5  5.5124269 ## 6   6  4.1369885 ## 7   7  6.4779875 ## 8   8  7.9473981 ## 9   9  9.5429963 ## 10 10  9.0859252 ## 11 11 11.4681544 ## 12 12 12.3629513 ## 13 13 11.6954565 ## 14 14 14.7377763 ## 15 15 16.8885049 orderly2::orderly_search('latest(name == \"random\")') ## [1] \"20240708-173328-eb7818c4\" orderly2::orderly_search('latest(name == \"random\" && parameter:n_samples > 10)') ## [1] \"20240708-173328-eb7818c4\" orderly2::orderly_dependency(\"random\", \"latest(parameter:n_samples > 10)\",                              c(\"randm.rds\" = \"data.rds\")) orderly2::orderly_dependency(\"random\", \"latest(parameter:n_samples == this:n)\",                              c(\"randm.rds\" = \"data.rds\"))"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"shared-resources","dir":"Articles","previous_headings":"","what":"Shared resources","title":"Introduction to orderly2","text":"Sometimes useful share data different reports, example common source utilities don’t warrant package, common data. , create directory shared orderly root put files directories might want share. Suppose shared directory contains file data.csv: can write orderly report use_shared uses shared file, use_shared.R containing: can run : resulting archive, file used shared directory present: general property orderly: tries save inputs alongside final results analysis, later can check see went analysis might changed versions.","code":"## . ## ├── archive ## │   ├── analysis ## │   │   └── 20240708-173328-62b7c059 ## │   │       ├── analysis.R ## │   │       ├── analysis.png ## │   │       └── incoming.rds ## │   ├── incoming_data ## │   │   ├── 20240708-173327-dd6e3856 ## │   │   │   ├── data.csv ## │   │   │   ├── data.rds ## │   │   │   └── incoming_data.R ## │   │   └── 20240708-173328-ada1f7ef ## │   │       ├── data.csv ## │   │       ├── data.rds ## │   │       └── incoming_data.R ## │   └── random ## │       └── 20240708-173328-eb7818c4 ## │           ├── data.rds ## │           └── random.R ## ├── draft ## │   ├── analysis ## │   ├── incoming_data ## │   └── random ## ├── orderly_config.yml ## ├── shared ## │   └── data.csv ## └── src ##     ├── analysis ##     │   └── analysis.R ##     ├── incoming_data ##     │   ├── data.csv ##     │   └── incoming_data.R ##     └── random ##         └── random.R orderly2::orderly_shared_resource(\"data.csv\") orderly2::orderly_artefact(\"analysis\", \"analysis.png\")  d <- read.csv(\"data.csv\") png(\"analysis.png\") plot(y ~ x, d) dev.off() id <- orderly2::orderly_run(\"use_shared\") ## ℹ Starting packet 'use_shared' `20240708-173329-6b8c373f` at 2024-07-08 17:33:29.4233 ## > orderly2::orderly_shared_resource(\"data.csv\") ## > orderly2::orderly_artefact(\"analysis\", \"analysis.png\") ## > d <- read.csv(\"data.csv\") ## > png(\"analysis.png\") ## > plot(y ~ x, d) ## > dev.off() ## agg_png  ##       2 ## ✔ Finished running use_shared.R ## ℹ Finished 20240708-173329-6b8c373f at 2024-07-08 17:33:29.460283 (0.03698277 secs) ## archive/use_shared ## └── 20240708-173329-6b8c373f ##     ├── analysis.png ##     ├── data.csv ##     └── use_shared.R"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"strict-mode","dir":"Articles","previous_headings":"","what":"Strict mode","title":"Introduction to orderly2","text":"previous version orderly (orderly1; see vignette(\"migrating\")) fussy input strictly declared report run, clear really required order run something. version 2 relaxed default, can opt old behaviours checks adding anywhere within orderly file (conventionally top). may make granular future, adding : copy files source directory (src/<reportname>/) draft directory report runs (draft/<reportname>/<packet-id>) declared orderly2::orderly_resource; leaves behind extra files left development warn end running packet files found part artefact Using strict mode also helps orderly2 clean src/<reportname> directory effectively interactive development (see next section).","code":"orderly2::orderly_strict_mode()"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"interactive-development","dir":"Articles","previous_headings":"","what":"Interactive development","title":"Introduction to orderly2","text":"Set working directory src/<reportname> orderly script fully executable (e.g., source Rstudio’s Source button, R’s source() function). Dependencies copied needed. , mix files within source directory. recommend per-source-directory .gitignore keep files version control (see ). soon implement support cleaning generated files directory. example, suppose interactively run incoming_data/incoming_data.R script, leave behind generated files. can report orderly2::orderly_cleanup_status: files unknown orderly tell prompt tell explicitly. can clean generated files running (suggested message): dry_run = TRUE argument can pass want see deleted without using status function. can also keep files git using orderly2::orderly_gitignore_update function: creates (updates) .gitignore file within report generated files included git. already accidentally committed gitignore real effect git surgery, see git manuals handy, profane, guide.","code":"orderly2::orderly_cleanup_status(\"incoming_data\") ## ✖ incoming_data is not clean: ## ℹ 1 file can be deleted by running 'orderly2::orderly_cleanup(\"incoming_data\")': ##   • data.rds orderly2::orderly_cleanup(\"incoming_data\") ## ℹ Deleting 1 file from 'incoming_data': ## • data.rds orderly2::orderly_gitignore_update(\"incoming_data\") ## ✔ Wrote 'src/incoming_data/.gitignore'"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"deleting-things-from-the-archive","dir":"Articles","previous_headings":"","what":"Deleting things from the archive","title":"Introduction to orderly2","text":"delete packets archive/ directory puts orderly2 inconsistent state metadata store. Sometimes matter (e.g., delete old copies never candidates inclusion orderly2::orderly_dependency never notice). However, delete recent copy packet try depend , get error. moment, two copies incoming_data task: run analysis task, pull recent version (20240708-173328-ada1f7ef). However, deleted manually (e.g., save space accidentally) corrupted (e.g., opening output Excel letting save changes) able included, running analysis fail: error tries fairly informative, telling us failed copying files 20240708-173328-ada1f7ef found packet corrupt, file data.rds found archive. also suggests fix; can tell orderly2 20240708-173328-ada1f7ef “orphaned” considered inclusion look dependencies. can carry suggestion just validate packet running can validate packets : option core.require_complete_tree enabled, process also look packets used now-deleted packet orphan , longer complete tree includes . want remove references orphaned packets, can use orderly2::orderly_prune_orphans() remove entirely:","code":"orderly2::orderly_metadata_extract(   name = \"incoming_data\",   extract = c(time = \"time.start\")) ##                         id                time ## 1 20240708-173327-dd6e3856 2024-07-08 17:33:27 ## 2 20240708-173328-ada1f7ef 2024-07-08 17:33:28 orderly2::orderly_run(\"analysis\") ## ℹ Starting packet 'analysis' `20240708-173330-0b2519a5` at 2024-07-08 17:33:30.046688 ## > orderly2::orderly_dependency(\"incoming_data\", \"latest()\", ## +                              c(\"incoming.rds\" = \"data.rds\")) ## ✖ Error running analysis.R ## ℹ Finished 20240708-173330-0b2519a5 at 2024-07-08 17:33:30.128869 (0.08218169 secs) ## Error in `orderly2::orderly_run()`: ## ! Failed to run report ## Caused by error in `orderly_copy_files()`: ## ! Unable to copy files, due to deleted packet 20240708-173328-ada1f7ef ## ℹ Consider 'orderly2::orderly_validate_archive(\"20240708-173328-ada1f7ef\", ##   action = \"orphan\")' to remove this packet from consideration ## Caused by error: ## ! File not found in archive ## ✖ data.rds orderly2::orderly_validate_archive(\"20240708-173328-ada1f7ef\", action = \"orphan\") orderly2::orderly_validate_archive(action = \"orphan\") ## ✔ 20240708-173327-dd6e3856 (incoming_data) is valid ## ✔ 20240708-173328-62b7c059 (analysis) is valid ## ✖ 20240708-173328-ada1f7ef (incoming_data) is invalid due to its files ## ✔ 20240708-173328-eb7818c4 (random) is valid ## ✔ 20240708-173329-6b8c373f (use_shared) is valid orderly2::orderly_prune_orphans() ## ℹ Pruning 1 orphan packet"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"debugging-and-coping-with-errors","dir":"Articles","previous_headings":"","what":"Debugging and coping with errors","title":"Introduction to orderly2","text":"(written)","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"interaction-with-version-control","dir":"Articles","previous_headings":"","what":"Interaction with version control","title":"Introduction to orderly2","text":"guidelines: Make sure exclude files git listing .gitignore: .outpack/ - nothing suitable version control archive/ - core.archive_path set non-null value, excluded. default archive draft/ - temporary draft directory orderly_envir.yml - used setting machine-specific configuration absolutely version control files: src/ main source analyses orderly_config.yml - high level configuration suitable sharing shared resource directory (configured orderly_config.yml) probably version controlled source repository end multiple people’s machines, configured differently. configuration option set via orderly2::orderly_config_set designed (potentially) different different users, configuration needs version controlled. also means reports/packets can’t directly refer values set . includes directory used save archive packets (enabled) names locations (equivalent git remotes). may find useful include scripts help users set common locations, like git, different users may use different names remote (e.g., one user may location called data another called data-incoming, depending perspective use location). orderly2 always try save information current state git source repository alongside packet metadata. includes current branch, commit (sha) remote url. try create links final version packet upstream source repository.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"interaction-with-the-outpack-store","dir":"Articles","previous_headings":"","what":"Interaction with the outpack store","title":"Introduction to orderly2","text":"alluded , .outpack directory contains lots information packets run, typically “bounds” normal use. effectively “database” information packets run. Understanding directory structured required using orderly, included avoidance mystery! See outpack documentation (vignette(\"outpack\")) details ideas . work , directory structure looks like: can perhaps inferred filenames, files .outpack/metadata/<packet-id> metadata packet run. files .outpack/location/<location-id>/<packet-id> holds information packet first known location (location special “local” location). default orderly configuration store final files directory called archive/, alternatively (additionally) can use content- addressable file store. enabled, .outpack directory looks like: files .outpack/files/ never modified deleted. approach storage naturally deduplicates file archive, large file used many places ever stored .","code":"## .outpack ## ├── config.json ## ├── index ## │   └── outpack.rds ## ├── location ## │   ├── local ## │   │   ├── 20240708-173327-dd6e3856 ## │   │   ├── 20240708-173328-62b7c059 ## │   │   ├── 20240708-173328-eb7818c4 ## │   │   └── 20240708-173329-6b8c373f ## │   └── orphan ## └── metadata ##     ├── 20240708-173327-dd6e3856 ##     ├── 20240708-173328-62b7c059 ##     ├── 20240708-173328-eb7818c4 ##     └── 20240708-173329-6b8c373f ## .outpack ## ├── config.json ## ├── files ## │   └── sha256 ## │       ├── 0a ## │       │   └── a82571c21c4e5f1f435e8bef2328dda5ef47e177d78d63d1c4ec647a5a388a ## │       ├── 25 ## │       │   └── 4947c281b203719c72949745123a1d017e2f9b50c048b1d24a0803d73ba0b8 ## │       ├── 40 ## │       │   └── f8c6c3993686ee8acf38ce936911d6f65372d489e78748ec6d1f27d277daa7 ## │       ├── 42 ## │       │   └── 438203a847c7cd496e3e603ddeeceb4e136f1bd18f1efb81332fa093964f7b ## │       ├── 51 ## │       │   └── f8070f04f57a8040ad215474b976bcddc891bfc544e6c6a0519e74d79cb104 ## │       ├── 5f ## │       │   └── 96f49230c2791c05706f24cb2335cd0fad5d3625dc6bca124c44a51857f3f8 ## │       ├── a6 ## │       │   └── 80ab7c65a52327a3d9c5499d114f513f18eabe7f63a98f9fc308c2b3744c82 ## │       ├── d9 ## │       │   └── 1699ae410cbd811e1f028f8a732e5162b7df854eec08d921141f965851272d ## │       ├── de ## │       │   └── 1329bc0e9f8ee0c8b3376eb09b71e93587f707dfeaa01d1a03ae32f97c928a ## │       └── ec ## │           └── b53285781a4d36c65168c80ee14f2af2c885423c6166b9425f40c3c6cd8297 ## ├── index ## │   └── outpack.rds ## ├── location ## │   ├── local ## │   │   ├── 20240708-173327-dd6e3856 ## │   │   ├── 20240708-173328-62b7c059 ## │   │   ├── 20240708-173328-eb7818c4 ## │   │   └── 20240708-173329-6b8c373f ## │   └── orphan ## └── metadata ##     ├── 20240708-173327-dd6e3856 ##     ├── 20240708-173328-62b7c059 ##     ├── 20240708-173328-eb7818c4 ##     └── 20240708-173329-6b8c373f"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"relationship-between-orderly-and-outpack","dir":"Articles","previous_headings":"","what":"Relationship between orderly and outpack","title":"Introduction to orderly2","text":"orderly2 package built metadata file storage system called outpack; implementing support working metadata archives languages (see outpack_server server implementation Rust outpack-py Python). metadata discussed detail vignette(\"metadata\") document general ideas fully mrc-ide/outpack","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"basic-overview","dir":"Articles","previous_headings":"","what":"Basic overview","title":"Outpack metadata","text":"“packet” conceptually directory, corresponding particular analysis data product, though necessarily stored. internal representation includes: metadata packet - files contains hashes along information done create environment created etc (see ) actual file contents every included file. might stored simple content-addressable file storage system (see ) actual copies directories originally created Every packet referenced uniquely primary key. use key format encodes current date time, well random data avoid collisions. exists dependency graph among packets, one packet depends another. edge graph hard link (one packet another id) also query (e.g., latest packet name) used find packet. means many ways looking thinking dependency graph. packets available locally, outpack repositories, typically (always) machines accessed HTTP API. conceptually similar git “remotes”. need distinguish packets “unpacked” (, packets every file available current archive) packets merely known (metadata files). sometimes refer unpacked packets “local” known “local” location special. use terms “archive” “repository” fairly interchangeably try nail .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"what-goes-into-a-packet","dir":"Articles","previous_headings":"","what":"What goes into a packet","title":"Outpack metadata","text":"packet must things: name (example model_fits). changed (rather changes tracked) currently way namespacing different repositories id (see “outpack id” ) created outpack point packet started set parameters - mapping strings scalar type (boolean, number, string). Packets may zero many parameters. set files - local (relative) path within packet (e.g., output/data.csv) also hash (e.g., sha256:69f6cf230416cf40828da251a0dad17cbbf078587883e826f3345ff08d1aaa7d) Information packet run (start end time), represented seconds post epoch, UTC (see ). addition may contain information : dependencies used, tracking name included packet, query used find , resolved id, set files extracted packet (source name, hash destination name) names scripts run create packet (typically also included files within final packet) information state git point packet run (current branch, hash, etc) information system used run packet (format unspecified might include hostname, operating system version, package/module versions, etc) custom metadata added implementation","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"types-of-users-of-outpack","dir":"Articles","previous_headings":"","what":"Types of users of outpack","title":"Outpack metadata","text":"types “persona” outpack user imagine exist guide decisions abut layout . extremes : user desktop wants develop run analyses, using files present packets available server. want visually inspect results running packets, need see files disk way makes sense human, care complete copy dependencies graph. server instance responsible delivering packets users (e.g., persona ). expected (able fetch) part outpack graph, expect anyone want visually inspect packets directly machine (, storage need human-readable). impacts two configuration options associated parts directory structure : user use content-addressable “file store” human-readable “archive” (explained detail ) require archive contains complete tree dependencies? expect first persona wants human readable archive contain full tree, second wants opposite.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"directory-layout","dir":"Articles","previous_headings":"","what":"Directory layout","title":"Outpack metadata","text":"section discusses files directory make outpack work, much come ; see . typical .outpack directory layout looks like : (note archive/ .outpack level). directories necessarily present; indeed required file .outpack/config.json.","code":".outpack/   config.json   files/   location/   metadata/ archive/"},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"configuration--outpackconfig-json","dir":"Articles","previous_headings":"Directory layout","what":"Configuration (.outpack/config.json)","title":"Outpack metadata","text":"outpack configuration schema defined config.json configuration format still subject change…","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"packet-metadata--outpackmetadata","dir":"Articles","previous_headings":"Directory layout","what":"Packet metadata (.outpack/metadata/)","title":"Outpack metadata","text":"file within directory filename outpack id (matching regular expression ^[0-9]{8}-[0-9]{6}-[0-9a-f]{8}$, see . file json file conforming schema metadata.json. present means outpack implementation can report information back packet (created, files contains, etc), packet metadata meaningful ; want know might come (location distributing packet) copy packet locally.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"location-information--outpacklocation","dir":"Articles","previous_headings":"Directory layout","what":"Location information (.outpack/location/)","title":"Outpack metadata","text":"directory matches regular expression ^[0-9]{8}$ (e.g., 457f4f2a) “location id” (see ) corresponding “location”. file within directory outpack id name, contains json location unpacked (installed) packet, hash metadata. file conforms schema location.json.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"a-file-store--outpackfiles","dir":"Articles","previous_headings":"Directory layout","what":"A file store (.outpack/files)","title":"Outpack metadata","text":"configuration option core.use_file_store true, outpack keeps content addressable file store files knows . much space efficient entire packet unpacked automatically deduplicates shared content among packets (e.g., large file present two packets stored ). file store layout described . storage format human-readable (indeed present within hidden directory .outpack). can enabled either server user","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"an-archive-archive-by-default","dir":"Articles","previous_headings":"Directory layout","what":"An archive (archive/ by default)","title":"Outpack metadata","text":"configuration option core.path_archive non-null directory path containing unpacked packets. packet available path <name> “name” packet, <id> outpack id. several files per packet, possibly directories. storage approach designed human readable, typically enabled outpack repository used laptop user wants interactively work files.","code":"archive/<name>/<id>/<files...>"},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"adding-a-packet","dir":"Articles","previous_headings":"","what":"Adding a packet","title":"Outpack metadata","text":"order make packet available locally, need import metadata files, mark packet available. roughly creating packet (.e., first place packet ever existed) importing packet elsewhere. Making packet available allows used dependency, allows serving packet acting location (http file protocols), guarantees files actually present locally.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"adding-metadata","dir":"Articles","previous_headings":"Adding a packet","what":"Adding metadata","title":"Outpack metadata","text":"can simply copy metadata file .outpack/metadata/<packet id> yet exist. make available anything yet known location. Dangling metadata (, metadata present directory known anywhere) currently mostly ignored.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"adding-files","dir":"Articles","previous_headings":"Adding a packet","what":"Adding files","title":"Outpack metadata","text":"repository uses file store, fill first, much easier think . can easily get difference files used packet (list files packet manifest) already file store looking hash turn. request missing files insert store. may leave “dangling” files (files referred packet) problem. repository human-readable archive uses file store, files present file store easy enough check file store requested path (local relative path packet manifest). update file store first, files guaranteed present. repository uses human readable archive, simplest thing request file remote. However, might efficient check locally previously fetched copies files content, verify modified, copy place rather re-downloading.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"marking-the-packet-as-unpacked-and-known-locally","dir":"Articles","previous_headings":"Adding a packet","what":"Marking the packet as unpacked and known locally","title":"Outpack metadata","text":"local location id, write file conforming location.json schema, containing packet id, time marked unpacked hash metadata.","code":".outpack/<local location id>/<packet id>"},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"ordering-of-operations","dir":"Articles","previous_headings":"Adding a packet","what":"Ordering of operations","title":"Outpack metadata","text":"need files metadata packet marked unpacked; note configurations guarantee every packet unpacked complete tree. can import files first, metadata; lot disadvantage either. mark package unpacked known locally though components present.","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"outpack-ids","dir":"Articles","previous_headings":"Details","what":"Outpack ids","title":"Outpack metadata","text":"Outpack ids match regular expression ^[0-9]{8}-[0-9]{6}-[0-9a-f]{8}$; encode UTC date-time prefix YYYYMMDD-HHMMSS- followed 8 hexadecimal digits. R implementation, encode current second first four digits (2 bytes) append 2 bytes cryptographically random data. id tries balance reasonable degree collision resistance (65536 combinations per millisecond), lexicographic sortability reasonable degree meaningfulness.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"location-ids","dir":"Articles","previous_headings":"Details","what":"Location ids","title":"Outpack metadata","text":"Location ids meaningless 4-byte (8 character) hex strings. immutable created different different machines even point location. location id mapped (via .outpack/config.json) location name human-readable name (e.g., production staging). requirement name different machines. One directories represents local location; can find mapping within configuration.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"representing-hashes","dir":"Articles","previous_headings":"Details","what":"Representing hashes","title":"Outpack metadata","text":"Outpack typically uses sha256 hashes, want able change future. wherever hash presented, algorithm included part string. example instead used md5 algorithm written schema currently assumes hash value represented hex string.","code":"sha256:69f6cf230416cf40828da251a0dad17cbbf078587883e826f3345ff08d1aaa7d md5:bd57f7123c6bfb95c3234ff56373b7f4"},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"times","dir":"Articles","previous_headings":"Details","what":"Times","title":"Outpack metadata","text":"store information times places (e.g., times packet run, imported, etc). Rather trying deal strings, always store time seconds since 1970-01-01 00:00.00 UTC (including fractional seconds, whatever accuracy system allows).","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"file-store","dir":"Articles","previous_headings":"Details","what":"File store","title":"Outpack metadata","text":"file store designed simple, sophisticated git, whose object store similar thing. general layout looks like: hopefully fairly obvious structure. Paths format: reason second level prevent performance degradation directories containing millions files, copying git. store designed cope different hashing algorithms, though R implementation outpack supports sha256 now. Multiple hashing algorithms supported hard linking content multiple places tree, might link ","code":".outpack/files   sha256/     5d/       dfaf1f4a2e15e8fe46dbed145bf2f84bba1b3367d0a56f73de08f8585dd153       ...     77/     ... <algorithm>/<first two bytes>/<remaining bytes> sha256/5d/dfaf1f4a2e15e8fe46dbed145bf2f84bba1b3367d0a56f73de08f8585dd153 md5/84/0bc6ad3ae479dccc1c49a1910b37bd"},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"so-long-yaml-and-thanks-for-all-the-whitespace-errors","dir":"Articles","previous_headings":"Summary of changes","what":"So long YAML and thanks for all the whitespace errors","title":"Migrating from orderly (1.x)","text":"obvious user-facing change (almost) YAML, definition inputs outputs report now defined within orderly file, <reportname>.R. orderly report previously orderly.yml file looked like end within orderly file looks like: think much clearer, comes documentation autocomplete support IDEs. fact, simple reports, special functions required, though ’ll find useful (see vignette(\"orderly2\")) specific changes: longer need list packages packages:, instead just use library() ordinary script. record state session regardless get record used longer need use sources: list scripts want source, instead use source() normal global_resources become orderly2::orderly_shared_resource (aren’t really global much shared). Note directory now always shared/ orderly root, may configure .","code":"parameters:   n_min:     default: 10 script: script.R source:   - functions.R resources:   - metadata.csv depends:   raw_data:     id: latest     use:       raw_data.csv: data.csv artefacts:   data:     description: Processed data     filenames: data.rds orderly2::orderly_parameters(n_min = 10) orderly2::orderly_dependency(\"raw_data\", \"latest\",                               files = c(\"raw_data.csv\" = \"data.csv\")) orderly2::orderly_resource(\"metadata.csv\") orderly2::orderly_artefact(\"Processed data\", \"data.rds\") source(\"functions.R\")"},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"implications","dir":"Articles","previous_headings":"Summary of changes > So long YAML and thanks for all the whitespace errors","what":"Implications","title":"Migrating from orderly (1.x)","text":"change widespread implications: can program things like dependencies, creating loop series parameter values, conditionally depending reports R script can basis orderly report, can add orderly functions annotating inputs outputs progressively","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"database-support-has-been-moved-into-a-plugin","dir":"Articles","previous_headings":"Summary of changes","what":"Database support has been moved into a plugin","title":"Migrating from orderly (1.x)","text":"version 1, built-support accessing data SQL databases; moved within orderly.db plugin. major features supported.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"no-more-commit","dir":"Articles","previous_headings":"Summary of changes","what":"No more commit","title":"Migrating from orderly (1.x)","text":"orderly2 longer requires separate orderly_commit() call orderly_run(); longer make distinction local draft archive packets. Instead, added finer-grained control dependencies resolved (locally, subset servers), generalises way draft/archive used practice. See ?orderly_run details dependencies resolved. implications deleting things; draft directory always easy target deletion, now deletion need tell orderly2 deleted things. See vignette(\"introduction\") details (section “Deleting things archive”).","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"no-more-testing-or-development-mode","dir":"Articles","previous_headings":"Summary of changes","what":"No more testing or development mode","title":"Migrating from orderly (1.x)","text":"two different, unsatisfactory, mechanisms developing orderly report: orderly_test_start (version 1.1.0) orderly_develop_start (1.1.0 onwards) worked initial setup, copying dependencies etc location work (new draft orderly_test_start source directory orderly_develop_start). orderly2 can just work directly within source directory, long working directory set src/<report-name>, orderly2 commands work expected. orderly1, need careful commit (git) results running analysis, encourage per-report .gitignore files help .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"new-language-agnostic-backend","dir":"Articles","previous_headings":"Summary of changes","what":"New, language-agnostic, backend","title":"Migrating from orderly (1.x)","text":"biggest change, perhaps least visible, orderly now built open spec outpack can implemented language. develop Python implementation , possibly languages. takes control metadata. split “orderly_” “outpack_” functions package, information see last section vignette(\"introduction\") also vignette(\"outpack\").","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"other-smaller-changes","dir":"Articles","previous_headings":"Summary of changes","what":"Other, smaller changes","title":"Migrating from orderly (1.x)","text":"“Global” resources become “shared” resources, always live shared/ orderly root (.e., longer configurable). reason want reports able run fairly independently orderly2 configuration (exception plugins). practice people really vary .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"what-is-missing-compared-with-orderly1","dir":"Articles","previous_headings":"","what":"What is missing compared with orderly1","title":"Migrating from orderly (1.x)","text":"changelog feature (implement something support ) extraction secrets vault (support ) automatic handling README files (may implement support )","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"how-to-migrate","dir":"Articles","previous_headings":"","what":"How to migrate","title":"Migrating from orderly (1.x)","text":"two parts migration: updating canonical copy orderly archive (ideally one ) updating source tree. steps done via outpack.orderly package. migrate archive first. every archive want retain (might archives stored locally, production servers staging servers). Archive migration happens place; , modify anything original location. archive old used old versions orderly1 possible process hiccups. Please let us know case. result process end new directory contains new archive conforming outpack spec containing orderly2 metadata. Next, migrate source tree. done place done fresh clone source git repository. report, examine orderly.yml files script files (often script.R), delete , write new orderly file adapt report work orderly2. possible perfect might need minor tweaking hopefully reasonable. One thing preserved (probably ) comments yaml often refer yaml formatting orderly1 features hopefully much problem. probably want manually tweak generated code anyway, take advantage new orderly2 features able compute dependencies. using OrderlyWeb, probably need pause migrating, replacement yet ready.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"what-about-the-packages","dir":"Articles","previous_headings":"","what":"What about the packages?","title":"Migrating from orderly (1.x)","text":"merge orderly2 orderly package, ready release can use . However, anticipate period coexistence legacy orderly1 systems develop orderly2. help small helper package orderly.helper can smooth namespace differences; may useful interact versions.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"what-were-the-problems-in-version-1","dir":"Articles","previous_headings":"","what":"What were the problems in version 1","title":"Migrating from orderly (1.x)","text":"YAML format inflexible, error prone users, leads duplication focussed around initial needs Vaccine Impact Modelling Consortium fairly easy get archive SQLite database inconsistent state (e.g., deleting moving files archive) SQLite database behaved poorly shared file systems","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/orderly2.html","id":"collaborative-analysis","dir":"Articles","previous_headings":"","what":"Collaborative analysis","title":"orderly2","text":"Many analyses involve single person single machine; case number workflow tools make orchestrating analysis easy. workflow model graph dependencies analysis, data flows. example might update data, whole pipeline rerun. update code forecast, forecasts report rerun. experience, model works well single-user setting falls collaborative setting, especially analysis partitioned person; Alice handling data pipeline, Bob running fits, Carol organising forecasts final report. context, changes upstream affect downstream analysis, require set care around integration might used version controlling source code. example, Alice dealing change incoming data format going break analysis time Bob trying get model fits working, Bob trying integrate code changes Alice’s new data. typically deal source code using branches within git; code Bob work branch isolated Alice’s changes. contexts like () data analysis products git. needed way versioning outputs step analysis controlling integrated subsequent analyses. Another way looking problem seek way making analysis composable way functions OOP achieve programs, way docker containerisation achieved deploying software. need way putting interfaces around pieces analysis allow people refer fetch somewhere run. conceptual pieces needed : way referring unambiguously (globally) piece analysis can depended upon, everyone can agree ’re talking piece analysis system storage results running analysis can shared among group system control integration pieces analysis takes place refer transportable unit analysis “packet”. conceptually directory files created running code, atomic unit work point view orderly. packet underlying source form, anyone can run. However, time people use pre-run packets collaborators run inputs onward analyses (see vignette(\"dependencies\") vignette(\"collaboration\") details).","code":"[raw data] -> [processed data] -> [model fits] -> [forecasts] -> [report]"},{"path":"https://mrc-ide.github.io/orderly2/articles/orderly2.html","id":"reproducible-analyses","dir":"Articles","previous_headings":"","what":"Reproducible analyses","title":"orderly2","text":"degree collaboration style requires reproducibility, several aspects . system describe , even though everyone can typically run step analysis, typically don’t. differs workflow tools, users may familiar .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/orderly2.html","id":"difference-from-a-workflow-system","dir":"Articles","previous_headings":"Reproducible analyses","what":"Difference from a workflow system","title":"orderly2","text":"Workflow systems hugely influential scientific computing, people co-opting build systems like make sophisticated systems designed parallel running large complex workflows nextflow. general approach define interdependencies among parts analysis, forming graph parts analysis track inputs outputs workflow. model computation lots good points: defines interface analysis allows (encourages) breaking monolithic analysis component pieces can reasoned allows high-level parallelism making obvious parts workflow can run concurrently tracking way data flows analysis, allows minimum amount recalculation done change, downstream parts triggered shared workspace online runner, allows degree collaboration long everyone happy working constantly changing set code analysis artefacts designed orderly2 working patterns suit . motivating reasons include: nodes computational graph expensive compute require exotic hardware. Workflows upstream data never settle, need know version data ends used particular analysis Workflows upstream analyses used many downstream analyses, upstream developer may know much downstream use Nondeterministic analyses, e.g., involving stochastic simulations, rerunning node expected return exact numerical results, can’t rely different users recovering results different occasions[*] cases missing piece need way versioning nodes within computational graph, shifting emphasis automatically rerunning portions graph tracking data flowed graph. turn shifts reproducibility emphasis “everyone run code get results” “everyone run code, instead work results”. familiar docker, approach similar working pre-built docker images, whereas workflow approach similar working directly Dockerfiles; many situations end result , approaches differ guarantees, computation happens, users refer versions. [*] discourage trying force determinism manually setting seeds, potential violate statistical properties random number streams, fragile best.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/orderly2.html","id":"what-is-reproducibility-anyway","dir":"Articles","previous_headings":"Reproducible analyses","what":"What is reproducibility anyway?","title":"orderly2","text":"Reproducibility means different things different people, even within narrow sense “rerun analysis retrieve results”. last decade, idea one able rerun piece analysis retrieve results slightly morphed one must rerun piece analysis. Similarly, emphasis utility reproducibility shifted authors able rerun work (confidence rerun ) hypothetical third party wanting rerun analysis. approach flips perspective around bit, based experiences collaborative research projects, draws (overly) ambitious aim : Can prove given set inputs produced given set outputs? quickly found impossible, provided systems place one satisfied statement given level trust system. piece analysis comes server primary way people run analyses web front-end (currently OrderlyWeb, soon Packit) know analysis run end--end modification orderly2 preserves inputs alongside outputs files present final packet files went analysis, recorded R package versions full set used. system naturally involves running multiple machines (typically analysts’ laptops, server perhaps HPC environment), way orderly2 treats paths, practically little problem getting analyses working multiple places, trivially satisfying typical reproducibility aim, even though people typically focussed . shift focus proved valuable. analysis run one occasion (e.g., regular reporting, simply updating figure final submission manuscript revision), outputs may change. Understanding changes happened important. orderly2 automatically saves lot metadata run easy find things might changed. , can start interrogating graph among packets find effect change ; find previously run packets pulled old version data set, used previous release package.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"the-basic-idea","dir":"Articles","previous_headings":"","what":"The basic idea","title":"Creating plugins","text":"plugin provided package, possibly thing package provides. plugin name must (currently) package name. functions package needs call orderly2::orderly_plugin orderly2::orderly_plugin_register create register plugin, respectively. make plugin available orderly project, two new bits configuration may present orderly_config.yml - one declares plugin used, configures plugin. use plugin individual report, functions plugin used, configure use plugin. Finally, can save information back final orderly2 metadata plugin . yaml-less design orderly2 (see vignette(\"migrating\") familiar orderly1), line plugin just package code fairly blurred, reasons writing plugin typically want make something easier reports, want action reflected orderly metadata.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"an-example","dir":"Articles","previous_headings":"","what":"An example","title":"Creating plugins","text":"example, ’ll implement stripped version database plugin inspired work (see `orderly.db fuller implementation). make work need functions: …process additional fields orderly_config.yml describe find database …can called orderly file access database …can add metadata final orderly metadata done ’ll start report side things, describing want happen, work implementation. directory structure minimal project orderly_config.yml file contains information shared possible uses plugin - case connection information database: plugin called example.db listed within plugins section, along configuration; case indicating path SQLite file can loaded . example.R file contains information use database specific report; case, making results query SELECT * mtcars cyl == 4 database available R object dat Normally, imagine calculation kept minimal purpose demonstration. implement need : create package write function handle configuration orderly_config.yml write function query() used example.R query ","code":"## . ## ├── orderly_config.yml ## └── src ##     └── example ##         └── example.R minimum_orderly_version: 1.99.0 plugins:   example.db:     path: /tmp/RtmpFoww4i/file1c2637c713b9 dat <- example.db::query(\"SELECT * FROM mtcars WHERE cyl == 4\") orderly2::orderly_artefact(\"Summary of data\", \"data.rds\")  saveRDS(summary(dat), \"data.rds\")"},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"create-a-tiny-package","dir":"Articles","previous_headings":"An example","what":"Create a tiny package","title":"Creating plugins","text":"lots package skeleton tools , favourite, usethis::create_package() probably reasonable job. thing package needs contain Imports: orderly2 DESCRIPTION field. simple package may structure like , DESCRIPTION file contains: NAMESPACE R/plugin.R files shown .","code":"## . ## ├── DESCRIPTION ## ├── NAMESPACE ## └── R ##     └── plugin.R Package: example.db Version: 0.0.1 License: CC0 Title: Orderly Database Example Plugin Description: Simple example of an orderly plugin. Authors@R: person('Orderly Authors', role = c('aut', 'cre'),     email = 'email@example.com') Imports: orderly2"},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"handle-the-configuration","dir":"Articles","previous_headings":"An example","what":"Handle the configuration","title":"Creating plugins","text":"required function plugin needs provide one process data orderly_config.yml. probably primarily concerned validation can fairly simple first, later ’ll expand report errors nicely: arguments data: deserialised section orderly_config.yml specific plugin filename: full path orderly_config.yml return value data argument auxiliary data added validation.","code":"db_config <- function(data, filename) {   data }"},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"evaluate-the-query","dir":"Articles","previous_headings":"An example","what":"Evaluate the query","title":"Creating plugins","text":"Finally, minimal example, need function actually query; example example.db::query: arguments whatever want user provide – nothing special orderly2. important function call orderly2::orderly_plugin_context returns information can use make plugin work. explained ?orderly2::orderly_plugin_context, example use just one element, config, configuration plugin (.e., return value function db_config); see orderly2::orderly_plugin_context context can accessed . last bit package code register plugin, calling orderly2::orderly_plugin_register within .onLoad() special R function called package loaded. means whenever packages loaded (regardless whether attached) register plugin. (important name argument matches package name, orderly2 trigger loading package based name configuration; may support multiple plugins within one package later.) Note query function appear within registration, just function read process configuration. final (minimal) package code : NAMESPACE file contains","code":"query <- function(sql) {   ctx <- orderly2::orderly_plugin_context(\"example.db\")   dbname <- ctx$config$path   con <- DBI::dbConnect(RSQLite::SQLite(), dbname)   on.exit(DBI::dbDisconnect(con))   DBI::dbGetQuery(con, sql) } .onLoad <- function(...) {   orderly2::orderly_plugin_register(     name = \"example.db\",     config = db_config) } db_config <- function(data, filename) {   data }  query <- function(sql) {   ctx <- orderly2::orderly_plugin_context(\"example.db\")   dbname <- ctx$config$path   con <- DBI::dbConnect(RSQLite::SQLite(), dbname)   on.exit(DBI::dbDisconnect(con))   DBI::dbGetQuery(con, sql) }  .onLoad <- function(...) {   orderly2::orderly_plugin_register(     name = \"example.db\",     config = db_config) } export(query)"},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"trying-it-out","dir":"Articles","previous_headings":"An example","what":"Trying it out","title":"Creating plugins","text":"order test package, needs loaded. can either installing package using pkgload::load_all() (may find pkgload::load_all(export_all = FALSE) gives reliable experience. Now, can run report:","code":"pkgload::load_all() ## ℹ Loading example.db orderly2::orderly_run(\"example\", root = path_root) ## ℹ Starting packet 'example' `20240708-173338-4f7ea6f4` at 2024-07-08 17:33:38.315095 ## > dat <- example.db::query(\"SELECT * FROM mtcars WHERE cyl == 4\") ## > orderly2::orderly_artefact(\"Summary of data\", \"data.rds\") ## > saveRDS(summary(dat), \"data.rds\") ## ✔ Finished running example.R ## ℹ Finished 20240708-173338-4f7ea6f4 at 2024-07-08 17:33:38.496032 (0.1809373 secs) ## [1] \"20240708-173338-4f7ea6f4\""},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"making-the-plugin-more-robust","dir":"Articles","previous_headings":"","what":"Making the plugin more robust","title":"Creating plugins","text":"plugin fairly fragile validation input data orderly_config.yml orderly.yml. fairly annoying yaml incredibly flexible reporting back information user might gone wrong hard. case, expect single key-value pair orderly_config.yml key path value path SQLite database. can easily expand configuration function report better back user misconfigure plugin: acceptable job preventing poor input suggesting user might look within configuration fix . Note return configuration data , can augment (otherwise change) data need.","code":"db_config <- function(data, filename) {   if (!is.list(data) || is.null(names(data)) || length(data) == 0) {     stop(\"Expected a named list for orderly_config.yml:example.db\")   }   if (length(data$path) != 1 || !is.character(data$path)) {     stop(\"Expected a string for orderly_config.yml:example.db:path\")   }   if (!file.exists(data$path)) {     stop(sprintf(       \"The database '%s' does not exist (orderly_config:example.db:path)\",       data$path))   }   data }"},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"saving-metadata-about-what-the-plugin-did","dir":"Articles","previous_headings":"","what":"Saving metadata about what the plugin did","title":"Creating plugins","text":"Nothing plugin saved report metadata unless save . Partly orderly.yml, saved final directory, serves sort record. However, probably want know something data returned . example might want save query string later can query without read process orderly.yml file statistics size data (e.g., number rows returned, columns) perhaps summary content hash can see content changed different versions report save metadata, use function orderly2::orderly_plugin_add_metadata; takes arguments plugin name, string like structure saved metadata (’ll use query) whatever data want save: function otherwise minimal version . also need provide serialisation function ensure metadata saved expected. saved metadata key query, get list back element query unnamed list many elements query calls given report. , ensure everything except cols length 1 (everything) gets turned scalar (1 [1]) serialise jsonlite::toJSON auto_unbox TRUE. Taking step , can also specify schema metadata conform save file inst/schema.json within package (path within inst fine). Finally, can also add deserialiation hook convert loaded metadata nice data.frame: Now, register plugin, provide path schema, along serialisation deserialisation functions: Now, orderly metadata saved (just running script part report) validate output passed orderly2::orderly_plugin_add_metadata schema, jsonvalidate installed (currently requires development version) R option outpack.schema_validate set TRUE (e.g., running options(outpack.schema_validate = TRUE)). final package structure: DESCRIPTION file NAMESPACE unchanged , schema shown just . plugin.R file contains code collected : (code .R file package, across several).","code":"query <- function(sql) {   ctx <- orderly2::orderly_plugin_context(\"example.db\")   dbname <- ctx$config$path   con <- DBI::dbConnect(RSQLite::SQLite(), dbname)   on.exit(DBI::dbDisconnect(con))   d <- DBI::dbGetQuery(con, sql)   info <- list(sql = sql, rows = nrow(d), cols = names(d))   orderly2::orderly_plugin_add_metadata(\"example.db\", \"query\", info)   d } db_serialise <- function(data) {   for (i in seq_along(data$query)) {     # Always save cols as a vector, even if length 1:     data$query[[i]]$cols <- I(data$query[[i]]$cols)   }   jsonlite::toJSON(data$query, auto_unbox = TRUE) } {     \"$schema\": \"http://json-schema.org/draft-07/schema#\",      \"type\": \"array\",     \"items\": {         \"type\": \"object\",         \"properties\": {             \"sql\": {                 \"type\": \"string\"             },             \"rows\": {                 \"type\": \"number\"             },             \"cols\": {                 \"type\": \"array\",                 \"items\": {                     \"type\": \"string\"                 }             }         },         \"required\": [\"sql\", \"rows\", \"cols\"],         \"additionalProperties\": false     } } .onLoad <- function(...) {   orderly2::orderly_plugin_register(     name = \"example.db\",     config = db_config,     serialise = db_serialise,     deserialise = db_deserialise,     schema = \"schema.json\") } ## . ## ├── archive ## │   └── example ## │       └── 20240708-173338-4f7ea6f4 ## │           ├── data.rds ## │           └── example.R ## ├── draft ## │   └── example ## ├── orderly_config.yml ## └── src ##     └── example ##         └── example.R db_config <- function(data, filename) {   if (!is.list(data) || is.null(names(data)) || length(data) == 0) {     stop(\"Expected a named list for orderly_config.yml:example.db\")   }   if (length(data$path) != 1 || !is.character(data$path)) {     stop(\"Expected a string for orderly_config.yml:example.db:path\")   }   if (!file.exists(data$path)) {     stop(sprintf(       \"The database '%s' does not exist (orderly_config:example.db:path)\",       data$path))   }   data }  query <- function(sql) {   ctx <- orderly2::orderly_plugin_context(\"example.db\")   dbname <- ctx$config$path   con <- DBI::dbConnect(RSQLite::SQLite(), dbname)   on.exit(DBI::dbDisconnect(con))   d <- DBI::dbGetQuery(con, sql)   info <- list(sql = sql, rows = nrow(d), cols = names(d))   orderly2::orderly_plugin_add_metadata(\"example.db\", \"query\", info)   d }  .onLoad <- function(...) {   orderly2::orderly_plugin_register(     name = \"example.db\",     config = db_config,     serialise = db_serialise,     deserialise = db_deserialise,     schema = \"schema.json\") } id <- orderly2::orderly_run(\"example\", root = path_root) ## ℹ Starting packet 'example' `20240708-173339-60aa2af8` at 2024-07-08 17:33:39.380817 ## > dat <- example.db::query(\"SELECT * FROM mtcars WHERE cyl == 4\") ## > orderly2::orderly_artefact(\"Summary of data\", \"data.rds\") ## > saveRDS(summary(dat), \"data.rds\") ## ✔ Finished running example.R ## ℹ Finished 20240708-173339-60aa2af8 at 2024-07-08 17:33:39.422584 (0.04176736 secs) meta <- orderly2::orderly_metadata(id, root = path_root) meta$custom$example.db ##                                   sql rows         cols ## 1 SELECT * FROM mtcars WHERE cyl == 4   11 mpg, cyl...."},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"potential-uses","dir":"Articles","previous_headings":"","what":"Potential uses","title":"Creating plugins","text":"need functionality similar example - pulling database functionality original version orderly something independent, turns useful fraction orderly use-cases. can imagine potential uses though, : Non-DBI-based database data extraction, customised routines pulling data database Download files shared location just use (e.g., SharePoint, OneDrive, AWS). orderly_config.yml contain account connection details orderly.yml contain mapping remote data/files local files. Rather writing environment , use path argument copy files correct place. Pull data web API just running follow basic pattern requiring configuration order able connect resource service, specification resources fetched, action actually fetch resource put place.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"structure-of-queries","dir":"Articles","previous_headings":"","what":"Structure of queries","title":"Orderly query DSL","text":"simple query finds recent packet; unlikely useful without scoping - see . complex queries expressed syntax valid R (also valid Julia close valid Python). complex query composed “tests” Every “test” uses boolean operator (<, >, <=, >=, ==, !=) left right hand side can one : lookup outpack metadata (parameter:x value parameter called x, name name packet, id id packet) lookup provided data pars (:x value pars$x) literal value (e.g., \"some_name\", 1, TRUE) Tests can grouped together (, !, &&, || might expect: parameter:x == 1 || parameter:x == 2 finds packets parameter x 1 2 name == \"data\" && parameter:x > 3 finds packets called “data” parameter x greater 3 (parameter:y == 2) && !(parameter:x == 1 || parameter:x == 2) finds parameter y 2 parameter x anything 1 2 (also written (parameter:y == 2) && (parameter:x != 1 && parameter:x != 2)) four functions latest(expr) finds latest packet satisfying expr - always returns length 1 character, NA_character_ suitable packet found. expr given latest packets returned. single(expr) like latest(expr) except error expr evaluate exactly one packet id usedby(expr, FALSE) expr either literal id expression returns 1 id. finds packets used generation packet id returned expr (see dependencies section details).","code":"latest() name == \"some_name\" parameter:x > 1"},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"special-simple-queries","dir":"Articles","previous_headings":"","what":"Special simple queries","title":"Orderly query DSL","text":"two shorthand queries: latest equivalent latest() (useful applied scope) string matching regular expression id (^([0-9]{8}-[0-9]{6}-[[:xdigit:]]{8})$) equivalent single(id == \"<id>\") \"<id>\" string provided)","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"scoping-queries","dir":"Articles","previous_headings":"","what":"Scoping queries","title":"Orderly query DSL","text":"WARNING: may remove Scoping queries can used shorthand filtering returned packets. future used reduce set packets searched speed query evaluation. join together main query (scope) && (expr), except expr call latest single. case combine latest((scope) && (expr)) single((scope) && (expr)). useful want limit search particular name location perform detailed search. example, query equivalent orderly2 uses functionality resolving dependencies orderly2::orderly_dependency.","code":"orderly_query(quote(parameter:x == 1), scope = quote(name == \"data\")) orderly_query(quote(parameter:x == 1 && name == \"data\"))"},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"scoping-on-name","dir":"Articles","previous_headings":"Scoping queries","what":"Scoping on name","title":"Orderly query DSL","text":"often users want scope name instead passing scope argument shorthand name argument usability. equivalent ","code":"orderly_query(quote(parameter:x == 1), name = \"data\") orderly_query(quote(parameter:x == 1), scope = quote(name == \"data\"))"},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"dependencies","dir":"Articles","previous_headings":"","what":"Dependencies","title":"Orderly query DSL","text":"2 packets, B depends output (.e. call (id_a, ...) running packet B) can draw .  equivalently say dependency B, B depends used B , B uses parent B, B child tree dependencies among packets might want search packets used another packet. can use query function usedby(id) list packets used id. search recursively packets used id parents parents’ parents . optional second arg immediate FALSE default, set TRUE search immediate (e.g. level 1) dependencies. able search dependencies like means packet structure like  want know id used C can find using orderly_search usedby can combined groupings scope: depth usedby recurse can controlled setting depth e.g. search just immediate parents C. depth can positive integer, default depth recurse finds parents. usedby can simplified using subqueries. Subqueries denoted curly braces {} can either named passed subquery arg can anonymous. query equivalent uses subquery C. two important things note usedby: query inside usedby search entire index, ignoring scope name parameters. want find packets used latest C. subquery C scoped return results. query inside usedby must return single result. ensure must either literal id, call latest call single well searching dependency tree using usedby can search uses function. setup reports , B C want know id C uses can find using uses usedby can combined search complex arrangements dependencies. something like  want search version E depends version used latest C can via searches tree C tree find version E. Note important added name == \"\" condition , missing usedby(latest(name == \"C\")) also return B single throw error multiple packets. can also search tree find D e.g. note E packet uses D need add name == \"E\" clause. can combine usedby uses complex searches, find D C run","code":"orderly_search(quote(usedby(latest(name == \"C\"))), name = \"A\") orderly_search(quote(usedby(latest(name == \"C\")) && parameter:year == 2022),                name = \"A\") orderly_search(quote(usedby(latest(name == \"C\"), depth = 1)), name = \"A\") orderly_search(quote(usedby({C}) && parameter:year == 2022),                name = \"A\",                subquery = list(C = quote(latest(name == \"C\")))) orderly_search(quote(uses(latest(name == \"A\"))),                name = \"C\") orderly_search(   quote(latest(uses(single(usedby(latest(name == \"C\")) && name == \"A\")))),   name = \"E\") orderly_query(   quote(usedby(single(uses(name == \"D\")))),   name = \"A\") orderly_query(   quote(usedby(single(uses(single(usedby(latest(name == \"C\")) && name == \"A\"))) && name == \"E\"))),   name = \"D\")"},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"simple-things","dir":"Articles","previous_headings":"Possible future queries and interface improvements","what":"Simple things","title":"Orderly query DSL","text":"orderly1 supports .null(parameter:x) might generalise support However, Python None Julia nothing, complicates things. Alternatively use missing(parameter:x)?","code":"parameter:x == NULL"},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"explain-the-query","dir":"Articles","previous_headings":"Possible future queries and interface improvements","what":"Explain the query","title":"Orderly query DSL","text":"Often, people want know “packet match”? good show query set packets fail query excluded. definitely bit work.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/troubleshooting.html","id":"outpack-files-accidentally-committed-to-git","dir":"Articles","previous_headings":"","what":"Outpack files accidentally committed to git","title":"Troubleshooting","text":"discussed orderly introduction, want commit files .outpack/, drafts/ archive/ (used) git create sorts problems line. directed , probably ended files git want undo situation. least painful way depends situation. now put guard rails try prevent happening, still happen modify .gitignore file force-add files example. situation, orderly2 shout : may directed page. just want continue working anyway, run suggested command: things work warning first time session: subsequent calls display warning: rest section discusses might permanently fix issue.","code":"orderly2::orderly_run(\"data\") ## Error in `orderly2::orderly_run()`: ## ! Detected 6 outpack files committed to git ## ✖ Detected files were found in '.outpack/' and 'archive/' ## ℹ For tips on resolving this, please see ##   <https://mrc-ide.github.io/orderly2/articles/troubleshooting.html> ## ℹ To turn this into a warning and continue anyway set the option ##   'orderly_git_error_is_warning' to TRUE by running ##   options(orderly_git_error_is_warning = TRUE) options(orderly_git_error_is_warning = TRUE) orderly2::orderly_run(\"data\") ## Warning in orderly2::orderly_run(\"data\"): Detected 6 outpack files committed to git ## ✖ Detected files were found in '.outpack/' and 'archive/' ## ℹ For tips on resolving this, please see ##   <https://mrc-ide.github.io/orderly2/articles/troubleshooting.html> ## This warning is displayed once per session. ## ✔ Wrote '.gitignore' ## ℹ Starting packet 'data' `20240708-173345-ec5aeb3c` at 2024-07-08 17:33:45.926435 ## > orderly2::orderly_artefact(\"Final data\", \"data.rds\") ## > saveRDS(mtcars, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240708-173345-ec5aeb3c at 2024-07-08 17:33:45.964067 (0.03763103 secs) ## [1] \"20240708-173345-ec5aeb3c\" orderly2::orderly_run(\"data\") ## ℹ Starting packet 'data' `20240708-173346-0ce7688a` at 2024-07-08 17:33:46.053698 ## > orderly2::orderly_artefact(\"Final data\", \"data.rds\") ## > saveRDS(mtcars, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240708-173346-0ce7688a at 2024-07-08 17:33:46.076102 (0.02240443 secs) ## [1] \"20240708-173346-0ce7688a\""},{"path":"https://mrc-ide.github.io/orderly2/articles/troubleshooting.html","id":"i-dont-care-about-my-history-at-all","dir":"Articles","previous_headings":"Outpack files accidentally committed to git","what":"I don’t care about my history at all","title":"Troubleshooting","text":"case just started project, yet collaborating anyone else (person willing re-clone sources). simplest thing : Delete .git directory entirely Run orderly2::orderly_gitignore_update(\"(root)\") set reasonable .gitignore prevent situation happening Run git add . add everything back (review git status make sure ’re happy) Run git commit -m \"Initial commit\" create single commit contains files currently repo history, also .outpack files previously pushed repo GitHub similar need set git remote add origin https://github.com/user/repo (replacing user/repo path, using git@github.com:user/repo use ssh talk GitHub) git branch -M main assuming using main default branch, now common git push --force -u origin main Note destructive require coordination collaborators changed history.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/troubleshooting.html","id":"i-just-want-this-to-go-away-and-nothing-i-have-committed-is-very-large","dir":"Articles","previous_headings":"Outpack files accidentally committed to git","what":"I just want this to go away and nothing I have committed is very large","title":"Troubleshooting","text":"care history, also committed files (e.g., committed files .outpack/ small large 100MB file archive/ preventing pushing GitHub) just delete offending files git without updating history, affecting local copies. Run git rm --cached .outpack (repeating draft archive needed) Run orderly2::orderly_gitignore_update(\"(root)\") set reasonable .gitignore prevent situation happening Run git add .gitignore also stage Run git commit -m \"Delete unwanted outpack files\" can push without issues.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/troubleshooting.html","id":"i-care-about-my-history-but-want-this-stuff-gone","dir":"Articles","previous_headings":"Outpack files accidentally committed to git","what":"I care about my history but want this stuff gone","title":"Troubleshooting","text":"working branch, unwanted files committed branch, simplest thing copy changes made somewhere safe, create new branch current main copy changes . somewhat automatically generating applying patch: unwanted files committed onto default branch, potentially gory history rewriting. See StackOverflow question, git docs currently recommended tool . Good luck!","code":"git diff -- src > changes.patch git checkout main git checkout -b changes-attempt2 git apply changes.patch git push -u origin changes-attempt2"},{"path":"https://mrc-ide.github.io/orderly2/authors.html","id":null,"dir":"","previous_headings":"","what":"Authors","title":"Authors and Citation","text":"Rich FitzJohn. Author, maintainer. Robert Ashton. Author. Alex Hill. Author. Imperial College Science, Technology Medicine. Copyright holder.","code":""},{"path":"https://mrc-ide.github.io/orderly2/authors.html","id":"citation","dir":"","previous_headings":"","what":"Citation","title":"Authors and Citation","text":"FitzJohn R, Ashton R, Hill (2024). orderly2: Orderly Next Generation. R package version 1.99.20, https://github.com/mrc-ide/orderly2.","code":"@Manual{,   title = {orderly2: Orderly Next Generation},   author = {Rich FitzJohn and Robert Ashton and Alex Hill},   year = {2024},   note = {R package version 1.99.20},   url = {https://github.com/mrc-ide/orderly2}, }"},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"orderly2","dir":"","previous_headings":"","what":"Orderly Next Generation","title":"Orderly Next Generation","text":"attendant hospital responsible non-medical care patients maintenance order cleanliness. soldier carries orders performs minor tasks officer. orderly2 package designed help make analysis reproducible. principal aim automate series basic steps process writing analyses, making easy : track inputs analysis (packages, code, data resources) store multiple versions analysis repeated track outputs analysis create analyses depend outputs previous analyses","code":""},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"get-started","dir":"","previous_headings":"","what":"Get started","title":"Orderly Next Generation","text":"Two vignettes provide overview package, depending tastes: vignette(\"orderly2\") describes problem orderly2 tries solve (read package website) vignette(\"introduction\") describes orderly2 example, walking basic features (read package website)","code":""},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"roadmap","dir":"","previous_headings":"","what":"Roadmap","title":"Orderly Next Generation","text":"package ground-rewrite orderly package, now ready use new projects adventurous users original package. next months orderly renamed orderly1, finalise API package. stable, rename back orderly2 orderly release CRAN replace existing package. migration path existing users described vignette(\"migrating\") updated based experience migrating people. web framework OrderlyWeb rewritten parallel (packit) provide migration path .","code":""},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"installation","dir":"","previous_headings":"","what":"Installation","title":"Orderly Next Generation","text":"Please install r-universe: prefer, can install GitHub remotes:","code":"install.packages(   \"orderly2\",   repos = c(\"https://mrc-ide.r-universe.dev\", \"https://cloud.r-project.org\")) remotes::install_github(\"mrc-ide/orderly2\")"},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"testing","dir":"","previous_headings":"","what":"Testing","title":"Orderly Next Generation","text":"run tests, need outpack_server available system path. One way achieve run tests also require development versions gert jsonvalidate. can installed rOpenSci r-universe, : Without , tests skipped, suite still run.","code":"cargo install --git https://github.com/mrc-ide/outpack_server install.packages(   c(\"gert\", \"jsonvalidate\"),   repos = c(\"https://ropensci.r-universe.dev\", \"https://cloud.r-project.org\"))"},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"licence","dir":"","previous_headings":"","what":"Licence","title":"Orderly Next Generation","text":"MIT © Imperial College Science, Technology Medicine","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_artefact.html","id":null,"dir":"Reference","previous_headings":"","what":"Declare orderly artefacts — orderly_artefact","title":"Declare orderly artefacts — orderly_artefact","text":"Declare artefact. turn number orderly features; see Details . can multiple calls function within orderly script.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_artefact.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Declare orderly artefacts — orderly_artefact","text":"","code":"orderly_artefact(description, files)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_artefact.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Declare orderly artefacts — orderly_artefact","text":"description name artefact files files within artefact","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_artefact.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Declare orderly artefacts — orderly_artefact","text":"Undefined","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_artefact.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Declare orderly artefacts — orderly_artefact","text":"(1) files matching copied src directory draft directory unless also listed resource orderly_resource(). feature enabled call function top level orderly script contains string literals (variables). (2) script fails produce files, orderly_run() fail, guaranteeing task really produce things need . (3) within final metadata, artefacts additional metadata; description provide grouping","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":null,"dir":"Reference","previous_headings":"","what":"Clean up source directory — orderly_cleanup","title":"Clean up source directory — orderly_cleanup","text":"Find, delete, file generated running report. comfortable , strongly recommended run orderly_cleanup_status first see deleted.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Clean up source directory — orderly_cleanup","text":"","code":"orderly_cleanup(name = NULL, dry_run = FALSE, root = NULL, locate = TRUE)  orderly_cleanup_status(name = NULL, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Clean up source directory — orderly_cleanup","text":"name Name report directory clean (.e., look src/<name> relative orderly root dry_run Logical, indicating delete anything, instead just print information root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, just outpack (see orderly_init details). locate Logical, indicating configuration searched .  TRUE config given, orderly looks working directory parents finds .outpack directory","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Clean up source directory — orderly_cleanup","text":"(currently unstable) object class orderly_cleanup_status within element delete indicates files deleted (orderly_cleanup_status) deleted (orderly_cleanup)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Clean up source directory — orderly_cleanup","text":"file deletion, look remove empty directories; orderly2 similar semantics git directories never directly tracked. recent gert (yet CRAN) ask git files ignored; ignored good candidates deletion! encourage keep per-report .gitignore lists files copy source directory, can use information clean files generation. Importantly, even file matches ignore rule committed repository, longer match ignore rule.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"notes-for-user-of-orderly-","dir":"Reference","previous_headings":"","what":"Notes for user of orderly1","title":"Clean up source directory — orderly_cleanup","text":"orderly1 function quite different semantics, full set possible files always knowable yaml file. , start point view list files compare directory.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Clean up source directory — orderly_cleanup","text":"","code":"# Create a simple example: path <- orderly2::orderly_example(\"default\") #> ✔ Created orderly root at '/tmp/RtmpfBuUfo/file174c1393fe1b'  # We simulate running a packet interactively by using 'source'; # you might have run this line-by-line, or with the \"Source\" # button in Rstudio. source(file.path(path, \"src/data/data.R\"), chdir = TRUE) #> Warning: cannot open file '/tmp/RtmpfBuUfo/file174c1393fe1b/src/data/data.R': No such file or directory #> Error in file(filename, \"r\", encoding = encoding): cannot open the connection  # Having run this, the output of the report is present in the # source directory: fs::dir_tree(path) #> Error: [ENOENT] Failed to search directory '/tmp/RtmpfBuUfo/file174c1393fe1b': no such file or directory  # We can detect what might want cleaning up by running # \"orderly_cleanup_status\": orderly2::orderly_cleanup_status(\"data\", root = path) #> Error: Directory does not exist: '/tmp/RtmpfBuUfo/file174c1393fe1b'  # Soon this will print more nicely to the screen, but for now you # can see that the status of \"data.rds\" is \"derived\", which means # that orderly knows that it is subject to being cleaned up; the # \"delete\" element shows what will be deleted.  # Do the actual deletion: orderly2::orderly_cleanup(\"data\", root = path) #> Error: Directory does not exist: '/tmp/RtmpfBuUfo/file174c1393fe1b'"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config.html","id":null,"dir":"Reference","previous_headings":"","what":"Read configuration — orderly_config","title":"Read configuration — orderly_config","text":"Read current orderly configuration, stored within outpack root, along orderly-specific extensions.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Read configuration — orderly_config","text":"","code":"orderly_config(root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Read configuration — orderly_config","text":"root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Read configuration — orderly_config","text":"list configuration options: core: important options outpack store, containing: path_archive: path human-readable packet archive, NULL disabled (set orderly_config_set core.path_archive) use_file_store: Indicates content-addressable file store enabled (core.use_file_store) require_complete_tree: Indicates outpack store requires dependencies fully available (core.require_complete_tree) hash_algorithm: hash algorithm used (currently modifiable) location: Information locations; see orderly_location_add, orderly_location_rename orderly_location_remove interact configuration, orderly_location_list simply list available locations. Returns data.frame columns name, id, priority, type args, args list column. orderly: list orderly-specific configuration; just minimum required version (minimum_orderly_version).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Read configuration — orderly_config","text":"","code":"# A default configuration in a new temporary directory path <- withr::local_tempdir() orderly2::orderly_init(path) #> ✔ Created orderly root at '/tmp/RtmpfBuUfo/file174c66475d05' orderly2::orderly_config(path) #> $core #> $core$path_archive #> [1] \"archive\" #>  #> $core$use_file_store #> [1] FALSE #>  #> $core$require_complete_tree #> [1] FALSE #>  #> $core$hash_algorithm #> [1] \"sha256\" #>  #>  #> $location #>    name  type args #> 1 local local      #>  #> $orderly #> $orderly$minimum_orderly_version #> [1] ‘1.99.0’ #>  #>"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":null,"dir":"Reference","previous_headings":"","what":"Set configuration options — orderly_config_set","title":"Set configuration options — orderly_config_set","text":"Set configuration options. can currently set; expanded time. See Details.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Set configuration options — orderly_config_set","text":"","code":"orderly_config_set(..., options = list(...), root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Set configuration options — orderly_config_set","text":"... Named options set (e.g., pass argument core.require_complete_tree = TRUE) options alternative ..., can pass list named options (e.g., list(core.require_complete_tree = TRUE)).  interface typically easier program . root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Set configuration options — orderly_config_set","text":"Nothing","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Set configuration options — orderly_config_set","text":"Options set order provided.  Currently, setting one option fails, options processed previous ones (rely behaviour, may change). Currently can set: core.require_complete_tree See orderly_init description options.","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Set configuration options — orderly_config_set","text":"","code":"# The default configuration does not include a file store, and # saves output within the \"archive\" directory: path <- withr::local_tempdir() orderly2::orderly_init(path) #> ✔ Created orderly root at '/tmp/RtmpfBuUfo/file174c69243531' fs::dir_tree(path, all = TRUE) #> /tmp/RtmpfBuUfo/file174c69243531 #> ├── .outpack #> │   ├── config.json #> │   ├── location #> │   └── metadata #> └── orderly_config.yml  # Change this after the fact: orderly2::orderly_config_set(core.use_file_store = TRUE,                              core.path_archive = NULL,                              root = path) fs::dir_tree(path, all = TRUE) #> /tmp/RtmpfBuUfo/file174c69243531 #> ├── .outpack #> │   ├── config.json #> │   ├── files #> │   ├── index #> │   │   └── outpack.rds #> │   ├── location #> │   └── metadata #> └── orderly_config.yml"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_copy_files.html","id":null,"dir":"Reference","previous_headings":"","what":"Copy files from a packet — orderly_copy_files","title":"Copy files from a packet — orderly_copy_files","text":"Copy files packet anywhere. Similar orderly_dependency except used active packet context. can use function pull files outpack root directory outside control outpack, example. Note arguments need must provided name, position, exception id query.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_copy_files.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Copy files from a packet — orderly_copy_files","text":"","code":"orderly_copy_files(   ...,   files,   dest,   overwrite = TRUE,   envir = parent.frame(),   options = NULL,   root = NULL,   locate = TRUE )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_copy_files.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Copy files from a packet — orderly_copy_files","text":"... Arguments passed orderly_query, perhaps just query expression files Files copy packet. can (1) character vector, case files copied without changing names, (2) named character vector, case name used destination name, (3) data.frame (including tbl_df, data.frame objects) containing columns , case files copied names . cases, want import directory files packet, must refer source trailing slash (e.g., c(= \"/\")), create local directory /... files upstream packet directory /. omit slash error thrown suggesting add slash intended. can use limited form string interpolation names argument; using ${variable} pick values envir substitute string.  similar interpolation might familiar glue::glue similar, much simpler concatenation fancy features supported. Note unfortunate, (us) avoidable inconsistency ; interpolation values environment query done using environment:x destination filename ${x}. dest directory copy overwrite Overwrite files destination; typically want, set FALSE prefer error thrown destination file already exists. envir Optionally, environment substitute query (using environment: prefix). default use calling environment, can explicitly pass want control lookup happens. options Optionally, orderly_search_options object controlling search performed, packets considered scope. provided, default options used (.e., orderly2::orderly_search_options()) root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_copy_files.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Copy files from a packet — orderly_copy_files","text":"Nothing, invisibly. Primarily called side effect copying files packet directory dest","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_copy_files.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Copy files from a packet — orderly_copy_files","text":"can call function id string, case search packet proceed regardless whether id present.  called arguments (e.g., string match id format, named argument name, subquery parameters) interpret arguments query orderly_search find id. error query return exactly one packet id, probably want use latest(). different ways might fail (recover failure): id known metadata store (known unpacked also known present remote) fail impossible resolve files. Consider refreshing metadata orderly_location_pull_metadata refresh . id unpacked local copy files referred can found, error default (see next option). However, sometimes file refer might also present downloaded packet depended , content file unchanged packet version locally. id unpacked, local copy file allow_remote TRUE try request file whatever remote selected orderly_location_pull_packet packet. Note empty directories might created failure.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_dependency.html","id":null,"dir":"Reference","previous_headings":"","what":"Declare a dependency — orderly_dependency","title":"Declare a dependency — orderly_dependency","text":"Declare dependency another packet","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_dependency.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Declare a dependency — orderly_dependency","text":"","code":"orderly_dependency(name, query, files)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_dependency.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Declare a dependency — orderly_dependency","text":"name name packet depend query query search ; often simply string latest, indicating recent version. may want complex query though. files Files copy packet. can (1) character vector, case files copied without changing names, (2) named character vector, case name used destination name, (3) data.frame (including tbl_df, data.frame objects) containing columns , case files copied names . cases, want import directory files packet, must refer source trailing slash (e.g., c(= \"/\")), create local directory /... files upstream packet directory /. omit slash error thrown suggesting add slash intended. can use limited form string interpolation names argument; using ${variable} pick values envir substitute string.  similar interpolation might familiar glue::glue similar, much simpler concatenation fancy features supported. Note unfortunate, (us) avoidable inconsistency ; interpolation values environment query done using environment:x destination filename ${x}.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_dependency.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Declare a dependency — orderly_dependency","text":"Undefined","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_dependency.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Declare a dependency — orderly_dependency","text":"See orderly_run details search options used select locations packets found , data fetched network. running interactively, obviously work, use orderly_interactive_set_search_options() set options function respond .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_description.html","id":null,"dir":"Reference","previous_headings":"","what":"Describe the current packet — orderly_description","title":"Describe the current packet — orderly_description","text":"Describe current packet","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_description.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Describe the current packet — orderly_description","text":"","code":"orderly_description(display = NULL, long = NULL, custom = NULL)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_description.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Describe the current packet — orderly_description","text":"display friendly name report; displayed locations web interface, packit. given, must scalar character. long longer description report. given, must scalar character. custom additional metadata. given, must named list, elements scalar atomics (character, number, logical).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_description.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Describe the current packet — orderly_description","text":"Undefined","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_example.html","id":null,"dir":"Reference","previous_headings":"","what":"Copy a simple orderly example — orderly_example","title":"Copy a simple orderly example — orderly_example","text":"Copy simple orderly example use docs. function form part workflow!","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_example.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Copy a simple orderly example — orderly_example","text":"","code":"orderly_example(name, ..., dest = NULL)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_example.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Copy a simple orderly example — orderly_example","text":"name name example copy. Currently \"default\" supported. ... Arguments passed orderly_init() dest destination. default use withr::local_tempfile() create temporary directory clean . suitable use orderly examples, may prefer provide path. path must already exist.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_example.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Copy a simple orderly example — orderly_example","text":"Invisibly, path example.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_example.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Copy a simple orderly example — orderly_example","text":"","code":"path <- orderly2::orderly_example(\"default\") #> ✔ Created orderly root at '/tmp/RtmpfBuUfo/file174c5e8a3cc6' orderly2::orderly_list_src(root = path) #> Error: Directory does not exist: '/tmp/RtmpfBuUfo/file174c5e8a3cc6'"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_gitignore_update.html","id":null,"dir":"Reference","previous_headings":"","what":"Update a gitignore file — orderly_gitignore_update","title":"Update a gitignore file — orderly_gitignore_update","text":"Update gitignore, useful prevent accidentally committing files source control generated. includes artefacts, shared resources dependencies (within report directory) global level contents .outpack directory, draft folder archive directory.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_gitignore_update.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Update a gitignore file — orderly_gitignore_update","text":"","code":"orderly_gitignore_update(name, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_gitignore_update.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Update a gitignore file — orderly_gitignore_update","text":"name name gitignore file update, string \"(root)\" root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, just outpack (see orderly_init details). locate Logical, indicating configuration searched .  TRUE config given, orderly looks working directory parents finds .outpack directory","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_gitignore_update.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Update a gitignore file — orderly_gitignore_update","text":"Nothing, called side effects","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_gitignore_update.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Update a gitignore file — orderly_gitignore_update","text":"function fails message edit '.gitignore', markers corrupted, look special markers within .gitignore file.  look like   edit file : lines appears file anything first two lines order get error message, search remove lines rerun.","code":"# ---VVV--- added by orderly ---VVV---------------- # Don't manually edit content between these markers ... patterns # ---^^^--- added by orderly ---^^^----------------"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_hash.html","id":null,"dir":"Reference","previous_headings":"","what":"Compute a hash — orderly_hash_file","title":"Compute a hash — orderly_hash_file","text":"Use orderly2's hashing functions.  intended advanced users, particular want create hashes consistent orderly2 within plugins. default behaviour use algorithm used orderly root (via root argument, usual root location approach). However, string provided algorithm can use alternative algorithm.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_hash.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Compute a hash — orderly_hash_file","text":"","code":"orderly_hash_file(path, algorithm = NULL, root = NULL, locate = TRUE)  orderly_hash_data(data, algorithm = NULL, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_hash.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Compute a hash — orderly_hash_file","text":"path name file hash algorithm name algorithm use, overriding orderly root. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml data string hash","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_hash.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Compute a hash — orderly_hash_file","text":"string format <algorithm>:<digest>","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_hash.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Compute a hash — orderly_hash_file","text":"","code":"orderly2::orderly_hash_data(\"hello\", \"md5\") #> [1] \"md5:5d41402abc4b2a76b9719d911017c592\""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":null,"dir":"Reference","previous_headings":"","what":"Initialise an orderly repository — orderly_init","title":"Initialise an orderly repository — orderly_init","text":"Initialise empty orderly repository, initialise source copy orderly repository (see Details). orderly repository defined presence file orderly_config.yml root, along directory .outpack/ level.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Initialise an orderly repository — orderly_init","text":"","code":"orderly_init(   root = \".\",   path_archive = \"archive\",   use_file_store = FALSE,   require_complete_tree = FALSE,   force = FALSE )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Initialise an orderly repository — orderly_init","text":"root path initialise repository root .  repository already initialised, operation checks options passed set repository (erroring ), otherwise nothing.  default path current working directory. path_archive Path archive directory, used store human-readable copies packets.  NULL, copy made, file_store must TRUE use_file_store Logical, indicating use content-addressable file-store source truth packets.  archive non-NULL, file-store used source truth duplicated files archive exist convenience. require_complete_tree Logical, indicating require complete tree packets.  currently affects orderly_location_pull_packet, requiring always operates recursive mode.  FALSE default, set TRUE want archive behave well location; TRUE always packets hold metadata . force Logical, indicating shold initialise orderly even directory empty.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Initialise an orderly repository — orderly_init","text":"full, normalised, path root, invisibly. Typically called side effect.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Initialise an orderly repository — orderly_init","text":"expected orderly_config.yml saved version control, .outpack excluded version control; means every clone project need call orderly2::orderly_init() initialise .outpack directory. forget , error thrown reminding need . can safely call orderly2::orderly_init() already-initialised directory, however, arguments passed must exactly match configuration current root, otherwise error thrown. Please use orderly_config_set change configuration, ensures change configuration possible. configuration options given match directory already uses, nothing happens. repository call orderly2::orderly_init() already initialised .outpack directory orderly_config.yml file, write file .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Initialise an orderly repository — orderly_init","text":"","code":"# We'll use an automatically cleaned-up directory for the root: path <- withr::local_tempdir()  # Initialise a new repository, setting an option: orderly2::orderly_init(path, use_file_store = TRUE) #> ✔ Created orderly root at '/tmp/RtmpfBuUfo/file174c46e6b11f'  # fs::dir_tree(path, all = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_interactive_set_search_options.html","id":null,"dir":"Reference","previous_headings":"","what":"Set search options for interactive use — orderly_interactive_set_search_options","title":"Set search options for interactive use — orderly_interactive_set_search_options","text":"Set search options interactive use orderly; see orderly_dependency orderly_run details. may either orderly_search_options object, list coerced one point use (NULL). applies current session, applies interactive uses orderly functions might received copy search_options via orderly_run","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_interactive_set_search_options.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Set search options for interactive use — orderly_interactive_set_search_options","text":"","code":"orderly_interactive_set_search_options(options = NULL)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_interactive_set_search_options.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Set search options for interactive use — orderly_interactive_set_search_options","text":"options Optional control locations, used orderly_dependency; see Details section orderly_run.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_interactive_set_search_options.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Set search options for interactive use — orderly_interactive_set_search_options","text":"Nothing, called side effects","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_list_src.html","id":null,"dir":"Reference","previous_headings":"","what":"List source reports — orderly_list_src","title":"List source reports — orderly_list_src","text":"List source reports - , directories within src/ look suitable running orderly; directories contain entrypoint file - .R file name directory (e.g., src/data/data.R corresponds data).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_list_src.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"List source reports — orderly_list_src","text":"","code":"orderly_list_src(root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_list_src.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"List source reports — orderly_list_src","text":"root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, just outpack (see orderly_init details). locate Logical, indicating configuration searched .  TRUE config given, orderly looks working directory parents finds .outpack directory","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_list_src.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"List source reports — orderly_list_src","text":"character vector names source reports, suitable passing orderly_run","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_list_src.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"List source reports — orderly_list_src","text":"","code":"path <- orderly2::orderly_example(\"default\") #> ✔ Created orderly root at '/tmp/RtmpfBuUfo/file174c42072cbd' orderly2::orderly_list_src(root = path) #> Error: Directory does not exist: '/tmp/RtmpfBuUfo/file174c42072cbd'"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":null,"dir":"Reference","previous_headings":"","what":"Add a new location — orderly_location_add","title":"Add a new location — orderly_location_add","text":"Add new location - place packets might found pulled local archive.  Currently file http based locations supported, limited support custom locations. Note adding location pull metadata , need call orderly_location_pull_metadata first.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Add a new location — orderly_location_add","text":"","code":"orderly_location_add(name, type, args, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Add a new location — orderly_location_add","text":"name short name location use.  use, one local orphan type type location add. Currently supported values path (location exists elsewhere filesystem) http (location accessed outpack's http API). args Arguments location driver. arguments vary depending type used, see Details. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Add a new location — orderly_location_add","text":"Nothing","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Add a new location — orderly_location_add","text":"currently support two types locations - path, points outpack archive accessible path (e.g., computer mounted network share), http, requires outpack server running url uses HTTP API communicate, packit, uses Packit web server.  types may added later, configuration options location types definitely needed future. Configuration options different location types: Path locations: path: path archive root. generally absolute path, behaviour outpack unreliable. HTTP locations: Accessing outpack HTTP requires outpack server running. interface expected change expand API, also move support things like TLS authentication. url: location server, including protocol, example http://example.com:8080 Packit locations: Packit locations work HTTPS, include everything outpack location also provide authentication later capabilities think. url: location server token: value login token (currently GitHub token read:org scope).  Later expand authentication modes supported. Custom locations: outpack implementations expected support path http locations, standard arguments .  expect implementations support custom locations, argument lists may vary implementations. allow , can pass location type \"custom\" list arguments.  expect argument 'driver' present among list.  example action, see outpack.sharepoint package. warned may change interface future, case may need update configuration.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":"warning","dir":"Reference","previous_headings":"","what":"Warning","title":"Add a new location — orderly_location_add","text":"API may change move support different types locations.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_list.html","id":null,"dir":"Reference","previous_headings":"","what":"List known pack locations — orderly_location_list","title":"List known pack locations — orderly_location_list","text":"List known locations.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_list.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"List known pack locations — orderly_location_list","text":"","code":"orderly_location_list(root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_list.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"List known pack locations — orderly_location_list","text":"root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_list.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"List known pack locations — orderly_location_list","text":"character vector location names. special name local always present.","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_metadata.html","id":null,"dir":"Reference","previous_headings":"","what":"Pull metadata from a location — orderly_location_pull_metadata","title":"Pull metadata from a location — orderly_location_pull_metadata","text":"Pull metadata location, updating index.  always relatively quick updates small files contain information can found remote packets.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_metadata.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Pull metadata from a location — orderly_location_pull_metadata","text":"","code":"orderly_location_pull_metadata(location = NULL, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_metadata.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Pull metadata from a location — orderly_location_pull_metadata","text":"location name location pull (see orderly_location_list possible values).  given, pulls locations.  \"local\" \"orphan\" locations always date pulling metadata nothing. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_metadata.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Pull metadata from a location — orderly_location_pull_metadata","text":"Nothing","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_packet.html","id":null,"dir":"Reference","previous_headings":"","what":"Pull one or more packets from a location — orderly_location_pull_packet","title":"Pull one or more packets from a location — orderly_location_pull_packet","text":"Pull one packets (including files) archive one locations. make files available use dependencies (e.g., orderly_dependency).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_packet.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Pull one or more packets from a location — orderly_location_pull_packet","text":"","code":"orderly_location_pull_packet(   ...,   options = NULL,   recursive = NULL,   root = NULL,   locate = TRUE )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_packet.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Pull one or more packets from a location — orderly_location_pull_packet","text":"... Arguments passed orderly_search. special case first argument character vector ids named dot arguments, interpret argument vector ids directly. careful , query may pull lot data particular, passing NULL match everything every remote ! options Options passed orderly_search. option allow_remote must TRUE otherwise packet possibly pulled, error thrown FALSE. recursive non-NULL, logical, indicating recursively pull packets referenced packets specified id.  might copy lot data!  NULL, default value given configuration option require_complete_tree. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_packet.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Pull one or more packets from a location — orderly_location_pull_packet","text":"Invisibly, ids packets pulled","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_packet.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Pull one or more packets from a location — orderly_location_pull_packet","text":"possible take long time pull packets, moving lot data operating slow connection.  Cancelling resuming pull fairly efficient, keep track files copied even case interrupted pull.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_push.html","id":null,"dir":"Reference","previous_headings":"","what":"Push tree to location — orderly_location_push","title":"Push tree to location — orderly_location_push","text":"Push tree location. function works packets known location files required create . pushes files required build packets pushes missing metadata server. process interrupted safe resume transfer files packets missed previous call.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_push.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Push tree to location — orderly_location_push","text":"","code":"orderly_location_push(packet_id, location, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_push.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Push tree to location — orderly_location_push","text":"packet_id One packets push server location name location push (see orderly_location_list possible values). root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_push.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Push tree to location — orderly_location_push","text":"Invisibly, details information actually moved (might less requested, depending dependencies packets already known location).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_remove.html","id":null,"dir":"Reference","previous_headings":"","what":"Remove a location — orderly_location_remove","title":"Remove a location — orderly_location_remove","text":"Remove existing location. packets location known elsewhere now associated 'orphan' location instead.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_remove.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Remove a location — orderly_location_remove","text":"","code":"orderly_location_remove(name, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_remove.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Remove a location — orderly_location_remove","text":"name short name location. remove local orphan root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_remove.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Remove a location — orderly_location_remove","text":"Nothing","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_rename.html","id":null,"dir":"Reference","previous_headings":"","what":"Rename a location — orderly_location_rename","title":"Rename a location — orderly_location_rename","text":"Rename existing location","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_rename.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Rename a location — orderly_location_rename","text":"","code":"orderly_location_rename(old, new, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_rename.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Rename a location — orderly_location_rename","text":"old current short name location. rename local orphan new desired short name location. one local orphan root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_rename.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Rename a location — orderly_location_rename","text":"Nothing","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata.html","id":null,"dir":"Reference","previous_headings":"","what":"Read outpack metadata — orderly_metadata","title":"Read outpack metadata — orderly_metadata","text":"Read metadata particular id. may want use orderly_search find id corresponding particular query.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Read outpack metadata — orderly_metadata","text":"","code":"orderly_metadata(id, root = NULL, locate = FALSE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Read outpack metadata — orderly_metadata","text":"id id fetch metadata . error thrown id known root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Read outpack metadata — orderly_metadata","text":"list metadata. See outpack schema details (https://github.com/mrc-ide/outpack)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":null,"dir":"Reference","previous_headings":"","what":"Extract metadata from orderly2 packets — orderly_metadata_extract","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"Extract metadata group packets.  experimental high-level function interacting metadata way hope useful. expand bit time goes , based feedback get let us know think.  See Details use .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"","code":"orderly_metadata_extract(..., extract = NULL, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"... Arguments passed orderly_search. special case first argument character vector ids named dot arguments, interpret argument vector ids directly. extract character vector columns extract, possibly named. See Details format. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"data.frame, columns vary based names extract; see Details information.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"Extracting data outpack metadata challenging way works data structures familiar R users, naturally tree structured, metadata may present packets (e.g., packet depend another dependency section, one run context without git git metadata). just want raw tree-structured data, can always use orderly_metadata load full metadata packet (even one currently available computer, just known ) structure data remain fairly constant across orderly2 versions. However, sometimes want extract data order ask specific questions like: parameter combinations available across range packets? particular set packets used? files packets produce? Later like ask even complex questions like: version file graph.png change? inputs changed versions? ...able answer questions requires similar approach interrogating metadata across range packets. orderly_metadata_extract function aims simplify process pulling bits metadata arranging data.frame (sorts) .  little mini-language extract argument simple rewriting results, can always . order use function need know metadata available; expand vignette worked examples make easier understand. function works top-level keys, : id: packet id (always returned) name: packet name parameters: key-value pair values, string keys atomic values. guarantee presence keys packets, types. time: key-value pair times, string keys time values (see DateTimeClasses; stored seconds since 1970 actual metadata). present start end always present. files: files present packet. data.frame (per packet), columns path (relative), size (bytes) hash. depends: dependencies used packet. data.frame (per packet), columns packet (id), query (string, used find packet) files (another data.frame columns corresponding filenames upstream packet, respectively) git: either metadata state git null. given sha branch strings, url array strings/character vector (can zero, one elements). session: information session packet run (unstandardised, even orderly version may change) custom: additional metadata added respective engine.  packets run orderly2, orderly field , list: artefacts: data.frame artefact information, containing columns description (string) paths (list column paths). shared: data.frame copied shared resources original name () name copied packet (). role: data.frame identified roles files, columns path role. description: list information orderly_description human-readable descriptions tags. session: list information session run, list platform containing information platform (R version version, operating system os system name system) packages containing columns package , version attached. nesting makes providing universally useful data format difficult; considering files data.frame files column, list data.frames; similar nestedness applies depends orderly custom data. However, able fairly easily process data format need . simplest extraction uses names top-level keys:   creates data.frame columns corresponding keys, one row per packet. name always string, character vector, parameters files complex, list columns. must provide id; always returned always first character vector column. can rename columns providing name entries within extract, example:   , except parameters column renamed pars. interestingly, can index structure like parameters; suppose want value parameter x, write:   allowed packet parameters element list. However, know type x (might vary packets). can add information though write:   create numeric column. packet value x non-integer, call orderly_metadata_extract fail error, packet lacks value x, missing value appropriate type added. Note coercion number, error non-NULL non-numeric value found.  Valid types use <type> boolean, number string (note differ slightly R's names want emphasise scalar quantities; also note integer may produce unexpected errors integer-like numeric values). can also use list default.  Things schema known scalar atomics (name) automatically simplified. can index array-valued elements (files depends) way object-valued elements:   get list column file names per packet another hashes, probably less useful data.frame get extracting just files longer hash information aligned. can index fairly deeply; possible get orderly \"display name\" :   path need extract dot (likely package name plugin, custom.orderly.db) need escape dot backslash (, custom.orderly\\.db). probably need two slashes use raw string (recent versions R).","code":"extract = c(\"name\", \"parameters\", \"files\") extract = c(\"name\", pars = \"parameters\", \"files\") extract = c(x = \"parameters.x\") extract = c(x = \"parameters.x is number\") extract = c(file_path = \"files.path\", file_hash = \"files.hash\") extract = c(display = \"custom.orderly.description.display is string\")"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":"custom-orderly-metadata","dir":"Reference","previous_headings":"","what":"Custom 'orderly' metadata","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"Within custom.orderly, additional fields can extracted. format subject change, stored metadata schema (short term) way deserialise . probably best rely right now, expand section can.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_read.html","id":null,"dir":"Reference","previous_headings":"","what":"Read outpack metadata json file — orderly_metadata_read","title":"Read outpack metadata json file — orderly_metadata_read","text":"Low-level function reading metadata deserialising . function can used directly read metadata json file without reference root contains . may useful context reading metadata file written part failed run.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_read.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Read outpack metadata json file — orderly_metadata_read","text":"","code":"orderly_metadata_read(path, plugins = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_read.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Read outpack metadata json file — orderly_metadata_read","text":"path Path json file plugins Try deserialise data loaded plugins (see Details).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_read.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Read outpack metadata json file — orderly_metadata_read","text":"list outpack metadata; see schema details. contrast reading json file directly jsonlite::fromJSON, function take care convert scalar length-one vectors expected types.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_read.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Read outpack metadata json file — orderly_metadata_read","text":"Custom metadata saved plugins may deserialised expected called function, designed operate separately valid orderly root (.e., load data file regardless came ). plugins TRUE (default) deserialise data matches loaded plugin.  means behaviour function depends loaded plugin packages. can force running orderly2::orderly_config() within orderly directory, load declared plugins.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_new.html","id":null,"dir":"Reference","previous_headings":"","what":"Create a new report — orderly_new","title":"Create a new report — orderly_new","text":"Create new empty report.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_new.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Create a new report — orderly_new","text":"","code":"orderly_new(name, template = NULL, force = FALSE, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_new.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Create a new report — orderly_new","text":"name name report template template use. acceptable values now NULL (uses built-default) FALSE suppresses default content.  may support customisable templates future - let us know useful. force Create orderly file - <name>.R within existing directory src/<name>; may useful already created directory files first want help creating orderly file. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, just outpack (see orderly_init details). locate Logical, indicating configuration searched .  TRUE config given, orderly looks working directory parents finds .outpack directory","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_new.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Create a new report — orderly_new","text":"Nothing, called side effects ","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parameters.html","id":null,"dir":"Reference","previous_headings":"","what":"Declare orderly parameters — orderly_parameters","title":"Declare orderly parameters — orderly_parameters","text":"Declare orderly parameters. one call within file, though enforced! Typically put close top, though order really matter. Parameters scalar atomic values (e.g. string, number boolean) defaults must present literally (.e., may come variable ). Provide NULL default, case parameter required.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parameters.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Declare orderly parameters — orderly_parameters","text":"","code":"orderly_parameters(...)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parameters.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Declare orderly parameters — orderly_parameters","text":"... number parameters","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parameters.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Declare orderly parameters — orderly_parameters","text":"Undefined","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parameters.html","id":"behaviour-in-interactive-sessions","dir":"Reference","previous_headings":"","what":"Behaviour in interactive sessions","title":"Declare orderly parameters — orderly_parameters","text":"running interactively (.e., via source() running orderly file session copy/paste Rstudio), orderly_parameters() function different behaviour. First, look current environment (likely global environment) values parameters - , variables bound names parameters.  parameters found look default values use possible, possible either error prompt based global option orderly_interactive_parameters_missing_error. TRUE, ask enter value parameters (strings need entered quotes).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parse_file.html","id":null,"dir":"Reference","previous_headings":"","what":"Parse the orderly entrypoint script — orderly_parse_file","title":"Parse the orderly entrypoint script — orderly_parse_file","text":"expert use .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parse_file.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Parse the orderly entrypoint script — orderly_parse_file","text":"","code":"orderly_parse_file(path)  orderly_parse_expr(exprs, filename)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parse_file.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Parse the orderly entrypoint script — orderly_parse_file","text":"path Path orderly_* script exprs Parsed AST orderly_* script filename Name orderly_* file include metadata","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parse_file.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Parse the orderly entrypoint script — orderly_parse_file","text":"Parsed orderly entrypoint script","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parse_file.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Parse the orderly entrypoint script — orderly_parse_file","text":"Parses details calls orderly_ -script functions intermediate representation downstream use. Also validates calls orderly_* -script functions well-formed.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_add_metadata.html","id":null,"dir":"Reference","previous_headings":"","what":"Add metadata from plugin — orderly_plugin_add_metadata","title":"Add metadata from plugin — orderly_plugin_add_metadata","text":"Add plugin-specific metadata running packet. take describing. accumulate number bits metadata arbitrary fields, later serialise json.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_add_metadata.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Add metadata from plugin — orderly_plugin_add_metadata","text":"","code":"orderly_plugin_add_metadata(name, field, data)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_add_metadata.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Add metadata from plugin — orderly_plugin_add_metadata","text":"name name plugin; must used orderly_plugin_register orderly_plugin_context field name field add data . required even plugin produces one sort data, case can remove later within serialisation function. data Arbitrary data added currently running packet","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_add_metadata.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Add metadata from plugin — orderly_plugin_add_metadata","text":"Nothing, called side effects","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_context.html","id":null,"dir":"Reference","previous_headings":"","what":"Fetch plugin context — orderly_plugin_context","title":"Fetch plugin context — orderly_plugin_context","text":"Fetch running context, use within plugin.  intention within free functions plugin makes available, call function get information state packet.  typically call orderly_plugin_add_metadata() afterwards.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_context.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Fetch plugin context — orderly_plugin_context","text":"","code":"orderly_plugin_context(name, envir)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_context.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Fetch plugin context — orderly_plugin_context","text":"name Name plugin envir environment calling function. can typically pass parent.frame() (rlang::caller_env()) function calling orderly_plugin_context() function called user. argument effect interactive use (envir almost certainly global environment).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_context.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Fetch plugin context — orderly_plugin_context","text":"list elements: is_active: logical, indicating running orderly_run(); may need change behaviour depending value. path: path running packet. almost always working directory, unless packet contains calls setwd() similar. may create files . config: configuration plugin, processing plugin's read function (see orderly_plugin_register) envir: environment packet running . Often global environment, assume ! may read write environment. src: path packet source directory. different current directory packet running, user interactively working report. may read directory must write parameters: parameters passed run report.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_context.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Fetch plugin context — orderly_plugin_context","text":"plugin function called, orderly2 running one two modes; (1) within orderly_run(), case part way creating packet brand new directory, possibly using special environment evaluation, (2) interactively, user developing report. plugin needs able support modes, function return information state help cope either case.","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_register.html","id":null,"dir":"Reference","previous_headings":"","what":"Register an orderly plugin — orderly_plugin_register","title":"Register an orderly plugin — orderly_plugin_register","text":"Create orderly plugin. plugin typically defined package used extend orderly enabling new functionality, declared orderly_config.yml orderly file, affecting running reports primarily creating new objects report environment.  system discussed detail vignette(\"plugins\"), expanded (likely breaking ways) soon.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_register.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Register an orderly plugin — orderly_plugin_register","text":"","code":"orderly_plugin_register(   name,   config,   serialise = NULL,   deserialise = NULL,   cleanup = NULL,   schema = NULL )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_register.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Register an orderly plugin — orderly_plugin_register","text":"name name plugin, typically package name config function read, check process configuration section orderly_config.yml. function passed deserialised data plugin's section orderly_config.yml, full path file.  order loading plugins defined, plugin must standalone try interact plugins load. return processed copy configuration data, passed second argument read. serialise function serialise metadata added plugin's functions outpack metadata. passed list entries pushed via orderly_plugin_add_metadata(); named list names corresponding field argument orderly_plugin_add_metadata list element unnamed list values corresponding data. NULL, serialisation done, metadata plugin added. deserialise function deserialise metadata serialised serialise function. intended help deal issues disambiguating unserialising objects json (scalars vs arrays lenth 1, data.frames vs lists--lists etc), make plugin nicer work orderly_metadata_extract(). function given single argument data data jsonlite::fromJSON(..., simplifyVector = FALSE) apply required simplifications , returning modified copy argument. cleanup Optionally, function clean state plugin uses. can call orderly_plugin_context within function access anything need . given, cleanup done. schema Optionally path, within package, schema metadata created plugin; omit .json extension.  file contains sources file inst/plugin/myschema.json pass plugin/myschema.  See vignette(\"plugins\") details.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_register.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Register an orderly plugin — orderly_plugin_register","text":"Nothing, function called side effect registering plugin.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_prune_orphans.html","id":null,"dir":"Reference","previous_headings":"","what":"Prune orphan packet metadata — orderly_prune_orphans","title":"Prune orphan packet metadata — orderly_prune_orphans","text":"Prune orphan packets metadata store.  function can used remove references packets longer reachable; happened deleted packet manually archive ran orderly_validate_archive removed location.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_prune_orphans.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Prune orphan packet metadata — orderly_prune_orphans","text":"","code":"orderly_prune_orphans(root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_prune_orphans.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Prune orphan packet metadata — orderly_prune_orphans","text":"root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_prune_orphans.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Prune orphan packet metadata — orderly_prune_orphans","text":"Invisibly, character vector orphaned packet ids","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_prune_orphans.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Prune orphan packet metadata — orderly_prune_orphans","text":"orphan packet used anywhere, can easily drop - never existed.  referenced metadata know elsewhere locally, problem upstream location (one happen). referenced packet run locally, metadata deleted. expose function mostly users want expunge permanently reference previously run packets. hope never need really reason run .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query.html","id":null,"dir":"Reference","previous_headings":"","what":"Construct outpack query — orderly_query","title":"Construct outpack query — orderly_query","text":"Construct outpack query, typically passed orderly_search","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Construct outpack query — orderly_query","text":"","code":"orderly_query(expr, name = NULL, scope = NULL, subquery = NULL)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Construct outpack query — orderly_query","text":"expr query expression. NULL expression matches everything. name Optionally, name packet scope query . intersected scope arg shorthand way running scope = list(name = \"name\") scope Optionally, scope query limit packets searched pars subquery Optionally, named list subqueries can referenced name expr.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Construct outpack query — orderly_query","text":"orderly_query object, modified, can passed orderly_search()","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query_explain.html","id":null,"dir":"Reference","previous_headings":"","what":"Explain a query — orderly_query_explain","title":"Explain a query — orderly_query_explain","text":"Explain query matched. experimental output change. moment, can tell query matches, fails match based one number &&-ed together clauses.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query_explain.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Explain a query — orderly_query_explain","text":"","code":"orderly_query_explain(   ...,   parameters = NULL,   envir = parent.frame(),   options = NULL,   root = NULL )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query_explain.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Explain a query — orderly_query_explain","text":"... Arguments passed orderly_query, perhaps just query expression parameters Optionally, named list parameters substitute query (using : prefix) envir Optionally, environment substitute query (using environment: prefix). default use calling environment, can explicitly pass want control lookup happens. options Optionally, orderly_search_options object controlling search performed, packets considered scope. provided, default options used (.e., orderly2::orderly_search_options()) root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query_explain.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Explain a query — orderly_query_explain","text":"object class orderly_query_explain, can inspected (contents subject change) print method show user-friendly summary query result.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_resource.html","id":null,"dir":"Reference","previous_headings":"","what":"Declare orderly resources — orderly_resource","title":"Declare orderly resources — orderly_resource","text":"Declare file, group files, orderly resource. explicitly declaring files resources orderly mark files immutable inputs validate analysis modify run orderly_run()","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_resource.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Declare orderly resources — orderly_resource","text":"","code":"orderly_resource(files)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_resource.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Declare orderly resources — orderly_resource","text":"files number names files","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_resource.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Declare orderly resources — orderly_resource","text":"Invisibly, character vector resources included call. rely order files expanded directories, likely platform dependent.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":null,"dir":"Reference","previous_headings":"","what":"Run a report — orderly_run","title":"Run a report — orderly_run","text":"Run report.  create new directory drafts/<reportname>, copy declared resources , run script check expected artefacts created.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Run a report — orderly_run","text":"","code":"orderly_run(   name,   parameters = NULL,   envir = NULL,   echo = TRUE,   search_options = NULL,   root = NULL,   locate = TRUE )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Run a report — orderly_run","text":"name Name report run. leading ./ src/ trailing / path parts removed (e.g., added autocomplete). parameters Parameters passed report. named list parameters declared orderly.yml.  parameter must scalar character, numeric, integer logical. envir environment used evaluate report script; default use global environment, may always wanted. echo Optional logical control printing output source() console. search_options Optional control locations, used orderly_dependency; converted orderly_search_options object, see Details. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, just outpack (see orderly_init details). locate Logical, indicating configuration searched .  TRUE config given, orderly looks working directory parents finds .outpack directory","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Run a report — orderly_run","text":"id created report (string)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"locations-used-in-dependency-resolution","dir":"Reference","previous_headings":"","what":"Locations used in dependency resolution","title":"Run a report — orderly_run","text":"packet depends packets, want control locations used find appropriate packets. control passed function argument orderly_dependency property way packet created packet ; importantly different users may different names locations makes little sense encode location name source code. Alternatively, want use different locations different contexts (initial development want include local copies packets possible dependencies vs resolving dependencies resolved one locations! Similarly, might want include packets known locations currently downloaded onto machine - pulling packets take anything seconds hours depending size speed network connection (pulling packets mean packet fails run). allow control can pass argument search_options, orderly_search_options object, allows control names locations use, whether metadata refreshed pull anything packets currently downloaded considered candidates. effect running interactively, case can specify search options (root specific) orderly_interactive_set_search_options","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"which-packets-might-be-selected-from-locations-","dir":"Reference","previous_headings":"","what":"Which packets might be selected from locations?","title":"Run a report — orderly_run","text":"search_options argument controls outpack searches packets given query anything might moved network (one outpack archive another). default everything resolved locally ; can depend packets unpacked within current archive.  pass search_options argument contains allow_remote = TRUE (see orderly_search_options packets known anywhere candidates using dependencies needed pull resolved files remote location. Note even packet locally present might needed - content anywhere else unpacked packet reuse content without re-fetching. pull_metadata = TRUE, refresh location metadata pulling, location argument controls locations pulled .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"equivalence-to-the-old-use-draft-option","dir":"Reference","previous_headings":"","what":"Equivalence to the old use_draft option","title":"Run a report — orderly_run","text":"location handling generalises orderly (v1)'s old use_draft option, terms location argument orderly2::orderly_search_options`: use_draft = TRUE location = \"local\" use_draft = FALSE location = c(...) provide locations except local (setdiff(orderly2::orderly_location_list(), \"local\")) use_draft = \"newer\" location = NULL (last option one people preferred new default behaviour). addition, resolve dependencies currently exist production right now options:   updates current metadata production, runs queries packets known remote, depends even (yet) locally.  functionality never available orderly version 1, though intended support .","code":"location = \"production\", pull_metadata = TRUE, require_unpacked = FALSE"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"running-with-a-source-tree-separate-from-outpack-root","dir":"Reference","previous_headings":"","what":"Running with a source tree separate from outpack root","title":"Run a report — orderly_run","text":"Sometimes useful run things different place disk outpack root. know two cases come : running reports within runner server, make clean clone source tree particular git reference new temporary directory run report , insert orderly repo fixed non-temporary location. user convenient torun report hard drive store archive metadata (larger) shared drive. first instance, source path <src> contains file orderly_config.yml directory src/ source reports, separate path <root> contains directory .outpack/ metadata - may also unpacked archive, .git/ directory depending configuration. (Later make sense support \"bare\" outpack layout.)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"manually-setting-report-source-directory","dir":"Reference","previous_headings":"","what":"Manually setting report source directory","title":"Run a report — orderly_run","text":"manually set report source directory, need set path directory ORDERLY_REPORT_SRC environment variable.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Run a report — orderly_run","text":"","code":"# Create a simple example: path <- orderly2::orderly_example(\"default\") #> ✔ Created orderly root at '/tmp/RtmpfBuUfo/file174c1399bc00'  # Run the 'data' task: orderly2::orderly_run(\"data\", root = path) #> Error: Directory does not exist: '/tmp/RtmpfBuUfo/file174c1399bc00'  # After running, a finished packet appears in the archive: fs::dir_tree(path) #> Error: [ENOENT] Failed to search directory '/tmp/RtmpfBuUfo/file174c1399bc00': no such file or directory  # and we can query the metadata: orderly2::orderly_metadata_extract(name = \"data\", root = path) #> Error: Directory does not exist: '/tmp/RtmpfBuUfo/file174c1399bc00'"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run_info.html","id":null,"dir":"Reference","previous_headings":"","what":"Information about currently running report — orderly_run_info","title":"Information about currently running report — orderly_run_info","text":"Fetch information actively running report.  allows reflect information report back part report, example embedding current report id, information computed dependencies. information slightly different format orderly version 1.x (currently) include information dependencies run outside orderly_run, never reliable previously.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run_info.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Information about currently running report — orderly_run_info","text":"","code":"orderly_run_info()"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run_info.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Information about currently running report — orderly_run_info","text":"list elements name: name current report id: id current report, NA running interactively root: orderly root path depends: data frame information dependencies (available interactively) index: integer sequence along calls orderly_dependency name: name dependency query: query used find dependency id: computed id included packet filename: file used packet : filename used locally","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search.html","id":null,"dir":"Reference","previous_headings":"","what":"Query outpack's database — orderly_search","title":"Query outpack's database — orderly_search","text":"Evaluate query outpack database, returning vector matching packet ids.  Note default searches packets unpacked available direct use computer; search within packets known locations (might know via metadata) need use options argument.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Query outpack's database — orderly_search","text":"","code":"orderly_search(   ...,   parameters = NULL,   envir = parent.frame(),   options = NULL,   root = NULL )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Query outpack's database — orderly_search","text":"... Arguments passed orderly_query, perhaps just query expression parameters Optionally, named list parameters substitute query (using : prefix) envir Optionally, environment substitute query (using environment: prefix). default use calling environment, can explicitly pass want control lookup happens. options Optionally, orderly_search_options object controlling search performed, packets considered scope. provided, default options used (.e., orderly2::orderly_search_options()) root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Query outpack's database — orderly_search","text":"character vector matching ids. case match query returning single value (e.g., latest(...) single(...)) character missing value (NA_character_)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search_options.html","id":null,"dir":"Reference","previous_headings":"","what":"Packet search options — orderly_search_options","title":"Packet search options — orderly_search_options","text":"Options controlling packet searches carried , example via orderly_search orderly_run. details never included metadata alongside query (, part query even though affect ).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search_options.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Packet search options — orderly_search_options","text":"","code":"orderly_search_options(   location = NULL,   allow_remote = FALSE,   pull_metadata = FALSE )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search_options.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Packet search options — orderly_search_options","text":"location Optional vector locations pull . might future expand allow wildcards exceptions. allow_remote Logical, indicating allow packets found currently unpacked (.e., known location metadata ). TRUE, conjunction orderly_dependency might pull large quantity data. pull_metadata Logical, indicating pull metadata immediately search. location given, pass orderly_location_pull_metadata filter locations update.  pulling many packets sequence, want update option FALSE first pull.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search_options.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Packet search options — orderly_search_options","text":"object class orderly_search_options modified creation (see note pull_metadata)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_shared_resource.html","id":null,"dir":"Reference","previous_headings":"","what":"Copy shared resources into a packet directory — orderly_shared_resource","title":"Copy shared resources into a packet directory — orderly_shared_resource","text":"Copy shared resources packet directory. can use share common resources (data code) multiple packets. Additional metadata added keep track files came .  Using function requires shared resources directory shared/ exists orderly root; error raised configured attempt fetch files.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_shared_resource.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Copy shared resources into a packet directory — orderly_shared_resource","text":"","code":"orderly_shared_resource(...)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_shared_resource.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Copy shared resources into a packet directory — orderly_shared_resource","text":"... shared resources copy. arguments named, name destination file value filename within shared resource directory. can use limited form string interpolation names argument; using ${variable} pick values envir substitute string.  similar interpolation might familiar glue::glue similar, much simpler concatenation fancy features supported.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_shared_resource.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Copy shared resources into a packet directory — orderly_shared_resource","text":"Invisibly, data.frame columns (fileames copied running packet) (filenames within shared/).  orderly_resource, rely ordering directory expansion performed.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_strict_mode.html","id":null,"dir":"Reference","previous_headings":"","what":"Enable orderly strict mode — orderly_strict_mode","title":"Enable orderly strict mode — orderly_strict_mode","text":"Put orderly2 \"strict mode\", closer defaults orderly 1.0.0; mode explicitly included files (via orderly_resource orderly_shared_resource) copied running packet, warn unexpected files end run.  Using strict mode allows orderly2 aggressive deletes files within source directory, accurate reports , faster start packets developing interactively.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_strict_mode.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Enable orderly strict mode — orderly_strict_mode","text":"","code":"orderly_strict_mode()"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_strict_mode.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Enable orderly strict mode — orderly_strict_mode","text":"Undefined","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_strict_mode.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Enable orderly strict mode — orderly_strict_mode","text":"future, may extend strict mode allow requiring computation occurs within orderly functions (.e., requirements run packet fully known actually running ).  likely default behaviour orderly_strict_mode gain argument. allow server processes either override value (enabling even explicitly given) /require .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_validate_archive.html","id":null,"dir":"Reference","previous_headings":"","what":"Validate unpacked packets. — orderly_validate_archive","title":"Validate unpacked packets. — orderly_validate_archive","text":"Validate unpacked packets. time, expect function become fully featured, validating .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_validate_archive.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Validate unpacked packets. — orderly_validate_archive","text":"","code":"orderly_validate_archive(..., action = \"inform\", root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_validate_archive.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Validate unpacked packets. — orderly_validate_archive","text":"... Either arguments search can constructed (useful options include name = \"something\"), character vector ids, leave empty validate everything. action action take finding invalid packet. See Details. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_validate_archive.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Validate unpacked packets. — orderly_validate_archive","text":"Invisibly, character vector repaired (invalid) packets.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_validate_archive.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Validate unpacked packets. — orderly_validate_archive","text":"actions can take finding invalid packet : inform (default): just print information problem orphan: mark packet orphaned within metadata, touch files archive (default directory archive/) - safe option leave consistent state without deleting anything. delete: addition marking packet orphan, also delete files archive. Later, add \"repair\" option try fix broken packets. validation interacts option core.require_complete_tree; option TRUE, packet valid (recursive) dependencies also valid, action apply packets also upstream dependencies invalidated.  validation happen even query implied ... include packets complete tree required. validation also interact core.use_file_store repair supported, becomes trivial.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":null,"dir":"Reference","previous_headings":"","what":"Query index — query_index","title":"Query index — query_index","text":"Class managing active index whilst evaluating query","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"public-fields","dir":"Reference","previous_headings":"","what":"Public fields","title":"Query index — query_index","text":"index packet index depends Named list data frames. Names packet ids, values packets depended packet id (.e. parents). uses Named list data frames. Names packet ids, values packets used packet id (.e. children). root outpack root object","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"public-methods","dir":"Reference","previous_headings":"","what":"Public methods","title":"Query index — query_index","text":"query_index$new() query_index$get_packet_depends() query_index$get_packet_uses()","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"method-new-","dir":"Reference","previous_headings":"","what":"Method new()","title":"Query index — query_index","text":"Create new query_index object","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Query index — query_index","text":"","code":"query_index$new(root, index, depends, uses)"},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Query index — query_index","text":"root outpack root object index packet index data frame depends Named list data frames. Names packet ids, values packets depended packet id (.e. parents). uses Named list data frames. Names packet ids, values packets used packet id (.e. children). data depends relationships flow direction.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"method-get-packet-depends-","dir":"Reference","previous_headings":"","what":"Method get_packet_depends()","title":"Query index — query_index","text":"Get ids packets packet depends specified level","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"usage-1","dir":"Reference","previous_headings":"","what":"Usage","title":"Query index — query_index","text":"","code":"query_index$get_packet_depends(id, depth)"},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"arguments-1","dir":"Reference","previous_headings":"","what":"Arguments","title":"Query index — query_index","text":"id id packet get parents depth Depth parents get, depth 1 gets immediate parents depth 2 gets parents parents parents, depth Inf recurse whole tree get parents","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"returns","dir":"Reference","previous_headings":"","what":"Returns","title":"Query index — query_index","text":"ids parents packet","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"method-get-packet-uses-","dir":"Reference","previous_headings":"","what":"Method get_packet_uses()","title":"Query index — query_index","text":"Get ids packets used packet","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"usage-2","dir":"Reference","previous_headings":"","what":"Usage","title":"Query index — query_index","text":"","code":"query_index$get_packet_uses(id, depth)"},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"arguments-2","dir":"Reference","previous_headings":"","what":"Arguments","title":"Query index — query_index","text":"id id packet get children depth Depth children get, depth 1 gets immediate children depth 2 gets children children children, depth Inf recurse whole tree get children","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"returns-1","dir":"Reference","previous_headings":"","what":"Returns","title":"Query index — query_index","text":"ids children packet","code":""}]
+[{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"type-of-location","dir":"Articles","previous_headings":"","what":"Type of location","title":"Collaborative analysis","text":"orderly2 supports two types locations default: path: orderly2 working copy can find filesystem. Note copy people can see (example network share, cloud-synced file system Dropbox OneDrive) see Sharing packets collaborators using shared file system details http: require running HTTP API, either via outpack_server packit location system somewhat extensible, details subject change. documentation behave regardless location stored mechanism transport, can focus instead workflows.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"an-example","dir":"Articles","previous_headings":"","what":"An example","title":"Collaborative analysis","text":"Suppose two researchers, Alice Bob, collaborating piece analysis. purposes example (find common situation practice) share source tree analysis git. Alice Bob clone repo onto machines, using favourite git client (terminal git, via GitHub desktop, gert) moment, including hidden files (except .git), Alice sees: point, cloning Alice .outpack directory (see ) Alice needs run orderly::orderly_init() first: plan work Bob, sharing results shared server, Alice adds orderly2 configuration orderly2::orderly_location_add(): done, can run analysis: Perhaps takes several goes Alice happy analysis, point something ready share. can “push” final packet onto server: Now, consider Bob. also needs source code cloned, orderly initialised, server added: (, Bob also used name server refer shared server, used anything; similar git’s use origin.) Bob can now query packets available server: seen new “data” packet , can pull locally (TODO: mrc-4414 makes nicer): Now Bob position develop packet Alice ran (20240710-102605-babfcf9e)","code":"fs::dir_tree(\".\", all = TRUE, glob = \".git\", invert = TRUE) ## . ## ├── orderly_config.yml ## └── src ##     └── data ##         └── data.R orderly2::orderly_list_src() ## [1] \"data\" orderly2::orderly_init(\".\") ## ✔ Created orderly root at '/tmp/Rtmp4tLdTj/file1a7d6ec6f83/alice' ## ✔ Wrote '.gitignore' orderly2::orderly_list_src() ## [1] \"data\" orderly2::orderly_location_add(   \"server\", \"http\", list(url = \"http://orderly.example.com\")) id <- orderly2::orderly_run(\"data\") ## ℹ Starting packet 'data' `20240710-102605-babfcf9e` at 2024-07-10 10:26:05.736024 ## > orderly2::orderly_artefact(\"Final data\", \"data.rds\") ## > saveRDS(mtcars, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240710-102605-babfcf9e at 2024-07-10 10:26:05.793755 (0.05773139 secs) orderly2::orderly_location_push(id, \"server\") orderly2::orderly_init(\".\") ## ✔ Created orderly root at '/tmp/Rtmp4tLdTj/file1a7d6ec6f83/bob' ## ✔ Wrote '.gitignore' orderly2::orderly_location_add(   \"server\", \"http\", list(url = \"http://orderly.example.com\")) orderly2::orderly_metadata_extract(   name = \"data\",   options = list(allow_remote = TRUE, pull_metadata = TRUE)) ##                         id name parameters ## 1 20240710-102605-babfcf9e data orderly2::orderly_location_pull_packet(id) ## ℹ Looking for suitable files already on disk ## ℹ Need to fetch 2 files (1.3 kB) from 1 location ## ⠙ Fetching file 1/2 (81 B) from 'server' | ETA:  0s [3ms] ## ✔ Fetched 2 files (1.3 kB) from 'server' in 20ms. ##"},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"possible-working-patterns","dir":"Articles","previous_headings":"","what":"Possible working patterns","title":"Collaborative analysis","text":"seen several broad patterns distributing packets. Central server, push allowed: case ; flexible requires everyone relaxed packets created ultimately trusting colleagues exercising good environment hygiene. Just rules enforced computer though, doesn’t mean might ideas within group can push. may useful push HPC system running computationally demanding tasks . Central server, final copy run : case, Alice Bob can run things server, “canonical” copies, nothing pushed server. Reports can run server ssh-ing onto server running packets manually (TODO: mrc-4412) using web interface (possible OrderlyWeb possible packit). approach works well primary goal confidence packets everyone works dependencies created “clean” environment unexpected global dependencies. web version, can also enforce things like running default branch. Alice Bob end collection packets local archives mix canonical ones (server) locally-created ones, local ones always dead end never get shared anyone. result, Alice Bob may delete archive directories without great concern. running packets, make sure packet run similar way packet run server, Bob may want ensure dependencies found server considered. , can create search options pass orderly2::orderly_run running report. developing, Bob may find passing orderly2::orderly_interactive_set_search_options() helpful mean calls orderly2::orderly_description() use packets server. Staging production servers: might set multiple servers, one staging, one production. staging servers never source truth might (might ) allow people push , less strict rules gets run (perhaps allowing running packets feature branch). setting, users end packets created locally, staging machines production server local archive. staging server probably end mix packets packets production. approach specifying search options useful choosing included final packet.","code":"search_options <- orderly2::orderly_search_options(location = \"server\")"},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"pulling-complete-trees-or-not","dir":"Articles","previous_headings":"","what":"Pulling complete trees, or not","title":"Collaborative analysis","text":"pull packets server (orderly2::orderly_location_pull_packet()), depending settings either end just packet request, packet plus recursive dependencies. controlled configuration option core.require_complete_tree, FALSE default; suitable users, servers set core.require_complete_tree = TRUE. situations , users never holding canonical source packets - matter don’t dependencies. hold metadata packets (, recursively) download dependencies later wanted inspect . Typically though, just extra space used disk. push packet server, always pushes complete tree.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"moving-as-little-data-as-possible","dir":"Articles","previous_headings":"","what":"Moving as little data as possible","title":"Collaborative analysis","text":"pulling packets, first user pulls metadata server. orderly2::orderly_location_metadata_pull run, metadata server updated client (mrc-4444 make lazier). now know everything server holds, hashes files contained packet. likely already know files within packet requested, client first looks see exist within archive, verifying changed. requests server files resolve locally. recursive pull (core.require_complete_tree = TRUE) look union files set packets currently missing archive. frequently much less full set files within packets. algorithm similar push: look full dependency chain target packet find file ids ask server yet know unknown subset get union file hashes ask server subset files yet know push files - done one time present done simultaneously async requests future files sever upload missing metadata, chronological order server verifies packet metadata dependencies referenced files multi-step process means avoid copying data already known , well avoiding moving data multiple times. also means process pushing (pulling) interrupted can safely resumed state always consistent.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"interaction-between-source-tree-and--outpack","dir":"Articles","previous_headings":"","what":"Interaction between source tree and .outpack","title":"Collaborative analysis","text":"important .outpack directory shared via git; warn now, can use orderly2::ordery_gitignore_update() automatically create suitable .gitignore file prevent accidentally committed. However, Alice Bob starting new machine : clone source repository run orderly2::orderly_init() within directory run orderly2 commands normal Without running orderly2::orderly_init(), get error prompting initialise repo. requirement different repositories share configuration options passed orderly2::orderly_init(); can enable disable file store different sets locations enabled, depending workflows. means , practice, association convention set related orderly2 locations source tree can orderly2 repository points locations refer different source trees!","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/collaboration.html","id":"sharing-packets-with-collaborators-using-a-shared-file-system","dir":"Articles","previous_headings":"","what":"Sharing packets with collaborators using a shared file system","title":"Collaborative analysis","text":"One simplest ways share packets collaborator shared file system, example network share, cloud-synced file system Dropbox OneDrive. can follows. Create new folder orderly remote shared file system. Alice synced path_sharepoint_alice. Initialise orderly location shared file system Create orderly store file store complete tree. See orderly2::orderly_init() details. Add location Push packets want share available collaborator pull. Note data pushed file store .outpack directory. .dot files considered hidden operating system, “show hidden files” file browser see pushed packet. collaborator can now pull shared file. , : Sync drive location machine. Bob synced path_sharepoint_bob Add location Pull metadata use packets desired","code":"orderly2::orderly_init(   root = path_sharepoint_alice,    path_archive = NULL,    use_file_store = TRUE,    require_complete_tree = TRUE ) ## ✔ Created orderly root at '/tmp/Rtmp4tLdTj/file1a7d6ec6f83/sharepoint' orderly2::orderly_location_add(   \"sharepoint\", \"path\", list(path = path_sharepoint_alice)) orderly2::orderly_location_push(id, \"sharepoint\") orderly2::orderly_location_add(   \"alices_orderly\", \"path\", list(path = path_sharepoint_bob)) orderly2::orderly_location_pull_metadata(   \"alices_orderly\" )"},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"using-dependencies","dir":"Articles","previous_headings":"","what":"Using dependencies","title":"Dependencies between packets","text":", show practically use dependencies common scenarios increasing complexity. code examples purposefully -simple order keep presentation straightforward, see end document discussion complex pieces code might “optimally” .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"basic-use","dir":"Articles","previous_headings":"Using dependencies","what":"Basic use","title":"Dependencies between packets","text":"primary mechanism using dependencies call orderly2::orderly_dependency() within orderly file; finds suitable completed packet copies files found within packet current report. src/analysis/analysis.R contains: , ’ve used orderly2::orderly_dependency() pull file data.rds recent version (latest()) data packet, ’ve used file normal make plot, ’ve saved analysis.png (similar example vignette(\"introduction\"), get us started). look metadata packet created analysis report, can see used 20240710-102609-362a30fc dependency: (indeed , one copy data packet pick ).","code":"## src ## ├── analysis ## │   └── analysis.R ## └── data ##     ├── data.R ##     └── data.csv orderly2::orderly_dependency(\"data\", \"latest()\", \"data.rds\") d <- readRDS(\"data.rds\") png(\"analysis.png\") plot(y ~ x, d) dev.off() id1 <- orderly2::orderly_run(\"data\") ## ℹ Starting packet 'data' `20240710-102609-362a30fc` at 2024-07-10 10:26:09.219197 ## > d <- read.csv(\"data.csv\") ## > d$z <- resid(lm(y ~ x, d)) ## > saveRDS(d, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240710-102609-362a30fc at 2024-07-10 10:26:09.254927 (0.03572965 secs) id2 <- orderly2::orderly_run(\"analysis\") ## ℹ Starting packet 'analysis' `20240710-102609-483418bb` at 2024-07-10 10:26:09.287808 ## > orderly2::orderly_dependency(\"data\", \"latest()\", \"data.rds\") ## ℹ Depending on data @ `20240710-102609-362a30fc` (via latest(name == \"data\")) ## > d <- readRDS(\"data.rds\") ## > png(\"analysis.png\") ## > plot(y ~ x, d) ## > dev.off() ## agg_png  ##       2 ## ✔ Finished running analysis.R ## ℹ Finished 20240710-102609-483418bb at 2024-07-10 10:26:09.380892 (0.09308434 secs) orderly2::orderly_metadata(id2)$depends ##                     packet                  query        files ## 1 20240710-102609-362a30fc latest(name == \"data\") data.rds...."},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"filtering-candidates-by-parameters","dir":"Articles","previous_headings":"Using dependencies","what":"Filtering candidates by parameters","title":"Dependencies between packets","text":"example, query simple — recently created packet name data. One common pattern see analysis might parameter (example country name) downstream analysis might share parameter want pull data country. src/data/data.R containing: can run several values cyl: follow-analysis contains: query latest(parameter:cyl == :cyl) says “find recent packet ’s parameter”cyl” (parameter:cyl) parameter currently running report (:cyl).","code":"## src ## ├── analysis ## │   └── analysis.R ## └── data ##     └── data.R orderly2::orderly_parameters(cyl = NULL) d <- mtcars[mtcars$cyl == cyl, ] saveRDS(d, \"data.rds\") orderly2::orderly_run(\"data\", list(cyl = 4)) ## ℹ Starting packet 'data' `20240710-102609-c0708a99` at 2024-07-10 10:26:09.754879 ## ℹ Parameters: ## • cyl: 4 ## > orderly2::orderly_parameters(cyl = NULL) ## > d <- mtcars[mtcars$cyl == cyl, ] ## > saveRDS(d, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240710-102609-c0708a99 at 2024-07-10 10:26:09.786579 (0.03170037 secs) ## [1] \"20240710-102609-c0708a99\" orderly2::orderly_run(\"data\", list(cyl = 6)) ## ℹ Starting packet 'data' `20240710-102609-cde98f1c` at 2024-07-10 10:26:09.807431 ## ℹ Parameters: ## • cyl: 6 ## > orderly2::orderly_parameters(cyl = NULL) ## > d <- mtcars[mtcars$cyl == cyl, ] ## > saveRDS(d, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240710-102609-cde98f1c at 2024-07-10 10:26:09.835665 (0.02823305 secs) ## [1] \"20240710-102609-cde98f1c\" orderly2::orderly_run(\"data\", list(cyl = 8)) ## ℹ Starting packet 'data' `20240710-102609-db26b192` at 2024-07-10 10:26:09.859044 ## ℹ Parameters: ## • cyl: 8 ## > orderly2::orderly_parameters(cyl = NULL) ## > d <- mtcars[mtcars$cyl == cyl, ] ## > saveRDS(d, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240710-102609-db26b192 at 2024-07-10 10:26:09.885346 (0.02630186 secs) ## [1] \"20240710-102609-db26b192\" orderly2::orderly_parameters(cyl = NULL) orderly2::orderly_dependency(   \"data\",   \"latest(parameter:cyl == this:cyl)\",   \"data.rds\") d <- readRDS(\"data.rds\") png(\"analysis.png\") plot(mpg ~ disp, d) dev.off() orderly2::orderly_run(\"analysis\", list(cyl = 4)) ## ℹ Starting packet 'analysis' `20240710-102610-0981d7cb` at 2024-07-10 10:26:10.04024 ## ℹ Parameters: ## • cyl: 4 ## > orderly2::orderly_parameters(cyl = NULL) ## > orderly2::orderly_dependency( ## +   \"data\", ## +   \"latest(parameter:cyl == this:cyl)\", ## +   \"data.rds\") ## ℹ Depending on data @ `20240710-102609-c0708a99` (via latest(parameter:cyl == this:cyl && name == \"data\")) ## > d <- readRDS(\"data.rds\") ## > png(\"analysis.png\") ## > plot(mpg ~ disp, d) ## > dev.off() ## agg_png  ##       2 ## ✔ Finished running analysis.R ## ℹ Finished 20240710-102610-0981d7cb at 2024-07-10 10:26:10.092804 (0.05256438 secs) ## [1] \"20240710-102610-0981d7cb\""},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"interpreting-errors","dir":"Articles","previous_headings":"Using dependencies","what":"Interpreting errors","title":"Dependencies between packets","text":"query fails resolve candidate error: error message tries fairly self explanatory; failed find packet satisfies querylatest(parameter:cyl == :cyl && name == \"data\"); note report name data become part query, two conditions matched . error suggests running rlang::last_error()$explanation information, can : orderly_query_explain object, tries come reasons query might matched; ’ll expand future let us know might like see. tells query can decomposed two subqueries (match parameter cyl 9000), matched packets B (match packet name data), matched 3 packets. subquery matched packets pairs don’t try guide towards problematic pairs. can also ask orderly2 explain query : save object can explore detail: (worked rlang::last_error()$explanation$parts$). can also use orderly2::orderly_metadata_extract work values might looked :","code":"orderly2::orderly_run(\"analysis\", list(cyl = 9000)) ## ℹ Starting packet 'analysis' `20240710-102610-2c69ef18` at 2024-07-10 10:26:10.17657 ## ℹ Parameters: ## • cyl: 9000 ## > orderly2::orderly_parameters(cyl = NULL) ## > orderly2::orderly_dependency( ## +   \"data\", ## +   \"latest(parameter:cyl == this:cyl)\", ## +   \"data.rds\") ## ✖ Error running analysis.R ## ℹ Finished 20240710-102610-2c69ef18 at 2024-07-10 10:26:10.238994 (0.06242466 secs) ## Error in `orderly2::orderly_run()`: ## ! Failed to run report ## Caused by error in `outpack_packet_use_dependency()`: ## ! Failed to find packet for query 'latest(parameter:cyl == this:cyl && ##   name == \"data\")' ## ℹ See 'rlang::last_error()$explanation' for details rlang::last_error()$explanation ## Evaluated query: 'latest(A && B)' and found 0 packets ## • A (parameter:cyl == this:cyl): 0 packets ##  ## • B (name == \"data\"): 3 packets orderly2::orderly_query_explain(   quote(latest(parameter:cyl == 9000)), name = \"data\") ## Evaluated query: 'latest(A && B)' and found 0 packets ## • A (parameter:cyl == 9000): 0 packets ##  ## • B (name == \"data\"): 3 packets explanation <- orderly2::orderly_query_explain(   quote(latest(parameter:cyl == 9000)), name = \"data\") explanation$parts$B ## $name ## [1] \"B\" ##  ## $str ## [1] \"name == \\\"data\\\"\" ##  ## $expr ## name == \"data\" ##  ## $n ## [1] 3 ##  ## $found ## [1] \"20240710-102609-c0708a99\" \"20240710-102609-cde98f1c\" ## [3] \"20240710-102609-db26b192\" orderly2::orderly_metadata_extract(   name = \"data\",   extract = c(cyl = \"parameters.cyl is number\")) ##                         id cyl ## 1 20240710-102609-c0708a99   4 ## 2 20240710-102609-cde98f1c   6 ## 3 20240710-102609-db26b192   8"},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"filtering-candidates-in-other-ways","dir":"Articles","previous_headings":"Using dependencies","what":"Filtering candidates in other ways","title":"Dependencies between packets","text":"saw two types filtering candidates: latest() selected recent packet latest(parameter:cyl == :cyl) found packet whose parameter matched one parameters. used latest(parameter:cyl == 4) hard code specific parameter value, used latest(parameter:cyl == environment:cyl) match whatever value cyl took evaluating environment. Instead query, can provide single id (e.g, 20240710-102610-0981d7cb), mean even new copies data packet created, dependency always resolve value. can chain together logical operations && (sides must true) || (either side must true), group conditions parentheses. addition ==, usual complement comparison operators work. might complex queries like practice people queries series restrictions &&.","code":"latest((parameter:x == 1 || parameter:x == 2) && parameter:y > 10)"},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"computing-dependencies-and-using-many-dependencies-at-once","dir":"Articles","previous_headings":"Using dependencies","what":"Computing dependencies and using many dependencies at once","title":"Dependencies between packets","text":"One common pattern map-reduce pattern set orderly reports. , set packets created vector parameters, perhaps chain different reports parameter, combined together. parameter p takes values “x”, “y” “z”, might look like: , D want combine three copies C packet, one p “x”, “y” “z”. Especially three values hard coded, might just write Note call vary second argument select different parameter value, third argument naming destination file different name (end three files data/). can write loop: , second argument use environment:p fetch value p calling environment - looping value take three values. name third argument, use special interpolation format ${p} substitute value p build filename.","code":"B(p = \"x\") -- C(p = \"x\")   /                          \\ A - B(p = \"y\") -- C(p = \"y\") - D   \\                          /     B(p = \"z\") -- C(p = \"z\") orderly2::orderly_dependency(\"C\", quote(latest(parameter:p == \"x\")),                              c(\"data/x.rds\" = \"result.rds\")) orderly2::orderly_dependency(\"C\", quote(latest(parameter:p == \"y\")),                              c(\"data/y.rds\" = \"result.rds\")) orderly2::orderly_dependency(\"C\", quote(latest(parameter:p == \"z\")),                              c(\"data/z.rds\" = \"result.rds\")) for (p in c(\"x\", \"y\", \"z\")) {   orderly2::orderly_dependency(\"C\", quote(latest(parameter:p == environment:p)),                                c(\"data/${p}.rds\" = \"result.rds\")) }"},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"how-dependencies-interact-with-locations","dir":"Articles","previous_headings":"","what":"How dependencies interact with locations","title":"Dependencies between packets","text":"default, packet unpacked local archive considered candidate inclusion orderly_dependency(). always want. locations selected, packets within considered candidates can controlled search_options argument orderly2::orderly_run (note argument orderly_run(), orderly_dependency() effect controlled runner report, writer report). three components affect packets selected location: character vector locations, matching location names. packets can found locations considered. mix locally created packets well ones people can see, specifying location = \"server\" limit packets available server, means end dependencies colleagues also get. allow_remote: controls willing download files location order satisfy dependency. TRUE, run report, might download files recent packets available location locally. pull_metadata: effect allow_remote also TRUE; causes metadata refreshed dependency resolution. discussion details ?orderly_run","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/dependencies.html","id":"other-points","dir":"Articles","previous_headings":"","what":"Other points","title":"Dependencies between packets","text":"used systems like targets, easy make reports smaller need . ’s real need make small, picking right size challenge. small, ’ll end writing lot code orchestrate running different reports pulling things together. ’ll end spending lot time whether things “date” one another really group things always wants run together. ’re big might end work want , order make change part piece analysis must run whole thing .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"installation","dir":"Articles","previous_headings":"","what":"Installation","title":"Introduction to orderly2","text":"","code":"install.packages(   \"orderly2\",   repos = c(\"https://mrc-ide.r-universe.dev\", \"https://cloud.r-project.org\"))"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"creating-an-empty-orderly-repository","dir":"Articles","previous_headings":"","what":"Creating an empty orderly repository","title":"Introduction to orderly2","text":"first step initialise empty orderly2 repository. orderly2 repository directory file orderly_config.yml within , since version 2 also directory .outpack/. Files within .outpack/ directory never directly modified users directory excluded version control (see orderly2::orderly_gitignore_update). Create orderly2 repository calling orderly2::orderly_init(): creates files: step performed completely empty directory, otherwise error thrown. Later, re-initialise orderly2 repository cloning new machine, working others; discussed vignette(\"collaboration\"). orderly_config.yml file contains little default: vignette, created orderly root R’s per-session temporary directory, deleted R exits. want use directory persist across restarting R (certainly want using orderly2 real project!) replace path within home directory, location control. rest vignette evaluate commands within directory, changing directory path ’ve created:","code":"path <- tempfile() # we'll use a temporary directory here - see note below orderly2::orderly_init(path) ## ✔ Created orderly root at '/tmp/Rtmpv6E3fl/file1b083976469a' ## . ## ├── .outpack ## │   ├── config.json ## │   ├── location ## │   └── metadata ## └── orderly_config.yml minimum_orderly_version: \"1.99.0\" setwd(path)"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"creating-your-first-orderly-report","dir":"Articles","previous_headings":"","what":"Creating your first orderly report","title":"Introduction to orderly2","text":"orderly report directory src/<name> containing orderly file <name>.R. file may special commands , now ’ll create one simple possible; ’ll create random data save disk. seems silly, imagine standing something like: downloading file external site resource running simulation saving output fitting model data merging set files together create final data set directory structure (ignoring .outpack) looks like: src/incoming_data/incoming_data.R contains: run report create new packet, use orderly2::orderly_run(): id created new identifier packet unique among packets (within reason) chronologically sortable. packet id sorts another packet’s id started packet. run report, directory structure looks like: things changed : file created ran report (data.rds; see script ) log happened report run packet created incoming_data.R data.csv, original input come source tree empty directory draft/incoming_data created orderly ran report first place addition, quite files changed within .outpack directory, covered . ’s ! Notice initial script just plain R script, can develop interactively within src/incoming_data directory. Note however, paths referred within relative src/incoming_data orderly repository root. important reports see world relative incoming_data.R file. created, can refer report id pull files wherever need , context another orderly report just copy desktop email someone. example, copy file data.rds created location outside orderly’s control copies data.rds new temporary directory dest name final.rds. uses orderly2’s outpack_ functions, designed interact outpack archives regardless created (orderly2 program creates outpack archives). Typically lower-level orderly_ functions.","code":"## . ## ├── orderly_config.yml ## └── src ##     └── incoming_data ##         ├── data.csv ##         └── incoming_data.R d <- read.csv(\"data.csv\") d$z <- resid(lm(y ~ x, d)) saveRDS(d, \"data.rds\") id <- orderly2::orderly_run(\"incoming_data\") ## ℹ Starting packet 'incoming_data' `20240710-102613-8bebab5b` at 2024-07-10 10:26:13.55152 ## > d <- read.csv(\"data.csv\") ## > d$z <- resid(lm(y ~ x, d)) ## > saveRDS(d, \"data.rds\") ## ✔ Finished running incoming_data.R ## ℹ Finished 20240710-102613-8bebab5b at 2024-07-10 10:26:13.616706 (0.06518626 secs) id ## [1] \"20240710-102613-8bebab5b\" ## . ## ├── archive ## │   └── incoming_data ## │       └── 20240710-102613-8bebab5b ## │           ├── data.csv ## │           ├── data.rds ## │           └── incoming_data.R ## ├── draft ## │   └── incoming_data ## ├── orderly_config.yml ## └── src ##     └── incoming_data ##         ├── data.csv ##         └── incoming_data.R dest <- tempfile() fs::dir_create(dest) orderly2::orderly_copy_files(id, files = c(\"final.rds\" = \"data.rds\"),                              dest = dest)"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"depending-on-packets-from-another-report","dir":"Articles","previous_headings":"","what":"Depending on packets from another report","title":"Introduction to orderly2","text":"Creating new dataset mostly useful someone else can use . introduce first special orderly commands can use orderly file src/ directory now looks like: src/analysis/analysis.R contains: , ’ve used orderly2::orderly_dependency() pull file data.rds recent version (latest()) data packet filename incoming.rds, ’ve used file normal make plot, ’ve saved analysis.png. can run just , using orderly2::orderly_run(): information dependencies, see vignette(\"dependencies\").","code":"## src ## ├── analysis ## │   └── analysis.R ## └── incoming_data ##     ├── data.csv ##     └── incoming_data.R orderly2::orderly_dependency(\"incoming_data\", \"latest()\",                              c(\"incoming.rds\" = \"data.rds\")) d <- readRDS(\"incoming.rds\") png(\"analysis.png\") plot(y ~ x, d) dev.off() id <- orderly2::orderly_run(\"analysis\") ## ℹ Starting packet 'analysis' `20240710-102614-134183cb` at 2024-07-10 10:26:14.078362 ## > orderly2::orderly_dependency(\"incoming_data\", \"latest()\", ## +                              c(\"incoming.rds\" = \"data.rds\")) ## ℹ Depending on incoming_data @ `20240710-102613-8bebab5b` (via latest(name == \"incoming_data\")) ## > d <- readRDS(\"incoming.rds\") ## > png(\"analysis.png\") ## > plot(y ~ x, d) ## > dev.off() ## agg_png  ##       2 ## ✔ Finished running analysis.R ## ℹ Finished 20240710-102614-134183cb at 2024-07-10 10:26:14.161607 (0.08324504 secs)"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"available-in-report-orderly-commands","dir":"Articles","previous_headings":"","what":"Available in-report orderly commands","title":"Introduction to orderly2","text":"function orderly2::orderly_dependency() designed operate packet runs. functions act adding metadata final packet, perhaps copying files directory. orderly2::orderly_description(): Provide longer name description report; can reflected tooling uses orderly metadata much informative short name. orderly2::orderly_parameters(): Declares parameters can passed control behaviour report. Parameters key-value pairs simple data (booleans, numbers, strings) report can respond . can also used queries orderly2::orderly_dependency() find packets satisfy criteria. orderly2::orderly_resource(): Declares file resource; file input report, comes source directory. default, orderly treats files directory resource, can useful mark explicitly, necessary “strict mode” (see ). Files marked resource immutable may deleted modified. orderly2::orderly_shared_resource(): Copies file “shared resources” directory shared/, can data files source code located root orderly repository. can reasonable way sharing data commonly used code among several reports. orderly2::orderly_artefact(): Declares file (set files) created report, even run. makes easier check report behaves expected can allow reasoning related set reports without running . declaring something artefact (especially conjunction “strict mode”) also easier clean src directories used interactive development (see ). orderly2::orderly_dependency(): Copy files one packet packet runs, seen . orderly2::orderly_strict_mode(): Declares report run “strict mode” (see ). addition, also function orderly::orderly_run_info() can used running report returns information currently running report (id, resolved dependencies etc). Let’s add additional annotations previous reports: , ’ve added block special orderly commands; go anywhere, example files refer . strict mode enabled (see ) orderly2::orderly_resource calls must go files used made available point (see ).","code":"orderly2::orderly_strict_mode() orderly2::orderly_resource(\"data.csv\") orderly2::orderly_artefact(\"Processed data\", \"data.rds\")  d <- read.csv(\"data.csv\") d$z <- resid(lm(y ~ x, d)) saveRDS(d, \"data.rds\") id <- orderly2::orderly_run(\"incoming_data\") ## ℹ Starting packet 'incoming_data' `20240710-102614-5fcf9915` at 2024-07-10 10:26:14.379319 ## > orderly2::orderly_strict_mode() ## > orderly2::orderly_resource(\"data.csv\") ## > orderly2::orderly_artefact(\"Processed data\", \"data.rds\") ## > d <- read.csv(\"data.csv\") ## > d$z <- resid(lm(y ~ x, d)) ## > saveRDS(d, \"data.rds\") ## ✔ Finished running incoming_data.R ## ℹ Finished 20240710-102614-5fcf9915 at 2024-07-10 10:26:14.405794 (0.02647424 secs)"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"parameterised-reports","dir":"Articles","previous_headings":"","what":"Parameterised reports","title":"Introduction to orderly2","text":"Much flexibility comes orderly graph comes using parameterised reports; reports take set parameters change behaviour based parameters. Downstream reports can depend parameterised report filter based suitable parameters. example, consider simple report generate samples based parameter: creates report single parameter n_samples default value 10. used define parameter default, defined multiple parameters can anything report switches value parameter: might read different URLs fetch different underlying data might fit different analysis might read different shared resources (see ) might depend different dependencies might produce different artefacts However, see parameters relatively heavyweight things try consistent set packets created report. report use control size generated data set. resulting file 15 rows, parameter passed affected report: can use parameters orderly’s search functions. example can find recent version packet running: can also pass parameter queries : can used within orderly2::orderly_dependency() (name == \"random\" part implied first name argument), example case report querying also parameters can use within query, using prefix. suppose downstream report simply uses n number samples might write: depend recent packet called random parameter n_samples value current report’s parameter n. See outpack query documentation much detail .","code":"orderly2::orderly_parameters(n_samples = 10) x <- seq_len(n_samples) d <- data.frame(x = x, y = x + rnorm(n_samples)) saveRDS(d, \"data.rds\") orderly2::orderly_parameters(n_samples = NULL) orderly2::orderly_parameters(n_samples = 10, distribution = \"normal\") id <- orderly2::orderly_run(\"random\", list(n_samples = 15)) ## ℹ Starting packet 'random' `20240710-102614-9f6109cc` at 2024-07-10 10:26:14.625971 ## ℹ Parameters: ## • n_samples: 15 ## > orderly2::orderly_parameters(n_samples = 10) ## > x <- seq_len(n_samples) ## > d <- data.frame(x = x, y = x + rnorm(n_samples)) ## > saveRDS(d, \"data.rds\") ## ✔ Finished running random.R ## ℹ Finished 20240710-102614-9f6109cc at 2024-07-10 10:26:14.657642 (0.03167081 secs) orderly2::orderly_copy_files(id, files = c(\"random.rds\" = \"data.rds\"),                              dest = dest) readRDS(file.path(dest, \"random.rds\")) ##     x          y ## 1   1  0.4463006 ## 2   2  2.6289820 ## 3   3  5.0650249 ## 4   4  2.3690106 ## 5   5  5.5124269 ## 6   6  4.1369885 ## 7   7  6.4779875 ## 8   8  7.9473981 ## 9   9  9.5429963 ## 10 10  9.0859252 ## 11 11 11.4681544 ## 12 12 12.3629513 ## 13 13 11.6954565 ## 14 14 14.7377763 ## 15 15 16.8885049 orderly2::orderly_search('latest(name == \"random\")') ## [1] \"20240710-102614-9f6109cc\" orderly2::orderly_search('latest(name == \"random\" && parameter:n_samples > 10)') ## [1] \"20240710-102614-9f6109cc\" orderly2::orderly_dependency(\"random\", \"latest(parameter:n_samples > 10)\",                              c(\"randm.rds\" = \"data.rds\")) orderly2::orderly_dependency(\"random\", \"latest(parameter:n_samples == this:n)\",                              c(\"randm.rds\" = \"data.rds\"))"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"shared-resources","dir":"Articles","previous_headings":"","what":"Shared resources","title":"Introduction to orderly2","text":"Sometimes useful share data different reports, example common source utilities don’t warrant package, common data. , create directory shared orderly root put files directories might want share. Suppose shared directory contains file data.csv: can write orderly report use_shared uses shared file, use_shared.R containing: can run : resulting archive, file used shared directory present: general property orderly: tries save inputs alongside final results analysis, later can check see went analysis might changed versions.","code":"## . ## ├── archive ## │   ├── analysis ## │   │   └── 20240710-102614-134183cb ## │   │       ├── analysis.R ## │   │       ├── analysis.png ## │   │       └── incoming.rds ## │   ├── incoming_data ## │   │   ├── 20240710-102613-8bebab5b ## │   │   │   ├── data.csv ## │   │   │   ├── data.rds ## │   │   │   └── incoming_data.R ## │   │   └── 20240710-102614-5fcf9915 ## │   │       ├── data.csv ## │   │       ├── data.rds ## │   │       └── incoming_data.R ## │   └── random ## │       └── 20240710-102614-9f6109cc ## │           ├── data.rds ## │           └── random.R ## ├── draft ## │   ├── analysis ## │   ├── incoming_data ## │   └── random ## ├── orderly_config.yml ## ├── shared ## │   └── data.csv ## └── src ##     ├── analysis ##     │   └── analysis.R ##     ├── incoming_data ##     │   ├── data.csv ##     │   └── incoming_data.R ##     └── random ##         └── random.R orderly2::orderly_shared_resource(\"data.csv\") orderly2::orderly_artefact(\"analysis\", \"analysis.png\")  d <- read.csv(\"data.csv\") png(\"analysis.png\") plot(y ~ x, d) dev.off() id <- orderly2::orderly_run(\"use_shared\") ## ℹ Starting packet 'use_shared' `20240710-102615-2421c3c7` at 2024-07-10 10:26:15.144337 ## > orderly2::orderly_shared_resource(\"data.csv\") ## > orderly2::orderly_artefact(\"analysis\", \"analysis.png\") ## > d <- read.csv(\"data.csv\") ## > png(\"analysis.png\") ## > plot(y ~ x, d) ## > dev.off() ## agg_png  ##       2 ## ✔ Finished running use_shared.R ## ℹ Finished 20240710-102615-2421c3c7 at 2024-07-10 10:26:15.181531 (0.03719354 secs) ## archive/use_shared ## └── 20240710-102615-2421c3c7 ##     ├── analysis.png ##     ├── data.csv ##     └── use_shared.R"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"strict-mode","dir":"Articles","previous_headings":"","what":"Strict mode","title":"Introduction to orderly2","text":"previous version orderly (orderly1; see vignette(\"migrating\")) fussy input strictly declared report run, clear really required order run something. version 2 relaxed default, can opt old behaviours checks adding anywhere within orderly file (conventionally top). may make granular future, adding : copy files source directory (src/<reportname>/) draft directory report runs (draft/<reportname>/<packet-id>) declared orderly2::orderly_resource; leaves behind extra files left development warn end running packet files found part artefact Using strict mode also helps orderly2 clean src/<reportname> directory effectively interactive development (see next section).","code":"orderly2::orderly_strict_mode()"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"interactive-development","dir":"Articles","previous_headings":"","what":"Interactive development","title":"Introduction to orderly2","text":"Set working directory src/<reportname> orderly script fully executable (e.g., source Rstudio’s Source button, R’s source() function). Dependencies copied needed. , mix files within source directory. recommend per-source-directory .gitignore keep files version control (see ). soon implement support cleaning generated files directory. example, suppose interactively run incoming_data/incoming_data.R script, leave behind generated files. can report orderly2::orderly_cleanup_status: files unknown orderly tell prompt tell explicitly. can clean generated files running (suggested message): dry_run = TRUE argument can pass want see deleted without using status function. can also keep files git using orderly2::orderly_gitignore_update function: creates (updates) .gitignore file within report generated files included git. already accidentally committed gitignore real effect git surgery, see git manuals handy, profane, guide.","code":"orderly2::orderly_cleanup_status(\"incoming_data\") ## ✖ incoming_data is not clean: ## ℹ 1 file can be deleted by running 'orderly2::orderly_cleanup(\"incoming_data\")': ##   • data.rds orderly2::orderly_cleanup(\"incoming_data\") ## ℹ Deleting 1 file from 'incoming_data': ## • data.rds orderly2::orderly_gitignore_update(\"incoming_data\") ## ✔ Wrote 'src/incoming_data/.gitignore'"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"deleting-things-from-the-archive","dir":"Articles","previous_headings":"","what":"Deleting things from the archive","title":"Introduction to orderly2","text":"delete packets archive/ directory puts orderly2 inconsistent state metadata store. Sometimes matter (e.g., delete old copies never candidates inclusion orderly2::orderly_dependency never notice). However, delete recent copy packet try depend , get error. moment, two copies incoming_data task: run analysis task, pull recent version (20240710-102614-5fcf9915). However, deleted manually (e.g., save space accidentally) corrupted (e.g., opening output Excel letting save changes) able included, running analysis fail: error tries fairly informative, telling us failed copying files 20240710-102614-5fcf9915 found packet corrupt, file data.rds found archive. also suggests fix; can tell orderly2 20240710-102614-5fcf9915 “orphaned” considered inclusion look dependencies. can carry suggestion just validate packet running can validate packets : option core.require_complete_tree enabled, process also look packets used now-deleted packet orphan , longer complete tree includes . want remove references orphaned packets, can use orderly2::orderly_prune_orphans() remove entirely:","code":"orderly2::orderly_metadata_extract(   name = \"incoming_data\",   extract = c(time = \"time.start\")) ##                         id                time ## 1 20240710-102613-8bebab5b 2024-07-10 10:26:13 ## 2 20240710-102614-5fcf9915 2024-07-10 10:26:14 orderly2::orderly_run(\"analysis\") ## ℹ Starting packet 'analysis' `20240710-102615-c4877123` at 2024-07-10 10:26:15.770787 ## > orderly2::orderly_dependency(\"incoming_data\", \"latest()\", ## +                              c(\"incoming.rds\" = \"data.rds\")) ## ✖ Error running analysis.R ## ℹ Finished 20240710-102615-c4877123 at 2024-07-10 10:26:15.853003 (0.08221602 secs) ## Error in `orderly2::orderly_run()`: ## ! Failed to run report ## Caused by error in `orderly_copy_files()`: ## ! Unable to copy files, due to deleted packet 20240710-102614-5fcf9915 ## ℹ Consider 'orderly2::orderly_validate_archive(\"20240710-102614-5fcf9915\", ##   action = \"orphan\")' to remove this packet from consideration ## Caused by error: ## ! File not found in archive ## ✖ data.rds orderly2::orderly_validate_archive(\"20240710-102614-5fcf9915\", action = \"orphan\") orderly2::orderly_validate_archive(action = \"orphan\") ## ✔ 20240710-102613-8bebab5b (incoming_data) is valid ## ✔ 20240710-102614-134183cb (analysis) is valid ## ✖ 20240710-102614-5fcf9915 (incoming_data) is invalid due to its files ## ✔ 20240710-102614-9f6109cc (random) is valid ## ✔ 20240710-102615-2421c3c7 (use_shared) is valid orderly2::orderly_prune_orphans() ## ℹ Pruning 1 orphan packet"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"debugging-and-coping-with-errors","dir":"Articles","previous_headings":"","what":"Debugging and coping with errors","title":"Introduction to orderly2","text":"(written)","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"interaction-with-version-control","dir":"Articles","previous_headings":"","what":"Interaction with version control","title":"Introduction to orderly2","text":"guidelines: Make sure exclude files git listing .gitignore: .outpack/ - nothing suitable version control archive/ - core.archive_path set non-null value, excluded. default archive draft/ - temporary draft directory orderly_envir.yml - used setting machine-specific configuration absolutely version control files: src/ main source analyses orderly_config.yml - high level configuration suitable sharing shared resource directory (configured orderly_config.yml) probably version controlled source repository end multiple people’s machines, configured differently. configuration option set via orderly2::orderly_config_set designed (potentially) different different users, configuration needs version controlled. also means reports/packets can’t directly refer values set . includes directory used save archive packets (enabled) names locations (equivalent git remotes). may find useful include scripts help users set common locations, like git, different users may use different names remote (e.g., one user may location called data another called data-incoming, depending perspective use location). orderly2 always try save information current state git source repository alongside packet metadata. includes current branch, commit (sha) remote url. try create links final version packet upstream source repository.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"interaction-with-the-outpack-store","dir":"Articles","previous_headings":"","what":"Interaction with the outpack store","title":"Introduction to orderly2","text":"alluded , .outpack directory contains lots information packets run, typically “bounds” normal use. effectively “database” information packets run. Understanding directory structured required using orderly, included avoidance mystery! See outpack documentation (vignette(\"outpack\")) details ideas . work , directory structure looks like: can perhaps inferred filenames, files .outpack/metadata/<packet-id> metadata packet run. files .outpack/location/<location-id>/<packet-id> holds information packet first known location (location special “local” location). default orderly configuration store final files directory called archive/, alternatively (additionally) can use content- addressable file store. enabled, .outpack directory looks like: files .outpack/files/ never modified deleted. approach storage naturally deduplicates file archive, large file used many places ever stored .","code":"## .outpack ## ├── config.json ## ├── index ## │   └── outpack.rds ## ├── location ## │   ├── local ## │   │   ├── 20240710-102613-8bebab5b ## │   │   ├── 20240710-102614-134183cb ## │   │   ├── 20240710-102614-9f6109cc ## │   │   └── 20240710-102615-2421c3c7 ## │   └── orphan ## └── metadata ##     ├── 20240710-102613-8bebab5b ##     ├── 20240710-102614-134183cb ##     ├── 20240710-102614-9f6109cc ##     └── 20240710-102615-2421c3c7 ## .outpack ## ├── config.json ## ├── files ## │   └── sha256 ## │       ├── 0a ## │       │   └── a82571c21c4e5f1f435e8bef2328dda5ef47e177d78d63d1c4ec647a5a388a ## │       ├── 25 ## │       │   └── 4947c281b203719c72949745123a1d017e2f9b50c048b1d24a0803d73ba0b8 ## │       ├── 40 ## │       │   └── f8c6c3993686ee8acf38ce936911d6f65372d489e78748ec6d1f27d277daa7 ## │       ├── 42 ## │       │   └── 438203a847c7cd496e3e603ddeeceb4e136f1bd18f1efb81332fa093964f7b ## │       ├── 51 ## │       │   └── f8070f04f57a8040ad215474b976bcddc891bfc544e6c6a0519e74d79cb104 ## │       ├── 5f ## │       │   └── 96f49230c2791c05706f24cb2335cd0fad5d3625dc6bca124c44a51857f3f8 ## │       ├── a6 ## │       │   └── 80ab7c65a52327a3d9c5499d114f513f18eabe7f63a98f9fc308c2b3744c82 ## │       ├── d9 ## │       │   └── 1699ae410cbd811e1f028f8a732e5162b7df854eec08d921141f965851272d ## │       ├── de ## │       │   └── 1329bc0e9f8ee0c8b3376eb09b71e93587f707dfeaa01d1a03ae32f97c928a ## │       └── ec ## │           └── b53285781a4d36c65168c80ee14f2af2c885423c6166b9425f40c3c6cd8297 ## ├── index ## │   └── outpack.rds ## ├── location ## │   ├── local ## │   │   ├── 20240710-102613-8bebab5b ## │   │   ├── 20240710-102614-134183cb ## │   │   ├── 20240710-102614-9f6109cc ## │   │   └── 20240710-102615-2421c3c7 ## │   └── orphan ## └── metadata ##     ├── 20240710-102613-8bebab5b ##     ├── 20240710-102614-134183cb ##     ├── 20240710-102614-9f6109cc ##     └── 20240710-102615-2421c3c7"},{"path":"https://mrc-ide.github.io/orderly2/articles/introduction.html","id":"relationship-between-orderly-and-outpack","dir":"Articles","previous_headings":"","what":"Relationship between orderly and outpack","title":"Introduction to orderly2","text":"orderly2 package built metadata file storage system called outpack; implementing support working metadata archives languages (see outpack_server server implementation Rust outpack-py Python). metadata discussed detail vignette(\"metadata\") document general ideas fully mrc-ide/outpack","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"basic-overview","dir":"Articles","previous_headings":"","what":"Basic overview","title":"Outpack metadata","text":"“packet” conceptually directory, corresponding particular analysis data product, though necessarily stored. internal representation includes: metadata packet - files contains hashes along information done create environment created etc (see ) actual file contents every included file. might stored simple content-addressable file storage system (see ) actual copies directories originally created Every packet referenced uniquely primary key. use key format encodes current date time, well random data avoid collisions. exists dependency graph among packets, one packet depends another. edge graph hard link (one packet another id) also query (e.g., latest packet name) used find packet. means many ways looking thinking dependency graph. packets available locally, outpack repositories, typically (always) machines accessed HTTP API. conceptually similar git “remotes”. need distinguish packets “unpacked” (, packets every file available current archive) packets merely known (metadata files). sometimes refer unpacked packets “local” known “local” location special. use terms “archive” “repository” fairly interchangeably try nail .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"what-goes-into-a-packet","dir":"Articles","previous_headings":"","what":"What goes into a packet","title":"Outpack metadata","text":"packet must things: name (example model_fits). changed (rather changes tracked) currently way namespacing different repositories id (see “outpack id” ) created outpack point packet started set parameters - mapping strings scalar type (boolean, number, string). Packets may zero many parameters. set files - local (relative) path within packet (e.g., output/data.csv) also hash (e.g., sha256:69f6cf230416cf40828da251a0dad17cbbf078587883e826f3345ff08d1aaa7d) Information packet run (start end time), represented seconds post epoch, UTC (see ). addition may contain information : dependencies used, tracking name included packet, query used find , resolved id, set files extracted packet (source name, hash destination name) names scripts run create packet (typically also included files within final packet) information state git point packet run (current branch, hash, etc) information system used run packet (format unspecified might include hostname, operating system version, package/module versions, etc) custom metadata added implementation","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"types-of-users-of-outpack","dir":"Articles","previous_headings":"","what":"Types of users of outpack","title":"Outpack metadata","text":"types “persona” outpack user imagine exist guide decisions abut layout . extremes : user desktop wants develop run analyses, using files present packets available server. want visually inspect results running packets, need see files disk way makes sense human, care complete copy dependencies graph. server instance responsible delivering packets users (e.g., persona ). expected (able fetch) part outpack graph, expect anyone want visually inspect packets directly machine (, storage need human-readable). impacts two configuration options associated parts directory structure : user use content-addressable “file store” human-readable “archive” (explained detail ) require archive contains complete tree dependencies? expect first persona wants human readable archive contain full tree, second wants opposite.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"directory-layout","dir":"Articles","previous_headings":"","what":"Directory layout","title":"Outpack metadata","text":"section discusses files directory make outpack work, much come ; see . typical .outpack directory layout looks like : (note archive/ .outpack level). directories necessarily present; indeed required file .outpack/config.json.","code":".outpack/   config.json   files/   location/   metadata/ archive/"},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"configuration--outpackconfig-json","dir":"Articles","previous_headings":"Directory layout","what":"Configuration (.outpack/config.json)","title":"Outpack metadata","text":"outpack configuration schema defined config.json configuration format still subject change…","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"packet-metadata--outpackmetadata","dir":"Articles","previous_headings":"Directory layout","what":"Packet metadata (.outpack/metadata/)","title":"Outpack metadata","text":"file within directory filename outpack id (matching regular expression ^[0-9]{8}-[0-9]{6}-[0-9a-f]{8}$, see . file json file conforming schema metadata.json. present means outpack implementation can report information back packet (created, files contains, etc), packet metadata meaningful ; want know might come (location distributing packet) copy packet locally.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"location-information--outpacklocation","dir":"Articles","previous_headings":"Directory layout","what":"Location information (.outpack/location/)","title":"Outpack metadata","text":"directory matches regular expression ^[0-9]{8}$ (e.g., 457f4f2a) “location id” (see ) corresponding “location”. file within directory outpack id name, contains json location unpacked (installed) packet, hash metadata. file conforms schema location.json.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"a-file-store--outpackfiles","dir":"Articles","previous_headings":"Directory layout","what":"A file store (.outpack/files)","title":"Outpack metadata","text":"configuration option core.use_file_store true, outpack keeps content addressable file store files knows . much space efficient entire packet unpacked automatically deduplicates shared content among packets (e.g., large file present two packets stored ). file store layout described . storage format human-readable (indeed present within hidden directory .outpack). can enabled either server user","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"an-archive-archive-by-default","dir":"Articles","previous_headings":"Directory layout","what":"An archive (archive/ by default)","title":"Outpack metadata","text":"configuration option core.path_archive non-null directory path containing unpacked packets. packet available path <name> “name” packet, <id> outpack id. several files per packet, possibly directories. storage approach designed human readable, typically enabled outpack repository used laptop user wants interactively work files.","code":"archive/<name>/<id>/<files...>"},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"adding-a-packet","dir":"Articles","previous_headings":"","what":"Adding a packet","title":"Outpack metadata","text":"order make packet available locally, need import metadata files, mark packet available. roughly creating packet (.e., first place packet ever existed) importing packet elsewhere. Making packet available allows used dependency, allows serving packet acting location (http file protocols), guarantees files actually present locally.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"adding-metadata","dir":"Articles","previous_headings":"Adding a packet","what":"Adding metadata","title":"Outpack metadata","text":"can simply copy metadata file .outpack/metadata/<packet id> yet exist. make available anything yet known location. Dangling metadata (, metadata present directory known anywhere) currently mostly ignored.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"adding-files","dir":"Articles","previous_headings":"Adding a packet","what":"Adding files","title":"Outpack metadata","text":"repository uses file store, fill first, much easier think . can easily get difference files used packet (list files packet manifest) already file store looking hash turn. request missing files insert store. may leave “dangling” files (files referred packet) problem. repository human-readable archive uses file store, files present file store easy enough check file store requested path (local relative path packet manifest). update file store first, files guaranteed present. repository uses human readable archive, simplest thing request file remote. However, might efficient check locally previously fetched copies files content, verify modified, copy place rather re-downloading.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"marking-the-packet-as-unpacked-and-known-locally","dir":"Articles","previous_headings":"Adding a packet","what":"Marking the packet as unpacked and known locally","title":"Outpack metadata","text":"local location id, write file conforming location.json schema, containing packet id, time marked unpacked hash metadata.","code":".outpack/<local location id>/<packet id>"},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"ordering-of-operations","dir":"Articles","previous_headings":"Adding a packet","what":"Ordering of operations","title":"Outpack metadata","text":"need files metadata packet marked unpacked; note configurations guarantee every packet unpacked complete tree. can import files first, metadata; lot disadvantage either. mark package unpacked known locally though components present.","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"outpack-ids","dir":"Articles","previous_headings":"Details","what":"Outpack ids","title":"Outpack metadata","text":"Outpack ids match regular expression ^[0-9]{8}-[0-9]{6}-[0-9a-f]{8}$; encode UTC date-time prefix YYYYMMDD-HHMMSS- followed 8 hexadecimal digits. R implementation, encode current second first four digits (2 bytes) append 2 bytes cryptographically random data. id tries balance reasonable degree collision resistance (65536 combinations per millisecond), lexicographic sortability reasonable degree meaningfulness.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"location-ids","dir":"Articles","previous_headings":"Details","what":"Location ids","title":"Outpack metadata","text":"Location ids meaningless 4-byte (8 character) hex strings. immutable created different different machines even point location. location id mapped (via .outpack/config.json) location name human-readable name (e.g., production staging). requirement name different machines. One directories represents local location; can find mapping within configuration.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"representing-hashes","dir":"Articles","previous_headings":"Details","what":"Representing hashes","title":"Outpack metadata","text":"Outpack typically uses sha256 hashes, want able change future. wherever hash presented, algorithm included part string. example instead used md5 algorithm written schema currently assumes hash value represented hex string.","code":"sha256:69f6cf230416cf40828da251a0dad17cbbf078587883e826f3345ff08d1aaa7d md5:bd57f7123c6bfb95c3234ff56373b7f4"},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"times","dir":"Articles","previous_headings":"Details","what":"Times","title":"Outpack metadata","text":"store information times places (e.g., times packet run, imported, etc). Rather trying deal strings, always store time seconds since 1970-01-01 00:00.00 UTC (including fractional seconds, whatever accuracy system allows).","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/metadata.html","id":"file-store","dir":"Articles","previous_headings":"Details","what":"File store","title":"Outpack metadata","text":"file store designed simple, sophisticated git, whose object store similar thing. general layout looks like: hopefully fairly obvious structure. Paths format: reason second level prevent performance degradation directories containing millions files, copying git. store designed cope different hashing algorithms, though R implementation outpack supports sha256 now. Multiple hashing algorithms supported hard linking content multiple places tree, might link ","code":".outpack/files   sha256/     5d/       dfaf1f4a2e15e8fe46dbed145bf2f84bba1b3367d0a56f73de08f8585dd153       ...     77/     ... <algorithm>/<first two bytes>/<remaining bytes> sha256/5d/dfaf1f4a2e15e8fe46dbed145bf2f84bba1b3367d0a56f73de08f8585dd153 md5/84/0bc6ad3ae479dccc1c49a1910b37bd"},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"so-long-yaml-and-thanks-for-all-the-whitespace-errors","dir":"Articles","previous_headings":"Summary of changes","what":"So long YAML and thanks for all the whitespace errors","title":"Migrating from orderly (1.x)","text":"obvious user-facing change (almost) YAML, definition inputs outputs report now defined within orderly file, <reportname>.R. orderly report previously orderly.yml file looked like end within orderly file looks like: think much clearer, comes documentation autocomplete support IDEs. fact, simple reports, special functions required, though ’ll find useful (see vignette(\"orderly2\")) specific changes: longer need list packages packages:, instead just use library() ordinary script. record state session regardless get record used longer need use sources: list scripts want source, instead use source() normal global_resources become orderly2::orderly_shared_resource (aren’t really global much shared). Note directory now always shared/ orderly root, may configure .","code":"parameters:   n_min:     default: 10 script: script.R source:   - functions.R resources:   - metadata.csv depends:   raw_data:     id: latest     use:       raw_data.csv: data.csv artefacts:   data:     description: Processed data     filenames: data.rds orderly2::orderly_parameters(n_min = 10) orderly2::orderly_dependency(\"raw_data\", \"latest\",                               files = c(\"raw_data.csv\" = \"data.csv\")) orderly2::orderly_resource(\"metadata.csv\") orderly2::orderly_artefact(\"Processed data\", \"data.rds\") source(\"functions.R\")"},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"implications","dir":"Articles","previous_headings":"Summary of changes > So long YAML and thanks for all the whitespace errors","what":"Implications","title":"Migrating from orderly (1.x)","text":"change widespread implications: can program things like dependencies, creating loop series parameter values, conditionally depending reports R script can basis orderly report, can add orderly functions annotating inputs outputs progressively","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"database-support-has-been-moved-into-a-plugin","dir":"Articles","previous_headings":"Summary of changes","what":"Database support has been moved into a plugin","title":"Migrating from orderly (1.x)","text":"version 1, built-support accessing data SQL databases; moved within orderly.db plugin. major features supported.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"no-more-commit","dir":"Articles","previous_headings":"Summary of changes","what":"No more commit","title":"Migrating from orderly (1.x)","text":"orderly2 longer requires separate orderly_commit() call orderly_run(); longer make distinction local draft archive packets. Instead, added finer-grained control dependencies resolved (locally, subset servers), generalises way draft/archive used practice. See ?orderly_run details dependencies resolved. implications deleting things; draft directory always easy target deletion, now deletion need tell orderly2 deleted things. See vignette(\"introduction\") details (section “Deleting things archive”).","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"no-more-testing-or-development-mode","dir":"Articles","previous_headings":"Summary of changes","what":"No more testing or development mode","title":"Migrating from orderly (1.x)","text":"two different, unsatisfactory, mechanisms developing orderly report: orderly_test_start (version 1.1.0) orderly_develop_start (1.1.0 onwards) worked initial setup, copying dependencies etc location work (new draft orderly_test_start source directory orderly_develop_start). orderly2 can just work directly within source directory, long working directory set src/<report-name>, orderly2 commands work expected. orderly1, need careful commit (git) results running analysis, encourage per-report .gitignore files help .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"new-language-agnostic-backend","dir":"Articles","previous_headings":"Summary of changes","what":"New, language-agnostic, backend","title":"Migrating from orderly (1.x)","text":"biggest change, perhaps least visible, orderly now built open spec outpack can implemented language. develop Python implementation , possibly languages. takes control metadata. split “orderly_” “outpack_” functions package, information see last section vignette(\"introduction\") also vignette(\"outpack\").","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"other-smaller-changes","dir":"Articles","previous_headings":"Summary of changes","what":"Other, smaller changes","title":"Migrating from orderly (1.x)","text":"“Global” resources become “shared” resources, always live shared/ orderly root (.e., longer configurable). reason want reports able run fairly independently orderly2 configuration (exception plugins). practice people really vary .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"what-is-missing-compared-with-orderly1","dir":"Articles","previous_headings":"","what":"What is missing compared with orderly1","title":"Migrating from orderly (1.x)","text":"changelog feature (implement something support ) extraction secrets vault (support ) automatic handling README files (may implement support )","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"how-to-migrate","dir":"Articles","previous_headings":"","what":"How to migrate","title":"Migrating from orderly (1.x)","text":"two parts migration: updating canonical copy orderly archive (ideally one ) updating source tree. steps done via outpack.orderly package. migrate archive first. every archive want retain (might archives stored locally, production servers staging servers). Archive migration happens place; , modify anything original location. archive old used old versions orderly1 possible process hiccups. Please let us know case. result process end new directory contains new archive conforming outpack spec containing orderly2 metadata. Next, migrate source tree. done place done fresh clone source git repository. report, examine orderly.yml files script files (often script.R), delete , write new orderly file adapt report work orderly2. possible perfect might need minor tweaking hopefully reasonable. One thing preserved (probably ) comments yaml often refer yaml formatting orderly1 features hopefully much problem. probably want manually tweak generated code anyway, take advantage new orderly2 features able compute dependencies. using OrderlyWeb, probably need pause migrating, replacement yet ready.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"what-about-the-packages","dir":"Articles","previous_headings":"","what":"What about the packages?","title":"Migrating from orderly (1.x)","text":"merge orderly2 orderly package, ready release can use . However, anticipate period coexistence legacy orderly1 systems develop orderly2. help small helper package orderly.helper can smooth namespace differences; may useful interact versions.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/migrating.html","id":"what-were-the-problems-in-version-1","dir":"Articles","previous_headings":"","what":"What were the problems in version 1","title":"Migrating from orderly (1.x)","text":"YAML format inflexible, error prone users, leads duplication focussed around initial needs Vaccine Impact Modelling Consortium fairly easy get archive SQLite database inconsistent state (e.g., deleting moving files archive) SQLite database behaved poorly shared file systems","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/orderly2.html","id":"collaborative-analysis","dir":"Articles","previous_headings":"","what":"Collaborative analysis","title":"orderly2","text":"Many analyses involve single person single machine; case number workflow tools make orchestrating analysis easy. workflow model graph dependencies analysis, data flows. example might update data, whole pipeline rerun. update code forecast, forecasts report rerun. experience, model works well single-user setting falls collaborative setting, especially analysis partitioned person; Alice handling data pipeline, Bob running fits, Carol organising forecasts final report. context, changes upstream affect downstream analysis, require set care around integration might used version controlling source code. example, Alice dealing change incoming data format going break analysis time Bob trying get model fits working, Bob trying integrate code changes Alice’s new data. typically deal source code using branches within git; code Bob work branch isolated Alice’s changes. contexts like () data analysis products git. needed way versioning outputs step analysis controlling integrated subsequent analyses. Another way looking problem seek way making analysis composable way functions OOP achieve programs, way docker containerisation achieved deploying software. need way putting interfaces around pieces analysis allow people refer fetch somewhere run. conceptual pieces needed : way referring unambiguously (globally) piece analysis can depended upon, everyone can agree ’re talking piece analysis system storage results running analysis can shared among group system control integration pieces analysis takes place refer transportable unit analysis “packet”. conceptually directory files created running code, atomic unit work point view orderly. packet underlying source form, anyone can run. However, time people use pre-run packets collaborators run inputs onward analyses (see vignette(\"dependencies\") vignette(\"collaboration\") details).","code":"[raw data] -> [processed data] -> [model fits] -> [forecasts] -> [report]"},{"path":"https://mrc-ide.github.io/orderly2/articles/orderly2.html","id":"reproducible-analyses","dir":"Articles","previous_headings":"","what":"Reproducible analyses","title":"orderly2","text":"degree collaboration style requires reproducibility, several aspects . system describe , even though everyone can typically run step analysis, typically don’t. differs workflow tools, users may familiar .","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/orderly2.html","id":"difference-from-a-workflow-system","dir":"Articles","previous_headings":"Reproducible analyses","what":"Difference from a workflow system","title":"orderly2","text":"Workflow systems hugely influential scientific computing, people co-opting build systems like make sophisticated systems designed parallel running large complex workflows nextflow. general approach define interdependencies among parts analysis, forming graph parts analysis track inputs outputs workflow. model computation lots good points: defines interface analysis allows (encourages) breaking monolithic analysis component pieces can reasoned allows high-level parallelism making obvious parts workflow can run concurrently tracking way data flows analysis, allows minimum amount recalculation done change, downstream parts triggered shared workspace online runner, allows degree collaboration long everyone happy working constantly changing set code analysis artefacts designed orderly2 working patterns suit . motivating reasons include: nodes computational graph expensive compute require exotic hardware. Workflows upstream data never settle, need know version data ends used particular analysis Workflows upstream analyses used many downstream analyses, upstream developer may know much downstream use Nondeterministic analyses, e.g., involving stochastic simulations, rerunning node expected return exact numerical results, can’t rely different users recovering results different occasions[*] cases missing piece need way versioning nodes within computational graph, shifting emphasis automatically rerunning portions graph tracking data flowed graph. turn shifts reproducibility emphasis “everyone run code get results” “everyone run code, instead work results”. familiar docker, approach similar working pre-built docker images, whereas workflow approach similar working directly Dockerfiles; many situations end result , approaches differ guarantees, computation happens, users refer versions. [*] discourage trying force determinism manually setting seeds, potential violate statistical properties random number streams, fragile best.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/orderly2.html","id":"what-is-reproducibility-anyway","dir":"Articles","previous_headings":"Reproducible analyses","what":"What is reproducibility anyway?","title":"orderly2","text":"Reproducibility means different things different people, even within narrow sense “rerun analysis retrieve results”. last decade, idea one able rerun piece analysis retrieve results slightly morphed one must rerun piece analysis. Similarly, emphasis utility reproducibility shifted authors able rerun work (confidence rerun ) hypothetical third party wanting rerun analysis. approach flips perspective around bit, based experiences collaborative research projects, draws (overly) ambitious aim : Can prove given set inputs produced given set outputs? quickly found impossible, provided systems place one satisfied statement given level trust system. piece analysis comes server primary way people run analyses web front-end (currently OrderlyWeb, soon Packit) know analysis run end--end modification orderly2 preserves inputs alongside outputs files present final packet files went analysis, recorded R package versions full set used. system naturally involves running multiple machines (typically analysts’ laptops, server perhaps HPC environment), way orderly2 treats paths, practically little problem getting analyses working multiple places, trivially satisfying typical reproducibility aim, even though people typically focussed . shift focus proved valuable. analysis run one occasion (e.g., regular reporting, simply updating figure final submission manuscript revision), outputs may change. Understanding changes happened important. orderly2 automatically saves lot metadata run easy find things might changed. , can start interrogating graph among packets find effect change ; find previously run packets pulled old version data set, used previous release package.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"the-basic-idea","dir":"Articles","previous_headings":"","what":"The basic idea","title":"Creating plugins","text":"plugin provided package, possibly thing package provides. plugin name must (currently) package name. functions package needs call orderly2::orderly_plugin orderly2::orderly_plugin_register create register plugin, respectively. make plugin available orderly project, two new bits configuration may present orderly_config.yml - one declares plugin used, configures plugin. use plugin individual report, functions plugin used, configure use plugin. Finally, can save information back final orderly2 metadata plugin . yaml-less design orderly2 (see vignette(\"migrating\") familiar orderly1), line plugin just package code fairly blurred, reasons writing plugin typically want make something easier reports, want action reflected orderly metadata.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"an-example","dir":"Articles","previous_headings":"","what":"An example","title":"Creating plugins","text":"example, ’ll implement stripped version database plugin inspired work (see `orderly.db fuller implementation). make work need functions: …process additional fields orderly_config.yml describe find database …can called orderly file access database …can add metadata final orderly metadata done ’ll start report side things, describing want happen, work implementation. directory structure minimal project orderly_config.yml file contains information shared possible uses plugin - case connection information database: plugin called example.db listed within plugins section, along configuration; case indicating path SQLite file can loaded . example.R file contains information use database specific report; case, making results query SELECT * mtcars cyl == 4 database available R object dat Normally, imagine calculation kept minimal purpose demonstration. implement need : create package write function handle configuration orderly_config.yml write function query() used example.R query ","code":"## . ## ├── orderly_config.yml ## └── src ##     └── example ##         └── example.R minimum_orderly_version: 1.99.0 plugins:   example.db:     path: /tmp/RtmplIsFe8/file1bfa3d5e9c71 dat <- example.db::query(\"SELECT * FROM mtcars WHERE cyl == 4\") orderly2::orderly_artefact(\"Summary of data\", \"data.rds\")  saveRDS(summary(dat), \"data.rds\")"},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"create-a-tiny-package","dir":"Articles","previous_headings":"An example","what":"Create a tiny package","title":"Creating plugins","text":"lots package skeleton tools , favourite, usethis::create_package() probably reasonable job. thing package needs contain Imports: orderly2 DESCRIPTION field. simple package may structure like , DESCRIPTION file contains: NAMESPACE R/plugin.R files shown .","code":"## . ## ├── DESCRIPTION ## ├── NAMESPACE ## └── R ##     └── plugin.R Package: example.db Version: 0.0.1 License: CC0 Title: Orderly Database Example Plugin Description: Simple example of an orderly plugin. Authors@R: person('Orderly Authors', role = c('aut', 'cre'),     email = 'email@example.com') Imports: orderly2"},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"handle-the-configuration","dir":"Articles","previous_headings":"An example","what":"Handle the configuration","title":"Creating plugins","text":"required function plugin needs provide one process data orderly_config.yml. probably primarily concerned validation can fairly simple first, later ’ll expand report errors nicely: arguments data: deserialised section orderly_config.yml specific plugin filename: full path orderly_config.yml return value data argument auxiliary data added validation.","code":"db_config <- function(data, filename) {   data }"},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"evaluate-the-query","dir":"Articles","previous_headings":"An example","what":"Evaluate the query","title":"Creating plugins","text":"Finally, minimal example, need function actually query; example example.db::query: arguments whatever want user provide – nothing special orderly2. important function call orderly2::orderly_plugin_context returns information can use make plugin work. explained ?orderly2::orderly_plugin_context, example use just one element, config, configuration plugin (.e., return value function db_config); see orderly2::orderly_plugin_context context can accessed . last bit package code register plugin, calling orderly2::orderly_plugin_register within .onLoad() special R function called package loaded. means whenever packages loaded (regardless whether attached) register plugin. (important name argument matches package name, orderly2 trigger loading package based name configuration; may support multiple plugins within one package later.) Note query function appear within registration, just function read process configuration. final (minimal) package code : NAMESPACE file contains","code":"query <- function(sql) {   ctx <- orderly2::orderly_plugin_context(\"example.db\")   dbname <- ctx$config$path   con <- DBI::dbConnect(RSQLite::SQLite(), dbname)   on.exit(DBI::dbDisconnect(con))   DBI::dbGetQuery(con, sql) } .onLoad <- function(...) {   orderly2::orderly_plugin_register(     name = \"example.db\",     config = db_config) } db_config <- function(data, filename) {   data }  query <- function(sql) {   ctx <- orderly2::orderly_plugin_context(\"example.db\")   dbname <- ctx$config$path   con <- DBI::dbConnect(RSQLite::SQLite(), dbname)   on.exit(DBI::dbDisconnect(con))   DBI::dbGetQuery(con, sql) }  .onLoad <- function(...) {   orderly2::orderly_plugin_register(     name = \"example.db\",     config = db_config) } export(query)"},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"trying-it-out","dir":"Articles","previous_headings":"An example","what":"Trying it out","title":"Creating plugins","text":"order test package, needs loaded. can either installing package using pkgload::load_all() (may find pkgload::load_all(export_all = FALSE) gives reliable experience. Now, can run report:","code":"pkgload::load_all() ## ℹ Loading example.db orderly2::orderly_run(\"example\", root = path_root) ## ℹ Starting packet 'example' `20240710-102624-0adf8529` at 2024-07-10 10:26:24.046976 ## > dat <- example.db::query(\"SELECT * FROM mtcars WHERE cyl == 4\") ## > orderly2::orderly_artefact(\"Summary of data\", \"data.rds\") ## > saveRDS(summary(dat), \"data.rds\") ## ✔ Finished running example.R ## ℹ Finished 20240710-102624-0adf8529 at 2024-07-10 10:26:24.226441 (0.1794653 secs) ## [1] \"20240710-102624-0adf8529\""},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"making-the-plugin-more-robust","dir":"Articles","previous_headings":"","what":"Making the plugin more robust","title":"Creating plugins","text":"plugin fairly fragile validation input data orderly_config.yml orderly.yml. fairly annoying yaml incredibly flexible reporting back information user might gone wrong hard. case, expect single key-value pair orderly_config.yml key path value path SQLite database. can easily expand configuration function report better back user misconfigure plugin: acceptable job preventing poor input suggesting user might look within configuration fix . Note return configuration data , can augment (otherwise change) data need.","code":"db_config <- function(data, filename) {   if (!is.list(data) || is.null(names(data)) || length(data) == 0) {     stop(\"Expected a named list for orderly_config.yml:example.db\")   }   if (length(data$path) != 1 || !is.character(data$path)) {     stop(\"Expected a string for orderly_config.yml:example.db:path\")   }   if (!file.exists(data$path)) {     stop(sprintf(       \"The database '%s' does not exist (orderly_config:example.db:path)\",       data$path))   }   data }"},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"saving-metadata-about-what-the-plugin-did","dir":"Articles","previous_headings":"","what":"Saving metadata about what the plugin did","title":"Creating plugins","text":"Nothing plugin saved report metadata unless save . Partly orderly.yml, saved final directory, serves sort record. However, probably want know something data returned . example might want save query string later can query without read process orderly.yml file statistics size data (e.g., number rows returned, columns) perhaps summary content hash can see content changed different versions report save metadata, use function orderly2::orderly_plugin_add_metadata; takes arguments plugin name, string like structure saved metadata (’ll use query) whatever data want save: function otherwise minimal version . also need provide serialisation function ensure metadata saved expected. saved metadata key query, get list back element query unnamed list many elements query calls given report. , ensure everything except cols length 1 (everything) gets turned scalar (1 [1]) serialise jsonlite::toJSON auto_unbox TRUE. Taking step , can also specify schema metadata conform save file inst/schema.json within package (path within inst fine). Finally, can also add deserialiation hook convert loaded metadata nice data.frame: Now, register plugin, provide path schema, along serialisation deserialisation functions: Now, orderly metadata saved (just running script part report) validate output passed orderly2::orderly_plugin_add_metadata schema, jsonvalidate installed (currently requires development version) R option outpack.schema_validate set TRUE (e.g., running options(outpack.schema_validate = TRUE)). final package structure: DESCRIPTION file NAMESPACE unchanged , schema shown just . plugin.R file contains code collected : (code .R file package, across several).","code":"query <- function(sql) {   ctx <- orderly2::orderly_plugin_context(\"example.db\")   dbname <- ctx$config$path   con <- DBI::dbConnect(RSQLite::SQLite(), dbname)   on.exit(DBI::dbDisconnect(con))   d <- DBI::dbGetQuery(con, sql)   info <- list(sql = sql, rows = nrow(d), cols = names(d))   orderly2::orderly_plugin_add_metadata(\"example.db\", \"query\", info)   d } db_serialise <- function(data) {   for (i in seq_along(data$query)) {     # Always save cols as a vector, even if length 1:     data$query[[i]]$cols <- I(data$query[[i]]$cols)   }   jsonlite::toJSON(data$query, auto_unbox = TRUE) } {     \"$schema\": \"http://json-schema.org/draft-07/schema#\",      \"type\": \"array\",     \"items\": {         \"type\": \"object\",         \"properties\": {             \"sql\": {                 \"type\": \"string\"             },             \"rows\": {                 \"type\": \"number\"             },             \"cols\": {                 \"type\": \"array\",                 \"items\": {                     \"type\": \"string\"                 }             }         },         \"required\": [\"sql\", \"rows\", \"cols\"],         \"additionalProperties\": false     } } .onLoad <- function(...) {   orderly2::orderly_plugin_register(     name = \"example.db\",     config = db_config,     serialise = db_serialise,     deserialise = db_deserialise,     schema = \"schema.json\") } ## . ## ├── archive ## │   └── example ## │       └── 20240710-102624-0adf8529 ## │           ├── data.rds ## │           └── example.R ## ├── draft ## │   └── example ## ├── orderly_config.yml ## └── src ##     └── example ##         └── example.R db_config <- function(data, filename) {   if (!is.list(data) || is.null(names(data)) || length(data) == 0) {     stop(\"Expected a named list for orderly_config.yml:example.db\")   }   if (length(data$path) != 1 || !is.character(data$path)) {     stop(\"Expected a string for orderly_config.yml:example.db:path\")   }   if (!file.exists(data$path)) {     stop(sprintf(       \"The database '%s' does not exist (orderly_config:example.db:path)\",       data$path))   }   data }  query <- function(sql) {   ctx <- orderly2::orderly_plugin_context(\"example.db\")   dbname <- ctx$config$path   con <- DBI::dbConnect(RSQLite::SQLite(), dbname)   on.exit(DBI::dbDisconnect(con))   d <- DBI::dbGetQuery(con, sql)   info <- list(sql = sql, rows = nrow(d), cols = names(d))   orderly2::orderly_plugin_add_metadata(\"example.db\", \"query\", info)   d }  .onLoad <- function(...) {   orderly2::orderly_plugin_register(     name = \"example.db\",     config = db_config,     serialise = db_serialise,     deserialise = db_deserialise,     schema = \"schema.json\") } id <- orderly2::orderly_run(\"example\", root = path_root) ## ℹ Starting packet 'example' `20240710-102625-1d2e2744` at 2024-07-10 10:26:25.117168 ## > dat <- example.db::query(\"SELECT * FROM mtcars WHERE cyl == 4\") ## > orderly2::orderly_artefact(\"Summary of data\", \"data.rds\") ## > saveRDS(summary(dat), \"data.rds\") ## ✔ Finished running example.R ## ℹ Finished 20240710-102625-1d2e2744 at 2024-07-10 10:26:25.160065 (0.04289699 secs) meta <- orderly2::orderly_metadata(id, root = path_root) meta$custom$example.db ##                                   sql rows         cols ## 1 SELECT * FROM mtcars WHERE cyl == 4   11 mpg, cyl...."},{"path":"https://mrc-ide.github.io/orderly2/articles/plugins.html","id":"potential-uses","dir":"Articles","previous_headings":"","what":"Potential uses","title":"Creating plugins","text":"need functionality similar example - pulling database functionality original version orderly something independent, turns useful fraction orderly use-cases. can imagine potential uses though, : Non-DBI-based database data extraction, customised routines pulling data database Download files shared location just use (e.g., SharePoint, OneDrive, AWS). orderly_config.yml contain account connection details orderly.yml contain mapping remote data/files local files. Rather writing environment , use path argument copy files correct place. Pull data web API just running follow basic pattern requiring configuration order able connect resource service, specification resources fetched, action actually fetch resource put place.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"structure-of-queries","dir":"Articles","previous_headings":"","what":"Structure of queries","title":"Orderly query DSL","text":"simple query finds recent packet; unlikely useful without scoping - see . complex queries expressed syntax valid R (also valid Julia close valid Python). complex query composed “tests” Every “test” uses boolean operator (<, >, <=, >=, ==, !=) left right hand side can one : lookup outpack metadata (parameter:x value parameter called x, name name packet, id id packet) lookup provided data pars (:x value pars$x) literal value (e.g., \"some_name\", 1, TRUE) Tests can grouped together (, !, &&, || might expect: parameter:x == 1 || parameter:x == 2 finds packets parameter x 1 2 name == \"data\" && parameter:x > 3 finds packets called “data” parameter x greater 3 (parameter:y == 2) && !(parameter:x == 1 || parameter:x == 2) finds parameter y 2 parameter x anything 1 2 (also written (parameter:y == 2) && (parameter:x != 1 && parameter:x != 2)) four functions latest(expr) finds latest packet satisfying expr - always returns length 1 character, NA_character_ suitable packet found. expr given latest packets returned. single(expr) like latest(expr) except error expr evaluate exactly one packet id usedby(expr, FALSE) expr either literal id expression returns 1 id. finds packets used generation packet id returned expr (see dependencies section details).","code":"latest() name == \"some_name\" parameter:x > 1"},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"special-simple-queries","dir":"Articles","previous_headings":"","what":"Special simple queries","title":"Orderly query DSL","text":"two shorthand queries: latest equivalent latest() (useful applied scope) string matching regular expression id (^([0-9]{8}-[0-9]{6}-[[:xdigit:]]{8})$) equivalent single(id == \"<id>\") \"<id>\" string provided)","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"scoping-queries","dir":"Articles","previous_headings":"","what":"Scoping queries","title":"Orderly query DSL","text":"WARNING: may remove Scoping queries can used shorthand filtering returned packets. future used reduce set packets searched speed query evaluation. join together main query (scope) && (expr), except expr call latest single. case combine latest((scope) && (expr)) single((scope) && (expr)). useful want limit search particular name location perform detailed search. example, query equivalent orderly2 uses functionality resolving dependencies orderly2::orderly_dependency.","code":"orderly_query(quote(parameter:x == 1), scope = quote(name == \"data\")) orderly_query(quote(parameter:x == 1 && name == \"data\"))"},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"scoping-on-name","dir":"Articles","previous_headings":"Scoping queries","what":"Scoping on name","title":"Orderly query DSL","text":"often users want scope name instead passing scope argument shorthand name argument usability. equivalent ","code":"orderly_query(quote(parameter:x == 1), name = \"data\") orderly_query(quote(parameter:x == 1), scope = quote(name == \"data\"))"},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"dependencies","dir":"Articles","previous_headings":"","what":"Dependencies","title":"Orderly query DSL","text":"2 packets, B depends output (.e. call (id_a, ...) running packet B) can draw .  equivalently say dependency B, B depends used B , B uses parent B, B child tree dependencies among packets might want search packets used another packet. can use query function usedby(id) list packets used id. search recursively packets used id parents parents’ parents . optional second arg immediate FALSE default, set TRUE search immediate (e.g. level 1) dependencies. able search dependencies like means packet structure like  want know id used C can find using orderly_search usedby can combined groupings scope: depth usedby recurse can controlled setting depth e.g. search just immediate parents C. depth can positive integer, default depth recurse finds parents. usedby can simplified using subqueries. Subqueries denoted curly braces {} can either named passed subquery arg can anonymous. query equivalent uses subquery C. two important things note usedby: query inside usedby search entire index, ignoring scope name parameters. want find packets used latest C. subquery C scoped return results. query inside usedby must return single result. ensure must either literal id, call latest call single well searching dependency tree using usedby can search uses function. setup reports , B C want know id C uses can find using uses usedby can combined search complex arrangements dependencies. something like  want search version E depends version used latest C can via searches tree C tree find version E. Note important added name == \"\" condition , missing usedby(latest(name == \"C\")) also return B single throw error multiple packets. can also search tree find D e.g. note E packet uses D need add name == \"E\" clause. can combine usedby uses complex searches, find D C run","code":"orderly_search(quote(usedby(latest(name == \"C\"))), name = \"A\") orderly_search(quote(usedby(latest(name == \"C\")) && parameter:year == 2022),                name = \"A\") orderly_search(quote(usedby(latest(name == \"C\"), depth = 1)), name = \"A\") orderly_search(quote(usedby({C}) && parameter:year == 2022),                name = \"A\",                subquery = list(C = quote(latest(name == \"C\")))) orderly_search(quote(uses(latest(name == \"A\"))),                name = \"C\") orderly_search(   quote(latest(uses(single(usedby(latest(name == \"C\")) && name == \"A\")))),   name = \"E\") orderly_query(   quote(usedby(single(uses(name == \"D\")))),   name = \"A\") orderly_query(   quote(usedby(single(uses(single(usedby(latest(name == \"C\")) && name == \"A\"))) && name == \"E\"))),   name = \"D\")"},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"simple-things","dir":"Articles","previous_headings":"Possible future queries and interface improvements","what":"Simple things","title":"Orderly query DSL","text":"orderly1 supports .null(parameter:x) might generalise support However, Python None Julia nothing, complicates things. Alternatively use missing(parameter:x)?","code":"parameter:x == NULL"},{"path":"https://mrc-ide.github.io/orderly2/articles/query.html","id":"explain-the-query","dir":"Articles","previous_headings":"Possible future queries and interface improvements","what":"Explain the query","title":"Orderly query DSL","text":"Often, people want know “packet match”? good show query set packets fail query excluded. definitely bit work.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/troubleshooting.html","id":"outpack-files-accidentally-committed-to-git","dir":"Articles","previous_headings":"","what":"Outpack files accidentally committed to git","title":"Troubleshooting","text":"discussed orderly introduction, want commit files .outpack/, drafts/ archive/ (used) git create sorts problems line. directed , probably ended files git want undo situation. least painful way depends situation. now put guard rails try prevent happening, still happen modify .gitignore file force-add files example. situation, orderly2 shout : may directed page. just want continue working anyway, run suggested command: things work warning first time session: subsequent calls display warning: rest section discusses might permanently fix issue.","code":"orderly2::orderly_run(\"data\") ## Error in `orderly2::orderly_run()`: ## ! Detected 6 outpack files committed to git ## ✖ Detected files were found in '.outpack/' and 'archive/' ## ℹ For tips on resolving this, please see ##   <https://mrc-ide.github.io/orderly2/articles/troubleshooting.html> ## ℹ To turn this into a warning and continue anyway set the option ##   'orderly_git_error_is_warning' to TRUE by running ##   options(orderly_git_error_is_warning = TRUE) options(orderly_git_error_is_warning = TRUE) orderly2::orderly_run(\"data\") ## Warning in orderly2::orderly_run(\"data\"): Detected 6 outpack files committed to git ## ✖ Detected files were found in '.outpack/' and 'archive/' ## ℹ For tips on resolving this, please see ##   <https://mrc-ide.github.io/orderly2/articles/troubleshooting.html> ## This warning is displayed once per session. ## ✔ Wrote '.gitignore' ## ℹ Starting packet 'data' `20240710-102631-d2ef2801` at 2024-07-10 10:26:31.827093 ## > orderly2::orderly_artefact(\"Final data\", \"data.rds\") ## > saveRDS(mtcars, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240710-102631-d2ef2801 at 2024-07-10 10:26:31.863847 (0.03675437 secs) ## [1] \"20240710-102631-d2ef2801\" orderly2::orderly_run(\"data\") ## ℹ Starting packet 'data' `20240710-102631-f313e685` at 2024-07-10 10:26:31.95268 ## > orderly2::orderly_artefact(\"Final data\", \"data.rds\") ## > saveRDS(mtcars, \"data.rds\") ## ✔ Finished running data.R ## ℹ Finished 20240710-102631-f313e685 at 2024-07-10 10:26:31.97453 (0.02185059 secs) ## [1] \"20240710-102631-f313e685\""},{"path":"https://mrc-ide.github.io/orderly2/articles/troubleshooting.html","id":"i-dont-care-about-my-history-at-all","dir":"Articles","previous_headings":"Outpack files accidentally committed to git","what":"I don’t care about my history at all","title":"Troubleshooting","text":"case just started project, yet collaborating anyone else (person willing re-clone sources). simplest thing : Delete .git directory entirely Run orderly2::orderly_gitignore_update(\"(root)\") set reasonable .gitignore prevent situation happening Run git add . add everything back (review git status make sure ’re happy) Run git commit -m \"Initial commit\" create single commit contains files currently repo history, also .outpack files previously pushed repo GitHub similar need set git remote add origin https://github.com/user/repo (replacing user/repo path, using git@github.com:user/repo use ssh talk GitHub) git branch -M main assuming using main default branch, now common git push --force -u origin main Note destructive require coordination collaborators changed history.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/troubleshooting.html","id":"i-just-want-this-to-go-away-and-nothing-i-have-committed-is-very-large","dir":"Articles","previous_headings":"Outpack files accidentally committed to git","what":"I just want this to go away and nothing I have committed is very large","title":"Troubleshooting","text":"care history, also committed files (e.g., committed files .outpack/ small large 100MB file archive/ preventing pushing GitHub) just delete offending files git without updating history, affecting local copies. Run git rm --cached .outpack (repeating draft archive needed) Run orderly2::orderly_gitignore_update(\"(root)\") set reasonable .gitignore prevent situation happening Run git add .gitignore also stage Run git commit -m \"Delete unwanted outpack files\" can push without issues.","code":""},{"path":"https://mrc-ide.github.io/orderly2/articles/troubleshooting.html","id":"i-care-about-my-history-but-want-this-stuff-gone","dir":"Articles","previous_headings":"Outpack files accidentally committed to git","what":"I care about my history but want this stuff gone","title":"Troubleshooting","text":"working branch, unwanted files committed branch, simplest thing copy changes made somewhere safe, create new branch current main copy changes . somewhat automatically generating applying patch: unwanted files committed onto default branch, potentially gory history rewriting. See StackOverflow question, git docs currently recommended tool . Good luck!","code":"git diff -- src > changes.patch git checkout main git checkout -b changes-attempt2 git apply changes.patch git push -u origin changes-attempt2"},{"path":"https://mrc-ide.github.io/orderly2/authors.html","id":null,"dir":"","previous_headings":"","what":"Authors","title":"Authors and Citation","text":"Rich FitzJohn. Author, maintainer. Robert Ashton. Author. Alex Hill. Author. Imperial College Science, Technology Medicine. Copyright holder.","code":""},{"path":"https://mrc-ide.github.io/orderly2/authors.html","id":"citation","dir":"","previous_headings":"","what":"Citation","title":"Authors and Citation","text":"FitzJohn R, Ashton R, Hill (2024). orderly2: Orderly Next Generation. R package version 1.99.21, https://github.com/mrc-ide/orderly2.","code":"@Manual{,   title = {orderly2: Orderly Next Generation},   author = {Rich FitzJohn and Robert Ashton and Alex Hill},   year = {2024},   note = {R package version 1.99.21},   url = {https://github.com/mrc-ide/orderly2}, }"},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"orderly2","dir":"","previous_headings":"","what":"Orderly Next Generation","title":"Orderly Next Generation","text":"attendant hospital responsible non-medical care patients maintenance order cleanliness. soldier carries orders performs minor tasks officer. orderly2 package designed help make analysis reproducible. principal aim automate series basic steps process writing analyses, making easy : track inputs analysis (packages, code, data resources) store multiple versions analysis repeated track outputs analysis create analyses depend outputs previous analyses","code":""},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"get-started","dir":"","previous_headings":"","what":"Get started","title":"Orderly Next Generation","text":"Two vignettes provide overview package, depending tastes: vignette(\"orderly2\") describes problem orderly2 tries solve (read package website) vignette(\"introduction\") describes orderly2 example, walking basic features (read package website)","code":""},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"roadmap","dir":"","previous_headings":"","what":"Roadmap","title":"Orderly Next Generation","text":"package ground-rewrite orderly package, now ready use new projects adventurous users original package. next months orderly renamed orderly1, finalise API package. stable, rename back orderly2 orderly release CRAN replace existing package. migration path existing users described vignette(\"migrating\") updated based experience migrating people. web framework OrderlyWeb rewritten parallel (packit) provide migration path .","code":""},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"installation","dir":"","previous_headings":"","what":"Installation","title":"Orderly Next Generation","text":"Please install r-universe: prefer, can install GitHub remotes:","code":"install.packages(   \"orderly2\",   repos = c(\"https://mrc-ide.r-universe.dev\", \"https://cloud.r-project.org\")) remotes::install_github(\"mrc-ide/orderly2\")"},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"testing","dir":"","previous_headings":"","what":"Testing","title":"Orderly Next Generation","text":"run tests, need outpack_server available system path. One way achieve run tests also require development versions gert jsonvalidate. can installed rOpenSci r-universe, : Without , tests skipped, suite still run.","code":"cargo install --git https://github.com/mrc-ide/outpack_server install.packages(   c(\"gert\", \"jsonvalidate\"),   repos = c(\"https://ropensci.r-universe.dev\", \"https://cloud.r-project.org\"))"},{"path":"https://mrc-ide.github.io/orderly2/index.html","id":"licence","dir":"","previous_headings":"","what":"Licence","title":"Orderly Next Generation","text":"MIT © Imperial College Science, Technology Medicine","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_artefact.html","id":null,"dir":"Reference","previous_headings":"","what":"Declare orderly artefacts — orderly_artefact","title":"Declare orderly artefacts — orderly_artefact","text":"Declare artefact. turn number orderly features; see Details . can multiple calls function within orderly script.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_artefact.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Declare orderly artefacts — orderly_artefact","text":"","code":"orderly_artefact(description, files)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_artefact.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Declare orderly artefacts — orderly_artefact","text":"description name artefact files files within artefact","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_artefact.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Declare orderly artefacts — orderly_artefact","text":"Undefined","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_artefact.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Declare orderly artefacts — orderly_artefact","text":"(1) files matching copied src directory draft directory unless also listed resource orderly_resource(). feature enabled call function top level orderly script contains string literals (variables). (2) script fails produce files, orderly_run() fail, guaranteeing task really produce things need . (3) within final metadata, artefacts additional metadata; description provide grouping","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":null,"dir":"Reference","previous_headings":"","what":"Clean up source directory — orderly_cleanup","title":"Clean up source directory — orderly_cleanup","text":"Find, delete, file generated running report. comfortable , strongly recommended run orderly_cleanup_status first see deleted.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Clean up source directory — orderly_cleanup","text":"","code":"orderly_cleanup(name = NULL, dry_run = FALSE, root = NULL, locate = TRUE)  orderly_cleanup_status(name = NULL, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Clean up source directory — orderly_cleanup","text":"name Name report directory clean (.e., look src/<name> relative orderly root dry_run Logical, indicating delete anything, instead just print information root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, just outpack (see orderly_init details). locate Logical, indicating configuration searched .  TRUE config given, orderly looks working directory parents finds .outpack directory","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Clean up source directory — orderly_cleanup","text":"(currently unstable) object class orderly_cleanup_status within element delete indicates files deleted (orderly_cleanup_status) deleted (orderly_cleanup)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Clean up source directory — orderly_cleanup","text":"file deletion, look remove empty directories; orderly2 similar semantics git directories never directly tracked. recent gert (yet CRAN) ask git files ignored; ignored good candidates deletion! encourage keep per-report .gitignore lists files copy source directory, can use information clean files generation. Importantly, even file matches ignore rule committed repository, longer match ignore rule.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"notes-for-user-of-orderly-","dir":"Reference","previous_headings":"","what":"Notes for user of orderly1","title":"Clean up source directory — orderly_cleanup","text":"orderly1 function quite different semantics, full set possible files always knowable yaml file. , start point view list files compare directory.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_cleanup.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Clean up source directory — orderly_cleanup","text":"","code":"# Create a simple example: path <- orderly2::orderly_example(\"default\") #> ✔ Created orderly root at '/tmp/RtmpO3RBoN/file17206a1e0276'  # We simulate running a packet interactively by using 'source'; # you might have run this line-by-line, or with the \"Source\" # button in Rstudio. source(file.path(path, \"src/data/data.R\"), chdir = TRUE) #> Warning: cannot open file '/tmp/RtmpO3RBoN/file17206a1e0276/src/data/data.R': No such file or directory #> Error in file(filename, \"r\", encoding = encoding): cannot open the connection  # Having run this, the output of the report is present in the # source directory: fs::dir_tree(path) #> Error: [ENOENT] Failed to search directory '/tmp/RtmpO3RBoN/file17206a1e0276': no such file or directory  # We can detect what might want cleaning up by running # \"orderly_cleanup_status\": orderly2::orderly_cleanup_status(\"data\", root = path) #> Error: Directory does not exist: '/tmp/RtmpO3RBoN/file17206a1e0276'  # Soon this will print more nicely to the screen, but for now you # can see that the status of \"data.rds\" is \"derived\", which means # that orderly knows that it is subject to being cleaned up; the # \"delete\" element shows what will be deleted.  # Do the actual deletion: orderly2::orderly_cleanup(\"data\", root = path) #> Error: Directory does not exist: '/tmp/RtmpO3RBoN/file17206a1e0276'"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config.html","id":null,"dir":"Reference","previous_headings":"","what":"Read configuration — orderly_config","title":"Read configuration — orderly_config","text":"Read current orderly configuration, stored within outpack root, along orderly-specific extensions.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Read configuration — orderly_config","text":"","code":"orderly_config(root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Read configuration — orderly_config","text":"root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Read configuration — orderly_config","text":"list configuration options: core: important options outpack store, containing: path_archive: path human-readable packet archive, NULL disabled (set orderly_config_set core.path_archive) use_file_store: Indicates content-addressable file store enabled (core.use_file_store) require_complete_tree: Indicates outpack store requires dependencies fully available (core.require_complete_tree) hash_algorithm: hash algorithm used (currently modifiable) location: Information locations; see orderly_location_add, orderly_location_rename orderly_location_remove interact configuration, orderly_location_list simply list available locations. Returns data.frame columns name, id, priority, type args, args list column. orderly: list orderly-specific configuration; just minimum required version (minimum_orderly_version).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Read configuration — orderly_config","text":"","code":"# A default configuration in a new temporary directory path <- withr::local_tempdir() orderly2::orderly_init(path) #> ✔ Created orderly root at '/tmp/RtmpO3RBoN/file172041aa773c' orderly2::orderly_config(path) #> $core #> $core$path_archive #> [1] \"archive\" #>  #> $core$use_file_store #> [1] FALSE #>  #> $core$require_complete_tree #> [1] FALSE #>  #> $core$hash_algorithm #> [1] \"sha256\" #>  #>  #> $location #>    name  type args #> 1 local local      #>  #> $orderly #> $orderly$minimum_orderly_version #> [1] ‘1.99.0’ #>  #>"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":null,"dir":"Reference","previous_headings":"","what":"Set configuration options — orderly_config_set","title":"Set configuration options — orderly_config_set","text":"Set configuration options. can currently set; expanded time. See Details.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Set configuration options — orderly_config_set","text":"","code":"orderly_config_set(..., options = list(...), root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Set configuration options — orderly_config_set","text":"... Named options set (e.g., pass argument core.require_complete_tree = TRUE) options alternative ..., can pass list named options (e.g., list(core.require_complete_tree = TRUE)).  interface typically easier program . root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Set configuration options — orderly_config_set","text":"Nothing","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Set configuration options — orderly_config_set","text":"Options set order provided.  Currently, setting one option fails, options processed previous ones (rely behaviour, may change). Currently can set: core.require_complete_tree See orderly_init description options.","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_config_set.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Set configuration options — orderly_config_set","text":"","code":"# The default configuration does not include a file store, and # saves output within the \"archive\" directory: path <- withr::local_tempdir() orderly2::orderly_init(path) #> ✔ Created orderly root at '/tmp/RtmpO3RBoN/file17204247565d' fs::dir_tree(path, all = TRUE) #> /tmp/RtmpO3RBoN/file17204247565d #> ├── .outpack #> │   ├── config.json #> │   ├── location #> │   └── metadata #> └── orderly_config.yml  # Change this after the fact: orderly2::orderly_config_set(core.use_file_store = TRUE,                              core.path_archive = NULL,                              root = path) fs::dir_tree(path, all = TRUE) #> /tmp/RtmpO3RBoN/file17204247565d #> ├── .outpack #> │   ├── config.json #> │   ├── files #> │   ├── index #> │   │   └── outpack.rds #> │   ├── location #> │   └── metadata #> └── orderly_config.yml"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_copy_files.html","id":null,"dir":"Reference","previous_headings":"","what":"Copy files from a packet — orderly_copy_files","title":"Copy files from a packet — orderly_copy_files","text":"Copy files packet anywhere. Similar orderly_dependency except used active packet context. can use function pull files outpack root directory outside control outpack, example. Note arguments need must provided name, position, exception id query.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_copy_files.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Copy files from a packet — orderly_copy_files","text":"","code":"orderly_copy_files(   ...,   files,   dest,   overwrite = TRUE,   envir = parent.frame(),   options = NULL,   root = NULL,   locate = TRUE )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_copy_files.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Copy files from a packet — orderly_copy_files","text":"... Arguments passed orderly_query, perhaps just query expression files Files copy packet. can (1) character vector, case files copied without changing names, (2) named character vector, case name used destination name, (3) data.frame (including tbl_df, data.frame objects) containing columns , case files copied names . cases, want import directory files packet, must refer source trailing slash (e.g., c(= \"/\")), create local directory /... files upstream packet directory /. omit slash error thrown suggesting add slash intended. can use limited form string interpolation names argument; using ${variable} pick values envir substitute string.  similar interpolation might familiar glue::glue similar, much simpler concatenation fancy features supported. Note unfortunate, (us) avoidable inconsistency ; interpolation values environment query done using environment:x destination filename ${x}. dest directory copy overwrite Overwrite files destination; typically want, set FALSE prefer error thrown destination file already exists. envir Optionally, environment substitute query (using environment: prefix). default use calling environment, can explicitly pass want control lookup happens. options Optionally, orderly_search_options object controlling search performed, packets considered scope. provided, default options used (.e., orderly2::orderly_search_options()) root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_copy_files.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Copy files from a packet — orderly_copy_files","text":"Nothing, invisibly. Primarily called side effect copying files packet directory dest","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_copy_files.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Copy files from a packet — orderly_copy_files","text":"can call function id string, case search packet proceed regardless whether id present.  called arguments (e.g., string match id format, named argument name, subquery parameters) interpret arguments query orderly_search find id. error query return exactly one packet id, probably want use latest(). different ways might fail (recover failure): id known metadata store (known unpacked also known present remote) fail impossible resolve files. Consider refreshing metadata orderly_location_pull_metadata refresh . id unpacked local copy files referred can found, error default (see next option). However, sometimes file refer might also present downloaded packet depended , content file unchanged packet version locally. id unpacked, local copy file allow_remote TRUE try request file whatever remote selected orderly_location_pull_packet packet. Note empty directories might created failure.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_dependency.html","id":null,"dir":"Reference","previous_headings":"","what":"Declare a dependency — orderly_dependency","title":"Declare a dependency — orderly_dependency","text":"Declare dependency another packet","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_dependency.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Declare a dependency — orderly_dependency","text":"","code":"orderly_dependency(name, query, files)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_dependency.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Declare a dependency — orderly_dependency","text":"name name packet depend query query search ; often simply string latest, indicating recent version. may want complex query though. files Files copy packet. can (1) character vector, case files copied without changing names, (2) named character vector, case name used destination name, (3) data.frame (including tbl_df, data.frame objects) containing columns , case files copied names . cases, want import directory files packet, must refer source trailing slash (e.g., c(= \"/\")), create local directory /... files upstream packet directory /. omit slash error thrown suggesting add slash intended. can use limited form string interpolation names argument; using ${variable} pick values envir substitute string.  similar interpolation might familiar glue::glue similar, much simpler concatenation fancy features supported. Note unfortunate, (us) avoidable inconsistency ; interpolation values environment query done using environment:x destination filename ${x}.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_dependency.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Declare a dependency — orderly_dependency","text":"Undefined","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_dependency.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Declare a dependency — orderly_dependency","text":"See orderly_run details search options used select locations packets found , data fetched network. running interactively, obviously work, use orderly_interactive_set_search_options() set options function respond .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_description.html","id":null,"dir":"Reference","previous_headings":"","what":"Describe the current packet — orderly_description","title":"Describe the current packet — orderly_description","text":"Describe current packet","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_description.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Describe the current packet — orderly_description","text":"","code":"orderly_description(display = NULL, long = NULL, custom = NULL)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_description.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Describe the current packet — orderly_description","text":"display friendly name report; displayed locations web interface, packit. given, must scalar character. long longer description report. given, must scalar character. custom additional metadata. given, must named list, elements scalar atomics (character, number, logical).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_description.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Describe the current packet — orderly_description","text":"Undefined","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_example.html","id":null,"dir":"Reference","previous_headings":"","what":"Copy a simple orderly example — orderly_example","title":"Copy a simple orderly example — orderly_example","text":"Copy simple orderly example use docs. function form part workflow!","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_example.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Copy a simple orderly example — orderly_example","text":"","code":"orderly_example(name, ..., dest = NULL)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_example.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Copy a simple orderly example — orderly_example","text":"name name example copy. Currently \"default\" supported. ... Arguments passed orderly_init() dest destination. default use withr::local_tempfile() create temporary directory clean . suitable use orderly examples, may prefer provide path. path must already exist.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_example.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Copy a simple orderly example — orderly_example","text":"Invisibly, path example.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_example.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Copy a simple orderly example — orderly_example","text":"","code":"path <- orderly2::orderly_example(\"default\") #> ✔ Created orderly root at '/tmp/RtmpO3RBoN/file172057385d9' orderly2::orderly_list_src(root = path) #> Error: Directory does not exist: '/tmp/RtmpO3RBoN/file172057385d9'"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_gitignore_update.html","id":null,"dir":"Reference","previous_headings":"","what":"Update a gitignore file — orderly_gitignore_update","title":"Update a gitignore file — orderly_gitignore_update","text":"Update gitignore, useful prevent accidentally committing files source control generated. includes artefacts, shared resources dependencies (within report directory) global level contents .outpack directory, draft folder archive directory.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_gitignore_update.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Update a gitignore file — orderly_gitignore_update","text":"","code":"orderly_gitignore_update(name, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_gitignore_update.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Update a gitignore file — orderly_gitignore_update","text":"name name gitignore file update, string \"(root)\" root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, just outpack (see orderly_init details). locate Logical, indicating configuration searched .  TRUE config given, orderly looks working directory parents finds .outpack directory","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_gitignore_update.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Update a gitignore file — orderly_gitignore_update","text":"Nothing, called side effects","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_gitignore_update.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Update a gitignore file — orderly_gitignore_update","text":"function fails message edit '.gitignore', markers corrupted, look special markers within .gitignore file.  look like   edit file : lines appears file anything first two lines order get error message, search remove lines rerun.","code":"# ---VVV--- added by orderly ---VVV---------------- # Don't manually edit content between these markers ... patterns # ---^^^--- added by orderly ---^^^----------------"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_hash.html","id":null,"dir":"Reference","previous_headings":"","what":"Compute a hash — orderly_hash_file","title":"Compute a hash — orderly_hash_file","text":"Use orderly2's hashing functions.  intended advanced users, particular want create hashes consistent orderly2 within plugins. default behaviour use algorithm used orderly root (via root argument, usual root location approach). However, string provided algorithm can use alternative algorithm.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_hash.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Compute a hash — orderly_hash_file","text":"","code":"orderly_hash_file(path, algorithm = NULL, root = NULL, locate = TRUE)  orderly_hash_data(data, algorithm = NULL, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_hash.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Compute a hash — orderly_hash_file","text":"path name file hash algorithm name algorithm use, overriding orderly root. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml data string hash","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_hash.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Compute a hash — orderly_hash_file","text":"string format <algorithm>:<digest>","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_hash.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Compute a hash — orderly_hash_file","text":"","code":"orderly2::orderly_hash_data(\"hello\", \"md5\") #> [1] \"md5:5d41402abc4b2a76b9719d911017c592\""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":null,"dir":"Reference","previous_headings":"","what":"Initialise an orderly repository — orderly_init","title":"Initialise an orderly repository — orderly_init","text":"Initialise empty orderly repository, initialise source copy orderly repository (see Details). orderly repository defined presence file orderly_config.yml root, along directory .outpack/ level.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Initialise an orderly repository — orderly_init","text":"","code":"orderly_init(   root = \".\",   path_archive = \"archive\",   use_file_store = FALSE,   require_complete_tree = FALSE,   force = FALSE )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Initialise an orderly repository — orderly_init","text":"root path initialise repository root .  repository already initialised, operation checks options passed set repository (erroring ), otherwise nothing.  default path current working directory. path_archive Path archive directory, used store human-readable copies packets.  NULL, copy made, file_store must TRUE use_file_store Logical, indicating use content-addressable file-store source truth packets.  archive non-NULL, file-store used source truth duplicated files archive exist convenience. require_complete_tree Logical, indicating require complete tree packets.  currently affects orderly_location_pull_packet, requiring always operates recursive mode.  FALSE default, set TRUE want archive behave well location; TRUE always packets hold metadata . force Logical, indicating shold initialise orderly even directory empty.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Initialise an orderly repository — orderly_init","text":"full, normalised, path root, invisibly. Typically called side effect.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Initialise an orderly repository — orderly_init","text":"expected orderly_config.yml saved version control, .outpack excluded version control; means every clone project need call orderly2::orderly_init() initialise .outpack directory. forget , error thrown reminding need . can safely call orderly2::orderly_init() already-initialised directory, however, arguments passed must exactly match configuration current root, otherwise error thrown. Please use orderly_config_set change configuration, ensures change configuration possible. configuration options given match directory already uses, nothing happens. repository call orderly2::orderly_init() already initialised .outpack directory orderly_config.yml file, write file .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_init.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Initialise an orderly repository — orderly_init","text":"","code":"# We'll use an automatically cleaned-up directory for the root: path <- withr::local_tempdir()  # Initialise a new repository, setting an option: orderly2::orderly_init(path, use_file_store = TRUE) #> ✔ Created orderly root at '/tmp/RtmpO3RBoN/file17204d71c6f8'  # fs::dir_tree(path, all = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_interactive_set_search_options.html","id":null,"dir":"Reference","previous_headings":"","what":"Set search options for interactive use — orderly_interactive_set_search_options","title":"Set search options for interactive use — orderly_interactive_set_search_options","text":"Set search options interactive use orderly; see orderly_dependency orderly_run details. may either orderly_search_options object, list coerced one point use (NULL). applies current session, applies interactive uses orderly functions might received copy search_options via orderly_run","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_interactive_set_search_options.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Set search options for interactive use — orderly_interactive_set_search_options","text":"","code":"orderly_interactive_set_search_options(options = NULL)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_interactive_set_search_options.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Set search options for interactive use — orderly_interactive_set_search_options","text":"options Optional control locations, used orderly_dependency; see Details section orderly_run.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_interactive_set_search_options.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Set search options for interactive use — orderly_interactive_set_search_options","text":"Nothing, called side effects","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_list_src.html","id":null,"dir":"Reference","previous_headings":"","what":"List source reports — orderly_list_src","title":"List source reports — orderly_list_src","text":"List source reports - , directories within src/ look suitable running orderly; directories contain entrypoint file - .R file name directory (e.g., src/data/data.R corresponds data).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_list_src.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"List source reports — orderly_list_src","text":"","code":"orderly_list_src(root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_list_src.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"List source reports — orderly_list_src","text":"root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, just outpack (see orderly_init details). locate Logical, indicating configuration searched .  TRUE config given, orderly looks working directory parents finds .outpack directory","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_list_src.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"List source reports — orderly_list_src","text":"character vector names source reports, suitable passing orderly_run","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_list_src.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"List source reports — orderly_list_src","text":"","code":"path <- orderly2::orderly_example(\"default\") #> ✔ Created orderly root at '/tmp/RtmpO3RBoN/file172037dab125' orderly2::orderly_list_src(root = path) #> Error: Directory does not exist: '/tmp/RtmpO3RBoN/file172037dab125'"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":null,"dir":"Reference","previous_headings":"","what":"Add a new location — orderly_location_add","title":"Add a new location — orderly_location_add","text":"Add new location - place packets might found pulled local archive.  Currently file http based locations supported, limited support custom locations. Note adding location pull metadata , need call orderly_location_pull_metadata first.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Add a new location — orderly_location_add","text":"","code":"orderly_location_add(name, type, args, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Add a new location — orderly_location_add","text":"name short name location use.  use, one local orphan type type location add. Currently supported values path (location exists elsewhere filesystem) http (location accessed outpack's http API). args Arguments location driver. arguments vary depending type used, see Details. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Add a new location — orderly_location_add","text":"Nothing","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Add a new location — orderly_location_add","text":"currently support two types locations - path, points outpack archive accessible path (e.g., computer mounted network share), http, requires outpack server running url uses HTTP API communicate, packit, uses Packit web server.  types may added later, configuration options location types definitely needed future. Configuration options different location types: Path locations: path: path archive root. generally absolute path, behaviour outpack unreliable. HTTP locations: Accessing outpack HTTP requires outpack server running. interface expected change expand API, also move support things like TLS authentication. url: location server, including protocol, example http://example.com:8080 Packit locations: Packit locations work HTTPS, include everything outpack location also provide authentication later capabilities think. url: location server token: value login token (currently GitHub token read:org scope).  Later expand authentication modes supported. Custom locations: outpack implementations expected support path http locations, standard arguments .  expect implementations support custom locations, argument lists may vary implementations. allow , can pass location type \"custom\" list arguments.  expect argument 'driver' present among list.  example action, see outpack.sharepoint package. warned may change interface future, case may need update configuration.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_add.html","id":"warning","dir":"Reference","previous_headings":"","what":"Warning","title":"Add a new location — orderly_location_add","text":"API may change move support different types locations.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_list.html","id":null,"dir":"Reference","previous_headings":"","what":"List known pack locations — orderly_location_list","title":"List known pack locations — orderly_location_list","text":"List known locations.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_list.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"List known pack locations — orderly_location_list","text":"","code":"orderly_location_list(root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_list.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"List known pack locations — orderly_location_list","text":"root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_list.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"List known pack locations — orderly_location_list","text":"character vector location names. special name local always present.","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_metadata.html","id":null,"dir":"Reference","previous_headings":"","what":"Pull metadata from a location — orderly_location_pull_metadata","title":"Pull metadata from a location — orderly_location_pull_metadata","text":"Pull metadata location, updating index.  always relatively quick updates small files contain information can found remote packets.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_metadata.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Pull metadata from a location — orderly_location_pull_metadata","text":"","code":"orderly_location_pull_metadata(location = NULL, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_metadata.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Pull metadata from a location — orderly_location_pull_metadata","text":"location name location pull (see orderly_location_list possible values).  given, pulls locations.  \"local\" \"orphan\" locations always date pulling metadata nothing. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_metadata.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Pull metadata from a location — orderly_location_pull_metadata","text":"Nothing","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_packet.html","id":null,"dir":"Reference","previous_headings":"","what":"Pull one or more packets from a location — orderly_location_pull_packet","title":"Pull one or more packets from a location — orderly_location_pull_packet","text":"Pull one packets (including files) archive one locations. make files available use dependencies (e.g., orderly_dependency).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_packet.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Pull one or more packets from a location — orderly_location_pull_packet","text":"","code":"orderly_location_pull_packet(   ...,   options = NULL,   recursive = NULL,   root = NULL,   locate = TRUE )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_packet.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Pull one or more packets from a location — orderly_location_pull_packet","text":"... Arguments passed orderly_search. special case first argument character vector ids named dot arguments, interpret argument vector ids directly. careful , query may pull lot data particular, passing NULL match everything every remote ! options Options passed orderly_search. option allow_remote must TRUE otherwise packet possibly pulled, error thrown FALSE. recursive non-NULL, logical, indicating recursively pull packets referenced packets specified id.  might copy lot data!  NULL, default value given configuration option require_complete_tree. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_packet.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Pull one or more packets from a location — orderly_location_pull_packet","text":"Invisibly, ids packets pulled","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_pull_packet.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Pull one or more packets from a location — orderly_location_pull_packet","text":"possible take long time pull packets, moving lot data operating slow connection.  Cancelling resuming pull fairly efficient, keep track files copied even case interrupted pull.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_push.html","id":null,"dir":"Reference","previous_headings":"","what":"Push tree to location — orderly_location_push","title":"Push tree to location — orderly_location_push","text":"Push tree location. function works packets known location files required create . pushes files required build packets pushes missing metadata server. process interrupted safe resume transfer files packets missed previous call.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_push.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Push tree to location — orderly_location_push","text":"","code":"orderly_location_push(packet_id, location, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_push.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Push tree to location — orderly_location_push","text":"packet_id One packets push server location name location push (see orderly_location_list possible values). root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_push.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Push tree to location — orderly_location_push","text":"Invisibly, details information actually moved (might less requested, depending dependencies packets already known location).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_remove.html","id":null,"dir":"Reference","previous_headings":"","what":"Remove a location — orderly_location_remove","title":"Remove a location — orderly_location_remove","text":"Remove existing location. packets location known elsewhere now associated 'orphan' location instead.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_remove.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Remove a location — orderly_location_remove","text":"","code":"orderly_location_remove(name, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_remove.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Remove a location — orderly_location_remove","text":"name short name location. remove local orphan root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_remove.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Remove a location — orderly_location_remove","text":"Nothing","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_rename.html","id":null,"dir":"Reference","previous_headings":"","what":"Rename a location — orderly_location_rename","title":"Rename a location — orderly_location_rename","text":"Rename existing location","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_rename.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Rename a location — orderly_location_rename","text":"","code":"orderly_location_rename(old, new, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_rename.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Rename a location — orderly_location_rename","text":"old current short name location. rename local orphan new desired short name location. one local orphan root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_location_rename.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Rename a location — orderly_location_rename","text":"Nothing","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata.html","id":null,"dir":"Reference","previous_headings":"","what":"Read outpack metadata — orderly_metadata","title":"Read outpack metadata — orderly_metadata","text":"Read metadata particular id. may want use orderly_search find id corresponding particular query.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Read outpack metadata — orderly_metadata","text":"","code":"orderly_metadata(id, root = NULL, locate = FALSE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Read outpack metadata — orderly_metadata","text":"id id fetch metadata . error thrown id known root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Read outpack metadata — orderly_metadata","text":"list metadata. See outpack schema details (https://github.com/mrc-ide/outpack)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":null,"dir":"Reference","previous_headings":"","what":"Extract metadata from orderly2 packets — orderly_metadata_extract","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"Extract metadata group packets.  experimental high-level function interacting metadata way hope useful. expand bit time goes , based feedback get let us know think.  See Details use .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"","code":"orderly_metadata_extract(..., extract = NULL, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"... Arguments passed orderly_search. special case first argument character vector ids named dot arguments, interpret argument vector ids directly. extract character vector columns extract, possibly named. See Details format. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"data.frame, columns vary based names extract; see Details information.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"Extracting data outpack metadata challenging way works data structures familiar R users, naturally tree structured, metadata may present packets (e.g., packet depend another dependency section, one run context without git git metadata). just want raw tree-structured data, can always use orderly_metadata load full metadata packet (even one currently available computer, just known ) structure data remain fairly constant across orderly2 versions. However, sometimes want extract data order ask specific questions like: parameter combinations available across range packets? particular set packets used? files packets produce? Later like ask even complex questions like: version file graph.png change? inputs changed versions? ...able answer questions requires similar approach interrogating metadata across range packets. orderly_metadata_extract function aims simplify process pulling bits metadata arranging data.frame (sorts) .  little mini-language extract argument simple rewriting results, can always . order use function need know metadata available; expand vignette worked examples make easier understand. function works top-level keys, : id: packet id (always returned) name: packet name parameters: key-value pair values, string keys atomic values. guarantee presence keys packets, types. time: key-value pair times, string keys time values (see DateTimeClasses; stored seconds since 1970 actual metadata). present start end always present. files: files present packet. data.frame (per packet), columns path (relative), size (bytes) hash. depends: dependencies used packet. data.frame (per packet), columns packet (id), query (string, used find packet) files (another data.frame columns corresponding filenames upstream packet, respectively) git: either metadata state git null. given sha branch strings, url array strings/character vector (can zero, one elements). session: information session packet run (unstandardised, even orderly version may change) custom: additional metadata added respective engine.  packets run orderly2, orderly field , list: artefacts: data.frame artefact information, containing columns description (string) paths (list column paths). shared: data.frame copied shared resources original name () name copied packet (). role: data.frame identified roles files, columns path role. description: list information orderly_description human-readable descriptions tags. session: list information session run, list platform containing information platform (R version version, operating system os system name system) packages containing columns package , version attached. nesting makes providing universally useful data format difficult; considering files data.frame files column, list data.frames; similar nestedness applies depends orderly custom data. However, able fairly easily process data format need . simplest extraction uses names top-level keys:   creates data.frame columns corresponding keys, one row per packet. name always string, character vector, parameters files complex, list columns. must provide id; always returned always first character vector column. can rename columns providing name entries within extract, example:   , except parameters column renamed pars. interestingly, can index structure like parameters; suppose want value parameter x, write:   allowed packet parameters element list. However, know type x (might vary packets). can add information though write:   create numeric column. packet value x non-integer, call orderly_metadata_extract fail error, packet lacks value x, missing value appropriate type added. Note coercion number, error non-NULL non-numeric value found.  Valid types use <type> boolean, number string (note differ slightly R's names want emphasise scalar quantities; also note integer may produce unexpected errors integer-like numeric values). can also use list default.  Things schema known scalar atomics (name) automatically simplified. can index array-valued elements (files depends) way object-valued elements:   get list column file names per packet another hashes, probably less useful data.frame get extracting just files longer hash information aligned. can index fairly deeply; possible get orderly \"display name\" :   path need extract dot (likely package name plugin, custom.orderly.db) need escape dot backslash (, custom.orderly\\.db). probably need two slashes use raw string (recent versions R).","code":"extract = c(\"name\", \"parameters\", \"files\") extract = c(\"name\", pars = \"parameters\", \"files\") extract = c(x = \"parameters.x\") extract = c(x = \"parameters.x is number\") extract = c(file_path = \"files.path\", file_hash = \"files.hash\") extract = c(display = \"custom.orderly.description.display is string\")"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_extract.html","id":"custom-orderly-metadata","dir":"Reference","previous_headings":"","what":"Custom 'orderly' metadata","title":"Extract metadata from orderly2 packets — orderly_metadata_extract","text":"Within custom.orderly, additional fields can extracted. format subject change, stored metadata schema (short term) way deserialise . probably best rely right now, expand section can.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_read.html","id":null,"dir":"Reference","previous_headings":"","what":"Read outpack metadata json file — orderly_metadata_read","title":"Read outpack metadata json file — orderly_metadata_read","text":"Low-level function reading metadata deserialising . function can used directly read metadata json file without reference root contains . may useful context reading metadata file written part failed run.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_read.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Read outpack metadata json file — orderly_metadata_read","text":"","code":"orderly_metadata_read(path, plugins = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_read.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Read outpack metadata json file — orderly_metadata_read","text":"path Path json file plugins Try deserialise data loaded plugins (see Details).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_read.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Read outpack metadata json file — orderly_metadata_read","text":"list outpack metadata; see schema details. contrast reading json file directly jsonlite::fromJSON, function take care convert scalar length-one vectors expected types.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_metadata_read.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Read outpack metadata json file — orderly_metadata_read","text":"Custom metadata saved plugins may deserialised expected called function, designed operate separately valid orderly root (.e., load data file regardless came ). plugins TRUE (default) deserialise data matches loaded plugin.  means behaviour function depends loaded plugin packages. can force running orderly2::orderly_config() within orderly directory, load declared plugins.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_new.html","id":null,"dir":"Reference","previous_headings":"","what":"Create a new report — orderly_new","title":"Create a new report — orderly_new","text":"Create new empty report.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_new.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Create a new report — orderly_new","text":"","code":"orderly_new(name, template = NULL, force = FALSE, root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_new.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Create a new report — orderly_new","text":"name name report template template use. acceptable values now NULL (uses built-default) FALSE suppresses default content.  may support customisable templates future - let us know useful. force Create orderly file - <name>.R within existing directory src/<name>; may useful already created directory files first want help creating orderly file. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, just outpack (see orderly_init details). locate Logical, indicating configuration searched .  TRUE config given, orderly looks working directory parents finds .outpack directory","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_new.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Create a new report — orderly_new","text":"Nothing, called side effects ","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parameters.html","id":null,"dir":"Reference","previous_headings":"","what":"Declare orderly parameters — orderly_parameters","title":"Declare orderly parameters — orderly_parameters","text":"Declare orderly parameters. one call within file, though enforced! Typically put close top, though order really matter. Parameters scalar atomic values (e.g. string, number boolean) defaults must present literally (.e., may come variable ). Provide NULL default, case parameter required.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parameters.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Declare orderly parameters — orderly_parameters","text":"","code":"orderly_parameters(...)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parameters.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Declare orderly parameters — orderly_parameters","text":"... number parameters","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parameters.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Declare orderly parameters — orderly_parameters","text":"Undefined","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parameters.html","id":"behaviour-in-interactive-sessions","dir":"Reference","previous_headings":"","what":"Behaviour in interactive sessions","title":"Declare orderly parameters — orderly_parameters","text":"running interactively (.e., via source() running orderly file session copy/paste Rstudio), orderly_parameters() function different behaviour. First, look current environment (likely global environment) values parameters - , variables bound names parameters.  parameters found look default values use possible, possible either error prompt based global option orderly_interactive_parameters_missing_error. TRUE, ask enter value parameters (strings need entered quotes).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parse_file.html","id":null,"dir":"Reference","previous_headings":"","what":"Parse the orderly entrypoint script — orderly_parse_file","title":"Parse the orderly entrypoint script — orderly_parse_file","text":"expert use .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parse_file.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Parse the orderly entrypoint script — orderly_parse_file","text":"","code":"orderly_parse_file(path)  orderly_parse_expr(exprs, filename)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parse_file.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Parse the orderly entrypoint script — orderly_parse_file","text":"path Path orderly_* script exprs Parsed AST orderly_* script filename Name orderly_* file include metadata","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parse_file.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Parse the orderly entrypoint script — orderly_parse_file","text":"Parsed orderly entrypoint script","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_parse_file.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Parse the orderly entrypoint script — orderly_parse_file","text":"Parses details calls orderly_ -script functions intermediate representation downstream use. Also validates calls orderly_* -script functions well-formed.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_add_metadata.html","id":null,"dir":"Reference","previous_headings":"","what":"Add metadata from plugin — orderly_plugin_add_metadata","title":"Add metadata from plugin — orderly_plugin_add_metadata","text":"Add plugin-specific metadata running packet. take describing. accumulate number bits metadata arbitrary fields, later serialise json.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_add_metadata.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Add metadata from plugin — orderly_plugin_add_metadata","text":"","code":"orderly_plugin_add_metadata(name, field, data)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_add_metadata.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Add metadata from plugin — orderly_plugin_add_metadata","text":"name name plugin; must used orderly_plugin_register orderly_plugin_context field name field add data . required even plugin produces one sort data, case can remove later within serialisation function. data Arbitrary data added currently running packet","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_add_metadata.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Add metadata from plugin — orderly_plugin_add_metadata","text":"Nothing, called side effects","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_context.html","id":null,"dir":"Reference","previous_headings":"","what":"Fetch plugin context — orderly_plugin_context","title":"Fetch plugin context — orderly_plugin_context","text":"Fetch running context, use within plugin.  intention within free functions plugin makes available, call function get information state packet.  typically call orderly_plugin_add_metadata() afterwards.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_context.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Fetch plugin context — orderly_plugin_context","text":"","code":"orderly_plugin_context(name, envir)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_context.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Fetch plugin context — orderly_plugin_context","text":"name Name plugin envir environment calling function. can typically pass parent.frame() (rlang::caller_env()) function calling orderly_plugin_context() function called user. argument effect interactive use (envir almost certainly global environment).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_context.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Fetch plugin context — orderly_plugin_context","text":"list elements: is_active: logical, indicating running orderly_run(); may need change behaviour depending value. path: path running packet. almost always working directory, unless packet contains calls setwd() similar. may create files . config: configuration plugin, processing plugin's read function (see orderly_plugin_register) envir: environment packet running . Often global environment, assume ! may read write environment. src: path packet source directory. different current directory packet running, user interactively working report. may read directory must write parameters: parameters passed run report.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_context.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Fetch plugin context — orderly_plugin_context","text":"plugin function called, orderly2 running one two modes; (1) within orderly_run(), case part way creating packet brand new directory, possibly using special environment evaluation, (2) interactively, user developing report. plugin needs able support modes, function return information state help cope either case.","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_register.html","id":null,"dir":"Reference","previous_headings":"","what":"Register an orderly plugin — orderly_plugin_register","title":"Register an orderly plugin — orderly_plugin_register","text":"Create orderly plugin. plugin typically defined package used extend orderly enabling new functionality, declared orderly_config.yml orderly file, affecting running reports primarily creating new objects report environment.  system discussed detail vignette(\"plugins\"), expanded (likely breaking ways) soon.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_register.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Register an orderly plugin — orderly_plugin_register","text":"","code":"orderly_plugin_register(   name,   config,   serialise = NULL,   deserialise = NULL,   cleanup = NULL,   schema = NULL )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_register.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Register an orderly plugin — orderly_plugin_register","text":"name name plugin, typically package name config function read, check process configuration section orderly_config.yml. function passed deserialised data plugin's section orderly_config.yml, full path file.  order loading plugins defined, plugin must standalone try interact plugins load. return processed copy configuration data, passed second argument read. serialise function serialise metadata added plugin's functions outpack metadata. passed list entries pushed via orderly_plugin_add_metadata(); named list names corresponding field argument orderly_plugin_add_metadata list element unnamed list values corresponding data. NULL, serialisation done, metadata plugin added. deserialise function deserialise metadata serialised serialise function. intended help deal issues disambiguating unserialising objects json (scalars vs arrays lenth 1, data.frames vs lists--lists etc), make plugin nicer work orderly_metadata_extract(). function given single argument data data jsonlite::fromJSON(..., simplifyVector = FALSE) apply required simplifications , returning modified copy argument. cleanup Optionally, function clean state plugin uses. can call orderly_plugin_context within function access anything need . given, cleanup done. schema Optionally path, within package, schema metadata created plugin; omit .json extension.  file contains sources file inst/plugin/myschema.json pass plugin/myschema.  See vignette(\"plugins\") details.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_plugin_register.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Register an orderly plugin — orderly_plugin_register","text":"Nothing, function called side effect registering plugin.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_prune_orphans.html","id":null,"dir":"Reference","previous_headings":"","what":"Prune orphan packet metadata — orderly_prune_orphans","title":"Prune orphan packet metadata — orderly_prune_orphans","text":"Prune orphan packets metadata store.  function can used remove references packets longer reachable; happened deleted packet manually archive ran orderly_validate_archive removed location.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_prune_orphans.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Prune orphan packet metadata — orderly_prune_orphans","text":"","code":"orderly_prune_orphans(root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_prune_orphans.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Prune orphan packet metadata — orderly_prune_orphans","text":"root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_prune_orphans.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Prune orphan packet metadata — orderly_prune_orphans","text":"Invisibly, character vector orphaned packet ids","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_prune_orphans.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Prune orphan packet metadata — orderly_prune_orphans","text":"orphan packet used anywhere, can easily drop - never existed.  referenced metadata know elsewhere locally, problem upstream location (one happen). referenced packet run locally, metadata deleted. expose function mostly users want expunge permanently reference previously run packets. hope never need really reason run .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query.html","id":null,"dir":"Reference","previous_headings":"","what":"Construct outpack query — orderly_query","title":"Construct outpack query — orderly_query","text":"Construct outpack query, typically passed orderly_search","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Construct outpack query — orderly_query","text":"","code":"orderly_query(expr, name = NULL, scope = NULL, subquery = NULL)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Construct outpack query — orderly_query","text":"expr query expression. NULL expression matches everything. name Optionally, name packet scope query . intersected scope arg shorthand way running scope = list(name = \"name\") scope Optionally, scope query limit packets searched pars subquery Optionally, named list subqueries can referenced name expr.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Construct outpack query — orderly_query","text":"orderly_query object, modified, can passed orderly_search()","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query_explain.html","id":null,"dir":"Reference","previous_headings":"","what":"Explain a query — orderly_query_explain","title":"Explain a query — orderly_query_explain","text":"Explain query matched. experimental output change. moment, can tell query matches, fails match based one number &&-ed together clauses.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query_explain.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Explain a query — orderly_query_explain","text":"","code":"orderly_query_explain(   ...,   parameters = NULL,   envir = parent.frame(),   options = NULL,   root = NULL )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query_explain.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Explain a query — orderly_query_explain","text":"... Arguments passed orderly_query, perhaps just query expression parameters Optionally, named list parameters substitute query (using : prefix) envir Optionally, environment substitute query (using environment: prefix). default use calling environment, can explicitly pass want control lookup happens. options Optionally, orderly_search_options object controlling search performed, packets considered scope. provided, default options used (.e., orderly2::orderly_search_options()) root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_query_explain.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Explain a query — orderly_query_explain","text":"object class orderly_query_explain, can inspected (contents subject change) print method show user-friendly summary query result.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_resource.html","id":null,"dir":"Reference","previous_headings":"","what":"Declare orderly resources — orderly_resource","title":"Declare orderly resources — orderly_resource","text":"Declare file, group files, orderly resource. explicitly declaring files resources orderly mark files immutable inputs validate analysis modify run orderly_run()","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_resource.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Declare orderly resources — orderly_resource","text":"","code":"orderly_resource(files)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_resource.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Declare orderly resources — orderly_resource","text":"files number names files","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_resource.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Declare orderly resources — orderly_resource","text":"Invisibly, character vector resources included call. rely order files expanded directories, likely platform dependent.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":null,"dir":"Reference","previous_headings":"","what":"Run a report — orderly_run","title":"Run a report — orderly_run","text":"Run report.  create new directory drafts/<reportname>, copy declared resources , run script check expected artefacts created.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Run a report — orderly_run","text":"","code":"orderly_run(   name,   parameters = NULL,   envir = NULL,   echo = TRUE,   search_options = NULL,   root = NULL,   locate = TRUE )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Run a report — orderly_run","text":"name Name report run. leading ./ src/ trailing / path parts removed (e.g., added autocomplete). parameters Parameters passed report. named list parameters declared orderly.yml.  parameter must scalar character, numeric, integer logical. envir environment used evaluate report script; default use global environment, may always wanted. echo Optional logical control printing output source() console. search_options Optional control locations, used orderly_dependency; converted orderly_search_options object, see Details. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, just outpack (see orderly_init details). locate Logical, indicating configuration searched .  TRUE config given, orderly looks working directory parents finds .outpack directory","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Run a report — orderly_run","text":"id created report (string)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"locations-used-in-dependency-resolution","dir":"Reference","previous_headings":"","what":"Locations used in dependency resolution","title":"Run a report — orderly_run","text":"packet depends packets, want control locations used find appropriate packets. control passed function argument orderly_dependency property way packet created packet ; importantly different users may different names locations makes little sense encode location name source code. Alternatively, want use different locations different contexts (initial development want include local copies packets possible dependencies vs resolving dependencies resolved one locations! Similarly, might want include packets known locations currently downloaded onto machine - pulling packets take anything seconds hours depending size speed network connection (pulling packets mean packet fails run). allow control can pass argument search_options, orderly_search_options object, allows control names locations use, whether metadata refreshed pull anything packets currently downloaded considered candidates. effect running interactively, case can specify search options (root specific) orderly_interactive_set_search_options","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"which-packets-might-be-selected-from-locations-","dir":"Reference","previous_headings":"","what":"Which packets might be selected from locations?","title":"Run a report — orderly_run","text":"search_options argument controls outpack searches packets given query anything might moved network (one outpack archive another). default everything resolved locally ; can depend packets unpacked within current archive.  pass search_options argument contains allow_remote = TRUE (see orderly_search_options packets known anywhere candidates using dependencies needed pull resolved files remote location. Note even packet locally present might needed - content anywhere else unpacked packet reuse content without re-fetching. pull_metadata = TRUE, refresh location metadata pulling, location argument controls locations pulled .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"equivalence-to-the-old-use-draft-option","dir":"Reference","previous_headings":"","what":"Equivalence to the old use_draft option","title":"Run a report — orderly_run","text":"location handling generalises orderly (v1)'s old use_draft option, terms location argument orderly2::orderly_search_options`: use_draft = TRUE location = \"local\" use_draft = FALSE location = c(...) provide locations except local (setdiff(orderly2::orderly_location_list(), \"local\")) use_draft = \"newer\" location = NULL (last option one people preferred new default behaviour). addition, resolve dependencies currently exist production right now options:   updates current metadata production, runs queries packets known remote, depends even (yet) locally.  functionality never available orderly version 1, though intended support .","code":"location = \"production\", pull_metadata = TRUE, require_unpacked = FALSE"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"running-with-a-source-tree-separate-from-outpack-root","dir":"Reference","previous_headings":"","what":"Running with a source tree separate from outpack root","title":"Run a report — orderly_run","text":"Sometimes useful run things different place disk outpack root. know two cases come : running reports within runner server, make clean clone source tree particular git reference new temporary directory run report , insert orderly repo fixed non-temporary location. user convenient torun report hard drive store archive metadata (larger) shared drive. first instance, source path <src> contains file orderly_config.yml directory src/ source reports, separate path <root> contains directory .outpack/ metadata - may also unpacked archive, .git/ directory depending configuration. (Later make sense support \"bare\" outpack layout.)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"manually-setting-report-source-directory","dir":"Reference","previous_headings":"","what":"Manually setting report source directory","title":"Run a report — orderly_run","text":"manually set report source directory, need set path directory ORDERLY_REPORT_SRC environment variable.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Run a report — orderly_run","text":"","code":"# Create a simple example: path <- orderly2::orderly_example(\"default\") #> ✔ Created orderly root at '/tmp/RtmpO3RBoN/file1720100cfa3f'  # Run the 'data' task: orderly2::orderly_run(\"data\", root = path) #> Error: Directory does not exist: '/tmp/RtmpO3RBoN/file1720100cfa3f'  # After running, a finished packet appears in the archive: fs::dir_tree(path) #> Error: [ENOENT] Failed to search directory '/tmp/RtmpO3RBoN/file1720100cfa3f': no such file or directory  # and we can query the metadata: orderly2::orderly_metadata_extract(name = \"data\", root = path) #> Error: Directory does not exist: '/tmp/RtmpO3RBoN/file1720100cfa3f'"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run_info.html","id":null,"dir":"Reference","previous_headings":"","what":"Information about currently running report — orderly_run_info","title":"Information about currently running report — orderly_run_info","text":"Fetch information actively running report.  allows reflect information report back part report, example embedding current report id, information computed dependencies. information slightly different format orderly version 1.x (currently) include information dependencies run outside orderly_run, never reliable previously.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run_info.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Information about currently running report — orderly_run_info","text":"","code":"orderly_run_info()"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_run_info.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Information about currently running report — orderly_run_info","text":"list elements name: name current report id: id current report, NA running interactively root: orderly root path depends: data frame information dependencies (available interactively) index: integer sequence along calls orderly_dependency name: name dependency query: query used find dependency id: computed id included packet filename: file used packet : filename used locally","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search.html","id":null,"dir":"Reference","previous_headings":"","what":"Query outpack's database — orderly_search","title":"Query outpack's database — orderly_search","text":"Evaluate query outpack database, returning vector matching packet ids.  Note default searches packets unpacked available direct use computer; search within packets known locations (might know via metadata) need use options argument.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Query outpack's database — orderly_search","text":"","code":"orderly_search(   ...,   parameters = NULL,   envir = parent.frame(),   options = NULL,   root = NULL )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Query outpack's database — orderly_search","text":"... Arguments passed orderly_query, perhaps just query expression parameters Optionally, named list parameters substitute query (using : prefix) envir Optionally, environment substitute query (using environment: prefix). default use calling environment, can explicitly pass want control lookup happens. options Optionally, orderly_search_options object controlling search performed, packets considered scope. provided, default options used (.e., orderly2::orderly_search_options()) root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Query outpack's database — orderly_search","text":"character vector matching ids. case match query returning single value (e.g., latest(...) single(...)) character missing value (NA_character_)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search_options.html","id":null,"dir":"Reference","previous_headings":"","what":"Packet search options — orderly_search_options","title":"Packet search options — orderly_search_options","text":"Options controlling packet searches carried , example via orderly_search orderly_run. details never included metadata alongside query (, part query even though affect ).","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search_options.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Packet search options — orderly_search_options","text":"","code":"orderly_search_options(   location = NULL,   allow_remote = FALSE,   pull_metadata = FALSE )"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search_options.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Packet search options — orderly_search_options","text":"location Optional vector locations pull . might future expand allow wildcards exceptions. allow_remote Logical, indicating allow packets found currently unpacked (.e., known location metadata ). TRUE, conjunction orderly_dependency might pull large quantity data. pull_metadata Logical, indicating pull metadata immediately search. location given, pass orderly_location_pull_metadata filter locations update.  pulling many packets sequence, want update option FALSE first pull.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_search_options.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Packet search options — orderly_search_options","text":"object class orderly_search_options modified creation (see note pull_metadata)","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_shared_resource.html","id":null,"dir":"Reference","previous_headings":"","what":"Copy shared resources into a packet directory — orderly_shared_resource","title":"Copy shared resources into a packet directory — orderly_shared_resource","text":"Copy shared resources packet directory. can use share common resources (data code) multiple packets. Additional metadata added keep track files came .  Using function requires shared resources directory shared/ exists orderly root; error raised configured attempt fetch files.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_shared_resource.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Copy shared resources into a packet directory — orderly_shared_resource","text":"","code":"orderly_shared_resource(...)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_shared_resource.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Copy shared resources into a packet directory — orderly_shared_resource","text":"... shared resources copy. arguments named, name destination file value filename within shared resource directory. can use limited form string interpolation names argument; using ${variable} pick values envir substitute string.  similar interpolation might familiar glue::glue similar, much simpler concatenation fancy features supported.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_shared_resource.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Copy shared resources into a packet directory — orderly_shared_resource","text":"Invisibly, data.frame columns (fileames copied running packet) (filenames within shared/).  orderly_resource, rely ordering directory expansion performed.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_strict_mode.html","id":null,"dir":"Reference","previous_headings":"","what":"Enable orderly strict mode — orderly_strict_mode","title":"Enable orderly strict mode — orderly_strict_mode","text":"Put orderly2 \"strict mode\", closer defaults orderly 1.0.0; mode explicitly included files (via orderly_resource orderly_shared_resource) copied running packet, warn unexpected files end run.  Using strict mode allows orderly2 aggressive deletes files within source directory, accurate reports , faster start packets developing interactively.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_strict_mode.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Enable orderly strict mode — orderly_strict_mode","text":"","code":"orderly_strict_mode()"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_strict_mode.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Enable orderly strict mode — orderly_strict_mode","text":"Undefined","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_strict_mode.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Enable orderly strict mode — orderly_strict_mode","text":"future, may extend strict mode allow requiring computation occurs within orderly functions (.e., requirements run packet fully known actually running ).  likely default behaviour orderly_strict_mode gain argument. allow server processes either override value (enabling even explicitly given) /require .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_validate_archive.html","id":null,"dir":"Reference","previous_headings":"","what":"Validate unpacked packets. — orderly_validate_archive","title":"Validate unpacked packets. — orderly_validate_archive","text":"Validate unpacked packets. time, expect function become fully featured, validating .","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_validate_archive.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Validate unpacked packets. — orderly_validate_archive","text":"","code":"orderly_validate_archive(..., action = \"inform\", root = NULL, locate = TRUE)"},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_validate_archive.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Validate unpacked packets. — orderly_validate_archive","text":"... Either arguments search can constructed (useful options include name = \"something\"), character vector ids, leave empty validate everything. action action take finding invalid packet. See Details. root path root directory, NULL (default) search one current working directory locate TRUE. function require directory configured orderly, can outpack root (see orderly_init details). locate Logical, indicating root searched .  TRUE, looks directory given root (working directory NULL) parents finds .outpack directory orderly_config.yml","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_validate_archive.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Validate unpacked packets. — orderly_validate_archive","text":"Invisibly, character vector repaired (invalid) packets.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/orderly_validate_archive.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Validate unpacked packets. — orderly_validate_archive","text":"actions can take finding invalid packet : inform (default): just print information problem orphan: mark packet orphaned within metadata, touch files archive (default directory archive/) - safe option leave consistent state without deleting anything. delete: addition marking packet orphan, also delete files archive. Later, add \"repair\" option try fix broken packets. validation interacts option core.require_complete_tree; option TRUE, packet valid (recursive) dependencies also valid, action apply packets also upstream dependencies invalidated.  validation happen even query implied ... include packets complete tree required. validation also interact core.use_file_store repair supported, becomes trivial.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":null,"dir":"Reference","previous_headings":"","what":"Query index — query_index","title":"Query index — query_index","text":"Class managing active index whilst evaluating query","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"public-fields","dir":"Reference","previous_headings":"","what":"Public fields","title":"Query index — query_index","text":"index packet index depends Named list data frames. Names packet ids, values packets depended packet id (.e. parents). uses Named list data frames. Names packet ids, values packets used packet id (.e. children). root outpack root object","code":""},{"path":[]},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"public-methods","dir":"Reference","previous_headings":"","what":"Public methods","title":"Query index — query_index","text":"query_index$new() query_index$get_packet_depends() query_index$get_packet_uses()","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"method-new-","dir":"Reference","previous_headings":"","what":"Method new()","title":"Query index — query_index","text":"Create new query_index object","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Query index — query_index","text":"","code":"query_index$new(root, index, depends, uses)"},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Query index — query_index","text":"root outpack root object index packet index data frame depends Named list data frames. Names packet ids, values packets depended packet id (.e. parents). uses Named list data frames. Names packet ids, values packets used packet id (.e. children). data depends relationships flow direction.","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"method-get-packet-depends-","dir":"Reference","previous_headings":"","what":"Method get_packet_depends()","title":"Query index — query_index","text":"Get ids packets packet depends specified level","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"usage-1","dir":"Reference","previous_headings":"","what":"Usage","title":"Query index — query_index","text":"","code":"query_index$get_packet_depends(id, depth)"},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"arguments-1","dir":"Reference","previous_headings":"","what":"Arguments","title":"Query index — query_index","text":"id id packet get parents depth Depth parents get, depth 1 gets immediate parents depth 2 gets parents parents parents, depth Inf recurse whole tree get parents","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"returns","dir":"Reference","previous_headings":"","what":"Returns","title":"Query index — query_index","text":"ids parents packet","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"method-get-packet-uses-","dir":"Reference","previous_headings":"","what":"Method get_packet_uses()","title":"Query index — query_index","text":"Get ids packets used packet","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"usage-2","dir":"Reference","previous_headings":"","what":"Usage","title":"Query index — query_index","text":"","code":"query_index$get_packet_uses(id, depth)"},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"arguments-2","dir":"Reference","previous_headings":"","what":"Arguments","title":"Query index — query_index","text":"id id packet get children depth Depth children get, depth 1 gets immediate children depth 2 gets children children children, depth Inf recurse whole tree get children","code":""},{"path":"https://mrc-ide.github.io/orderly2/reference/query_index.html","id":"returns-1","dir":"Reference","previous_headings":"","what":"Returns","title":"Query index — query_index","text":"ids children packet","code":""}]