Merge pull request #250 from FCP-INDI/update/1.8

📝 More 1.8 Updates

.circleci/config.yml

@@ -14,8 +14,9 @@ commands:
           name: ↑ Installing build dependencies
           command: |
             sudo apt-get update && sudo apt-get -y install git python3-dev graphviz graphviz-dev libgraphviz-dev pkg-config python3-sphinx
+            pip install --user -r requirements.txt
             pip install --user -r https://raw.githubusercontent.com/FCP-INDI/C-PAC/<< parameters.version >>/requirements.txt
-            pip install --user git+https://github.com/${CIRCLE_PROJECT_USERNAME}/C-PAC.git@<< parameters.version >> sphinx m2r numpydoc PyGithub "sphinxcontrib-bibtex<2.0" sphinxcontrib-fulltoc sphinxcontrib-programoutput semver torch
+            pip install --user git+https://github.com/${CIRCLE_PROJECT_USERNAME}/C-PAC.git@<< parameters.version >>
             git clone https://github.com/${CIRCLE_PROJECT_USERNAME}/C-PAC.git /home/circleci/build/C-PAC
             cd /home/circleci/build/C-PAC
             git checkout << parameters.version >>
@@ -24,13 +25,6 @@ commands:
             source ~/simple/bin/activate
             pip install cpac
             deactivate
-      - run:
-          name: 🖇️ Linking references
-          command: |
-            CPAC_DIRECTORY=$(python -c "import os, CPAC; print(os.path.abspath(os.path.dirname(CPAC.__file__)))")
-            ln ${CPAC_DIRECTORY}/resources/configs/1.7-1.8-nesting-mappings.yml docs/_sources/references/1.7-1.8-nesting-mappings.yml || true
-            ln ${CPAC_DIRECTORY}/resources/configs/1.7-1.8-deprecations.yml docs/_sources/references/1.7-1.8-deprecations.yml || true
-            curl -o docs/_sources/references/default_pipeline.yml -L https://raw.githubusercontent.com/${CIRCLE_PROJECT_USERNAME}/C-PAC/<< parameters.version >>/dev/docker_data/default_pipeline.yml || true
   
   run-cpac-commands:
     steps:
@@ -39,7 +33,7 @@ commands:
           name: 👊 Running cpac commands
           command: |
             source ~/simple/bin/activate
-            cpac run --help
+            cpac pull
             mkdir -p docs/_sources/user/cpac
             printf ".. code-block:: console\n\n   $ cpac --help\n\n" > docs/_sources/user/cpac/help.rst
             cpac --help | sed -e "s/.*/   &/" >> docs/_sources/user/cpac/help.rst

CONTRIBUTING.md

@@ -1,13 +1,33 @@
+<!-- TOC -->
+
+- [Branches](#branches)
+    - [Versions](#versions)
+        - [`nightly`](#nightly)
+        - [C-PAC release tags](#c-pac-release-tags)
+- [Guidelines](#guidelines)
+    - [Building](#building)
+        - [Let CircleCI build your drafts / works-in-progress](#let-circleci-build-your-drafts--works-in-progress)
+        - [Build locally (C-PAC ≥ v1.8.0)](#build-locally-c-pac-≥-v180)
+- [Flowcharts](#flowcharts)
+- [References and citations](#references-and-citations)
+- [Environment notes](#environment-notes)
+
+<!-- /TOC -->
+
+<a id="markdown-branches" name="branches"></a>
 ## Branches
 
 Please, always base changes on the `source` branch. `master` branch will be overwritten by the CI deployment.
 
+<a id="markdown-versions" name="versions"></a>
 ### Versions
 
+<a id="markdown-nightly" name="nightly"></a>
 #### `nightly`
 
 Pushes to `source` will rebuild docs at https://fcp-indi.github.io/docs/nightly
 
+<a id="markdown-c-pac-release-tags" name="c-pac-release-tags"></a>
 #### C-PAC release tags
 Tags in the format `v.*-source` will build docs for the given version.
 
@@ -21,6 +41,7 @@ Steps to build a release:
 5. (Force) push to the remote tag (`git push origin v`version`-source`).
 6. CircleCI should deploy the versioned documentation. If the version tag is the newest, it should also overwrite `latest` with these documents. 
 
+<a id="markdown-guidelines" name="guidelines"></a>
 ## Guidelines
 
 - Only write a document once, and liberally use the [reStructured Text `.. include::` directive](https://docutils.sourceforge.io/docs/ref/rst/directives.html#include) to include that document where appropriate.
@@ -37,16 +58,50 @@ Steps to build a release:
   '
   "
   ```
-- Let CircleCI build your drafts / works-in-progress
-    * Build environment will match actual docs build environment
-    * CircleCI takes ~2 minutes to build
-    1. Fork https://github.com/FCP-INDI/fcp-indi.github.com
-    1. In your fork's settings, set the GitHub Pages `source` to `master` branch
-        ![GitHub Pages settings example screenshot](./images/github-pages-settings-example.png)
-    1. Add your project on CircleCI
-    1. Merge your draft / work-in-progress into your fork's `source` branch. Make sure you push to your fork and not the main repository's `source` branch.
-    1. Your fork will publish at `https://[your_GitHub_username].github.io/fcp-indi.github.com/`.
 
+<a id="markdown-building" name="building"></a>
+### Building
+
+<a id="markdown-let-circleci-build-your-drafts--works-in-progress" name="let-circleci-build-your-drafts--works-in-progress"></a>
+#### Let CircleCI build your drafts / works-in-progress
+* Build environment will match actual docs build environment
+* CircleCI takes ~2 minutes to build
+1. Fork https://github.com/FCP-INDI/fcp-indi.github.com
+1. In your fork's settings, set the GitHub Pages `source` to `master` branch
+    ![GitHub Pages settings example screenshot](./images/github-pages-settings-example.png)
+1. Add your project on CircleCI
+1. Merge your draft / work-in-progress into your fork's `source` branch. Make sure you push to your fork and not the main repository's `source` branch.
+1. Your fork will publish at `https://[your_GitHub_username].github.io/fcp-indi.github.com/`.
+
+<a id="markdown-build-locally-c-pac-≥-v180" name="build-locally-c-pac-≥-v180"></a>
+#### Build locally (C-PAC ≥ v1.8.0)
+This documentation aspires to rely on a [single source of truth](https://en.wikipedia.org/wiki/Single_source_of_truth) where possible.  To this end, building this documentation requires an installation of the version of [C-PAC](https://github.com/FCP-INDI/C-PAC) that is being documented.
+
+Steps to build this documentation locally:
+1. Clone this repository.
+1. _(optional)_ <details><summary>Locally replicate the step "👊 Running cpac commands" from [.circleci/config](./.circleci/config) to generate [cpac](https://pypi.org/project/cpac/) usage strings.</summary>
+    Either perform this "👊 Running cpac commands" step in a separate Python environment or uninstall cpac after generating the usage string(s).
+    1. _(optional)_ Create an environment for cpac and activate this environment.
+    1. `pip install cpac`
+    1. If you don't have a local container for the version of C-PAC you're documenting, `cpac pull` to download the latest or `cpac pull --tag $TAG` to pull a specific version.
+    1. Generate ReStructuredText documents with cpac usage strings:
+       ```BASH
+        mkdir -p docs/_sources/user/cpac
+        printf ".. code-block:: console\n\n   $ cpac --help\n\n" > docs/_sources/user/cpac/help.rst
+        cpac --help | sed -e "s/.*/   &/" >> docs/_sources/user/cpac/help.rst
+        mkdir -p docs/_sources/user/run
+        printf "Usage: cpac run\n\`\`\`\`\`\`\`\`\`\`\`\`\`\`\`\n.. code-block:: console\n\n   $ cpac run --help\n\n" > docs/_sources/user/run/help.rst
+        cpac run --help | sed -e "s/.*/   &/" >> docs/_sources/user/run/help.rst
+        mkdir -p docs/_sources/user/utils
+        printf "Usage: cpac utils\n\`\`\`\`\`\`\`\`\`\`\`\`\`\`\`\`\`\n.. code-block:: console\n\n   $ cpac utils --help\n\n" > docs/_sources/user/utils/help.rst
+        cpac utils --help | sed -e "s/.*/   &/" >> docs/_sources/user/utils/help.rst
+        ```
+    1. `deactivate` your cpac environment if you used a separate environment or `pip uninstall cpac`.
+    </details>
+1. Locally install [C-PAC](https://github.com/FCP-INDI/C-PAC) from source.
+1. Run `./bin/build $VERSION` where `$VERSION` is the version to build (`nightly`, `latest`, or [<span title='Semantic Versioning'>semver</span>](https://semver.org/) for production, but this string can be anything you want locally). ![example version](./images/example_version.png)
+
+<a id="markdown-flowcharts" name="flowcharts"></a>
 ## Flowcharts
 
 - SVGs exported from Lucidchart have scaling coded in in `width` and `height` XML attributes. Add the XML attributes `preserveAspectRatio="xMinYMin meet"` and `viewBox` to the SVG element in the actual SVG files:
@@ -64,6 +119,7 @@ where `{width}` and `{height}` are the values already present in the existing `w
     <object data="../_static/path/to/chart.svg" type="image/svg+xml"></object>
 ```
 
+<a id="markdown-references-and-citations" name="references-and-citations"></a>
 ## References and citations
 
 [sphinxcontrib-bibtex](https://sphinxcontrib-bibtex.readthedocs.io/) is installed and configured. This extension creates links between the citations and the reference in the reference list and formats citations in referenced BibTeX files using built-in or [custom styles](https://github.com/FCP-INDI/fcp-indi.github.com/blob/source/docs/_sources/references/style.py). To use this Sphinx extension, 
@@ -114,6 +170,7 @@ The rendered file should look something like
 > 
 > <a name="ref1" href="#backref1">[1]</a> Craddock, C., Sikka, S., Cheung, B., Khanuja, R., Ghosh, S. S., Yan, C., Li, Q., Lurie, D., Vogelstein, J., Burns, R., Colcombe, S., Mennes, M., Kelly, C., Di Martino, A., Castellanos, F. X., and Milham, M. 2013. [Towards automated analysis of connectomes: the Configurable Pipeline for the Analysis of Connectomes (C-PAC).](http://www.frontiersin.org/neuroinformatics/10.3389/conf.fninf.2013.09.00042/full) *Frontiers in neuroinformatics* 42. doi:[10.3389/conf.fninf.2013.09.00042](https://dx.doi.org/10.3389/conf.fninf.2013.09.00042)
 
+<a id="markdown-environment-notes" name="environment-notes"></a>
 ## Environment notes
 * Because [C-PAC](https://github.com/FCP-INDI/C-PAC.git) and [cpac](https://github.com/FCP-INDI/cpac.git) have conflicting commandline commands, we first run any `cpac` commands in a virtual environment and spoof the `command-output` directive with `code-block` like 
    ```RST

bin/build

@@ -1,3 +1,10 @@
 #!/bin/bash
 
+# link references
+CPAC_DIRECTORY=$(python -c "import os, CPAC; print(os.path.abspath(os.path.dirname(CPAC.__file__)))")
+ln ${CPAC_DIRECTORY}/resources/configs/1.7-1.8-nesting-mappings.yml docs/_sources/references/1.7-1.8-nesting-mappings.yml || true
+ln ${CPAC_DIRECTORY}/resources/configs/1.7-1.8-deprecations.yml docs/_sources/references/1.7-1.8-deprecations.yml || true
+ln ${CPAC_DIRECTORY}/resources/configs/default_pipeline.yml docs/_sources/references/default_pipeline.yml || true
+
+# build docs
 sphinx-build -b html docs/_sources docs/$1

docs/_sources/conf.py

@@ -27,7 +27,7 @@
 
 sys.path.append(os.path.dirname(__file__))
 
-from references import CPAC_DocsStyle
+from references import CPAC_DocsStyle  # noqa: E402
 
 register_plugin('pybtex.style.formatting', 'cpac_docs_style', CPAC_DocsStyle)
 

docs/_sources/user/anat.rst

@@ -5,6 +5,27 @@
 
     <div class="flowchart-container"><object data="../_static/flowcharts/anatomical.svg" type="image/svg+xml"></object></div>
 
+Surface Analysis
+^^^^^^^^^^^^^^^^
+
+Surface analysis runs FreeSurfer recon-all and generates CSF, WM, GM masks, pial surface mesh, smoothed surface mesh, spherical surface mesh, white matter surface mesh, sulcal depth surface maps, cortical thickness surface maps and cortical volume surface maps.
+
+Configuring CPAC to run surface analysis:
+"""""""""""""""""""""""""""""""""""""""""
+
+.. figure:: /_images/anat_surface.png
+
+#. **FreeSurfer - [On,Off]:** FreeSurfer recon-all. Default is Off.
+
+Configuration Without the GUI
+""""""""""""""""""""""""""""""
+
+The following nested key/value pairs that will be set to these defaults if not defined in your :doc:`pipeline configuration YAML </user/pipelines/pipeline_config>`:
+
+.. literalinclude:: /references/default_pipeline.yml
+   :language: YAML
+   :lines: 152-159
+
 Initial Preprocessing
 ^^^^^^^^^^^^^^^^^^^^^
 

docs/_sources/user/brainlife.io_app.rst

@@ -0,0 +1,7 @@
+C-PAC brainlife.io app
+----------------------
+
+Please visit |DOI:10.25663/brainlife.app.399| to see documentation for C-PAC on brainlife.io and  brainlife.io generally.
+
+.. |DOI:10.25663/brainlife.app.399| image:: https://img.shields.io/badge/DOI-10.25663%2Fbrainlife.app.399-blue
+   :target: https://doi.org/10.25663/brainlife.app.399

docs/_sources/user/help.rst

@@ -9,6 +9,15 @@ If you have a question that is not answered in the User Guide or encounter an is
 
 View Crash Files
 ^^^^^^^^^^^^^^^^
+
+C-PAC ≥1.8.0
+************
+
+As of C-PAC v1.8.0, crash files are plain text files and can be read with any text editor.
+
+CPAC ≤1.7.2
+***********
+
 If you have the :ref:`cpac Python package <cpac-python-package>`, you can simply run 
 
 .. code-block:: console

docs/_sources/user/output_dir.rst

@@ -1,5 +1,33 @@
 Check Your Outputs
-=============================
+==================
+
+.. _1.8-outputs:
+
+C-PAC ≥ 1.8.0
+^^^^^^^^^^^^^
+
+The output structure of C-PAC ≥ 1.8.0 is based on `the Brain Imaging Data Structure v1.6.0-dev BIDS Derivatives specification <https://bids-specification.readthedocs.io/en/latest/05-derivatives/01-introduction.html>`_. The output directory structure is a simple tree like
+
+.. code-block:: text
+
+   └── pipeline-name
+       ├── log
+       │   └── pipeline_pipeline-name
+       │       └── subject_session
+       └── output
+           └── cpac_pipeline-name
+               └── subject_session
+                   ├── anat
+                   └── func
+
+.. note::
+
+   If any nodes use more memory at runtime than C-PAC estimates, C-PAC will report those instances near the end of the log in the terminal and in a log file called ``callback.log.resource_overusage.txt`` (beginning in v1.8.0). Please `report excessive memory usage <https://github.com/FCP-INDI/C-PAC/issues/new>`_ to the C-PAC team.
+
+.. _1.7-outputs:
+
+C-PAC ≤ 1.7.2
+^^^^^^^^^^^^^
 
 A standard C-PAC run output directory for one participant-session is shown below. A directory like this will exist for each participant-session in the run.
 
@@ -12,7 +40,7 @@ This is with all derivatives enabled, and the output, smoothing, and z-scoring o
 .. figure:: /_images/output_dir_default.png
 
 Descriptions
-^^^^^^^^^^^^^^^^^^^
+------------
 
 The output directory folders below are produced during a default run. If you enabled the additional outputs and have the full directory, proceed to the "Full Directory" section further below for descriptions of these extra outputs.
 
@@ -73,7 +101,7 @@ The output directory folders below are produced during a default run. If you ena
 * **QC-interface_{scan}_{nuisance/preprocessing strategy}.html**: :doc:`QC Interface HTML page </user/qc_interface>` for each scan and nuisance/preprocessing strategy combination.
 
 Sub-Directories
-^^^^^^^^^^^^^^^^^^^^^^^^^^
+---------------
 
 Each folder in the output directory may have a different amount of sub-directories, depending on how many functional scans specified in the data configuration, or how many pipeline customizations and forked strategies you specified in the pipeline configuration. For example, if there are multiple functional scans, you'll see a folder for each one if you enter any of the functional-derived outputs' folders, as seen below:
 
@@ -88,7 +116,8 @@ And sub-directories for each ROI provided for a derivative that may take in mult
 .. figure:: /_images/output_dir_masks.png
 
 Full Directory
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+--------------
+
 The C-PAC output directory is considerably larger when "Extra Functional Outputs", "Debugging Outputs", and both Smoothed/Non-smoothed and both z-score standardized/raw outputs are all enabled.
 
 In addition to the output directories described above under "Descriptions", the following outputs are also written to the output directory when all of the output options mentioned above are enabled:
@@ -142,7 +171,7 @@ In addition to the output directories described above under "Descriptions", the
 * **reho**: The direct output of ReHo, before warping to standard space.
 
 Visual Quality Control
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+----------------------
 
 C-PAC's data quality control (QC) interface allows you to take a quick glance at the overall quality of your results (registration quality, signal-to-noise ratio, movement plots, computed derivative histograms, etc.). In its current form, the QC interface is a collection of HTML pages - one for each participant-scan-nuisance regression strategy combination, and they can be found in the Output Directory under each participant's directory level.
 
@@ -151,7 +180,7 @@ C-PAC's data quality control (QC) interface allows you to take a quick glance at
 In future releases, more visualizations will be introduced, and the QC pages will be integrated into an easy-to-use dashboard that is updated throughout the process of your C-PAC run, and also provides information on the status of the pipeline. As always, we greatly appreciate any ideas, suggestions, or items on your wishlist and `take them into consideration <https://groups.google.com/forum/#!forum/cpax_forum>`_.
 
 Quick Look
-^^^^^^^^^^
+----------
 
 **Skull-stripping and Segmentation Quality**
 

docs/_sources/user/quick.rst

@@ -7,7 +7,9 @@ C-PAC Quickstart
 
 .. include:: cpac.rst
 
-For instructions to run in Docker or Singularity without installing cpac (Python package), see :doc:`All Run Options </user/running>`
+For instructions to run C-PAC in Docker or Singularity without installing cpac (Python package), see :doc:`All Run Options </user/running>`
+
+.. include:: brainlife.io_app.rst
 
 Default Pipeline
 ----------------

docs/_sources/user/quick_overview.rst

@@ -1,7 +1,7 @@
 Let's Go!
 ---------
 
-#. The recommended way to run C-PAC is through the use of a container such as Docker or Singularity. Using these containers can be facilitated by :doc:`cpac`.
+#. The recommended way to run C-PAC is through the use of a container such as Docker or Singularity. Using these containers can be facilitated by :doc:`cpac`. C-PAC is also available as a |brainlife.io/app|.
 
 #. Containers are completely pre-built environments that help ensure reproducibility (same exact library versions, behavior, etc.).
 
@@ -16,3 +16,6 @@ Let's Go!
 #. C-PAC can also pull input data from the cloud (AWS S3 buckets), so you can pull public data from any of the open-data releases immediately.
 
 #. More information regarding all of these points are available below, and throughout the rest of the User Guide.
+
+.. |brainlife.io/app| image:: https://img.shields.io/badge/brainlife.io-app-green.svg
+   :target: https://brainlife.io/app/5f3593e84615e04651bf9364

requirements.txt

@@ -0,0 +1,9 @@
+m2r
+numpydoc
+PyGithub
+sphinx
+sphinxcontrib-bibtex<2.0
+sphinxcontrib-fulltoc
+sphinxcontrib-programoutput
+semver
+torch