diff --git a/docs/Makefile b/docs/Makefile new file mode 100644 index 0000000..d4bb2cb --- /dev/null +++ b/docs/Makefile @@ -0,0 +1,20 @@ +# Minimal makefile for Sphinx documentation +# + +# You can set these variables from the command line, and also +# from the environment for the first two. +SPHINXOPTS ?= +SPHINXBUILD ?= sphinx-build +SOURCEDIR = . +BUILDDIR = _build + +# Put it first so that "make" without argument is like "make help". +help: + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +.PHONY: help Makefile + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/docs/api/index.rst b/docs/api/index.rst new file mode 100644 index 0000000..855f867 --- /dev/null +++ b/docs/api/index.rst @@ -0,0 +1,9 @@ +.. _api: + +Exorcist API Reference +====================== + +.. toctree:: + :maxdepth: 2 + + taskstatusdb diff --git a/docs/api/taskstatusdb.rst b/docs/api/taskstatusdb.rst new file mode 100644 index 0000000..c322ea1 --- /dev/null +++ b/docs/api/taskstatusdb.rst @@ -0,0 +1,10 @@ +Task Status Database +==================== + +.. module:: exorcist.taskdb + +.. autosummary:: + :toctree: generated/ + + AbstractTaskStatusDB + TaskStatusDB diff --git a/docs/conf.py b/docs/conf.py new file mode 100644 index 0000000..bc47567 --- /dev/null +++ b/docs/conf.py @@ -0,0 +1,31 @@ +# Configuration file for the Sphinx documentation builder. +# +# For the full list of built-in configuration values, see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html + +# -- Project information ----------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information + +project = 'Exorcist' +copyright = '2023, Open Free Energy' +author = 'Open Free Energy' + +# -- General configuration --------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration + +extensions = [ + "sphinx.ext.autodoc", + "sphinx.ext.autosummary", + "sphinx.ext.napoleon", +] + +templates_path = ['_templates'] +exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] + + + +# -- Options for HTML output ------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output + +html_theme = 'alabaster' +html_static_path = ['_static'] diff --git a/docs/guide/index.rst b/docs/guide/index.rst new file mode 100644 index 0000000..c6cdd8e --- /dev/null +++ b/docs/guide/index.rst @@ -0,0 +1,17 @@ +User Guide +========== + +This guide is designed to introduce developers to the ideas, architecture, +and terminology used by Exorcist. We will use the term "client" to refer to +developers who use Exorcist in their own projects, in order to distinguish +from the "users" of the client application. + +Exorcist is designed to be directly used by client code developers. It is +not intended for direct usage by end users, although we provide several +conveniences that client developers can use in their code to improve the end +user experience. + + +.. toctree:: + + intro diff --git a/docs/guide/intro.rst b/docs/guide/intro.rst new file mode 100644 index 0000000..61f2e2b --- /dev/null +++ b/docs/guide/intro.rst @@ -0,0 +1,55 @@ +Introduction to Exorcist +======================== + +Tasks form a directed acyclic graph (DAG) +----------------------------------------- + +Any large-scale campaign can be described as a directed acyclic graph of +tasks, where edges represent the flow of information from an earlier stage +to a later stage. This graph is directed because the outputs of one dask can +be the inputs of a future task, and it is acyclic because it is not possible +for a task to require its own outputs as an input. + +Task-based frameworks, like Exorcist, implement efficient methods for +executing these task graphs. + +Three databases in Exorcist +--------------------------- + +The basic idea of Exorcist is that to separate three types of data storage: + +* **Task Status Database**: The task status database is the core of + Exorcist, and fully implemented in Exorcist +* **Task Details Database**: The task details database is a key-value store + that describes the tasks to be performed; this is specific to the + application +* **Results Store**: The results store is a generic storage of result data + from the specific application. The only thing Exorcist needs to know is + whether the received result was a success or a failure (in which case, it + can be retried). + +Some aspects that distinguish Exorcist from similar tools are the separation +of task status from task descriptions, and the ability for the user to +customize how task descriptions or task status are stored. + +Practical usage +--------------- + +In practice, we tend to think of the user's experience in two stages: +planning the initial campaign, and the running the campaign. + +Planning the campaign +~~~~~~~~~~~~~~~~~~~~~ + +For many use cases, planning the campaign happens on the user's workstation +or on the head node of a cluster. While this can, in principle, be an +expensive stage, which runs on a compute node, it is more frequently +something that is done interactively by the user. + +The "campaign planner" tool must have write access to the task status +database and to the task details database. + +Running a campaign: the worker +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The second diff --git a/docs/index.rst b/docs/index.rst new file mode 100644 index 0000000..9f6d00b --- /dev/null +++ b/docs/index.rst @@ -0,0 +1,28 @@ +Exorcist +======== + +Exorcist is a tool for execution and orchestration of many-task computing. +It is specifically designed for a common case in the field of simulation, +where a large simulation campaign may include many loosely coupled +individual simulations, each of which may requires hours to days to run, and +the results need to be gathered into a large storage backend. + +At small to moderate scale, Exorcist can run without setting up any +long-running daemon. At larger scales, Exorcist can interface with standard +database backends, e.g. PostgreSQL. + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + guide/index + api/index + + + +Indices and tables +================== + +* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` diff --git a/docs/make.bat b/docs/make.bat new file mode 100644 index 0000000..32bb245 --- /dev/null +++ b/docs/make.bat @@ -0,0 +1,35 @@ +@ECHO OFF + +pushd %~dp0 + +REM Command file for Sphinx documentation + +if "%SPHINXBUILD%" == "" ( + set SPHINXBUILD=sphinx-build +) +set SOURCEDIR=. +set BUILDDIR=_build + +%SPHINXBUILD% >NUL 2>NUL +if errorlevel 9009 ( + echo. + echo.The 'sphinx-build' command was not found. Make sure you have Sphinx + echo.installed, then set the SPHINXBUILD environment variable to point + echo.to the full path of the 'sphinx-build' executable. Alternatively you + echo.may add the Sphinx directory to PATH. + echo. + echo.If you don't have Sphinx installed, grab it from + echo.https://www.sphinx-doc.org/ + exit /b 1 +) + +if "%1" == "" goto help + +%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% +goto end + +:help +%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% + +:end +popd diff --git a/exorcist/taskdb.py b/exorcist/taskdb.py index f99cc81..b41a763 100644 --- a/exorcist/taskdb.py +++ b/exorcist/taskdb.py @@ -136,7 +136,7 @@ class TaskStatusDB(AbstractTaskStatusDB): """Database for managing execution and orchestration of tasks. This implementation is built on SQLAlchemy. For simple usage, the - recommendation is to use the :method:`.from_filename` method to create + recommendation is to use the :meth:`.from_filename` method to create this object, rather than its ``__init__``. The ``__init__`` method takes a SQLAlchemy engine, which provides much more flexibility in choice of backend.