Merge branch 'main' into customized

.readthedocs.yaml

@@ -0,0 +1,21 @@
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Set the version of Python and other tools you might need
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.10"
+
+# Build documentation in the docs/ directory with Sphinx
+sphinx:
+  configuration: docs/conf.py
+
+# Specify dependencies to enable reproducible builds:
+# https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
+python:
+  install:
+  - requirements: docs/requirements.txt

docs/conf.py

@@ -35,6 +35,7 @@
     "sphinx.ext.todo",
     "sphinx.ext.coverage",
     "sphinx.ext.viewcode",
+    # "sphinx_search.extension",  # Disable until search is improved.
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -51,7 +52,7 @@
 
 # General information about the project.
 project = "Lab"
-copyright = "2011-2020 Jendrik Seipp et al."
+copyright = "Jendrik Seipp et al."
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the

docs/downward.tutorial.rst

@@ -123,15 +123,15 @@ Run tutorial experiment
 
 The files below are two experiment scripts, a ``project.py`` module that bundles
 common functionality for all experiments related to the project, a parser
-script, and a script for collecting results and making reports. You can use the
+module, and a script for collecting results and making reports. You can use the
 files as a basis for your own experiments. They are available in the `Lab repo
 <https://github.com/aibasel/lab/tree/main/examples/downward>`_. Copy the files
 into ``experiments/my-exp-dir``.
 
 .. highlight:: bash
 
-Make sure the experiment script and the parser are executable. Then you
-can see the available steps with ::
+Make sure the experiment script is executable. Then you can see the available
+steps with ::
 
     ./2020-09-11-A-cg-vs-ff.py
 
@@ -171,7 +171,7 @@ reference on all Downward Lab classes.
 .. literalinclude:: ../examples/downward/project.py
    :caption:
 
-.. literalinclude:: ../examples/downward/parser.py
+.. literalinclude:: ../examples/downward/custom_parser.py
    :caption:
 
 .. literalinclude:: ../examples/downward/01-evaluation.py

docs/faq.rst

@@ -38,12 +38,43 @@ again as above.
 I forgot to parse something. How can I run only the parsers again?
 ------------------------------------------------------------------
 
-See the `parsing documentation <lab.parser.html>`_ for how to write
-parsers. Once you have fixed your existing parsers or added new parsers,
-add ``exp.add_parse_again_step()`` to your experiment script
-``my-exp.py`` and then call ::
-
-    ./my-exp.py parse-again
+Now that parsing is done in its own experiment step, simply consult the `parsing
+documentation <lab.parser.html>`_ for how to amend your parsers and then run the
+"parse" experiment step again with ::
+
+    ./my-exp.py parse
+
+
+.. _portparsers:
+
+How do I port my parsers to version 8.x?
+----------------------------------------
+
+Since version 8.0, Lab has a dedicated "parse" experiment step. First of all,
+what are the benefits of this?
+
+* No need to write parsers in separate files.
+* Log output from solvers and parsers remains separate.
+* No need for ``exp.add_parse_again_step()``. Parsing and re-parsing is now
+  exactly the same.
+* Parsers are checked for syntax errors before the experiment is run.
+* Parsing runs much faster (for an experiment with 3 algorithms and 5 parsers
+  the parsing time went down from 51 minutes to 5 minutes, both measured on
+  cold file system caches).
+* As before, you can let the Slurm environment do the parsing for you and get
+  notified when the report is finished: ``./myexp.py build start parse fetch
+  report``
+
+To adapt your parsers to this new API, you need to make the following changes:
+
+* Your parser module (e.g., "custom_parser.py") does not have to be executable
+  anymore, but it must be importable and expose a :class:`Parser
+  <lab.parser.Parser>` instance (see the changes to the `translator parser
+  <https://github.com/aibasel/lab/pull/117/files#diff-0a679939eb576c6b402a00ab9b08a3339ecefe3713dc96f9ac6b0e05de9ff4f2>`_
+  for an example). Then, instead of ``exp.add_parser("custom_parser.py")`` use
+  ``from custom_parser import MyParser`` and ``exp.add_parser(MyParser())``.
+* Remove ``exp.add_parse_again_step()`` and insert ``exp.add_step("parse",
+  exp.parse)`` after ``exp.add_step("start", exp.start_runs)``.
 
 
 How can I compute a new attribute from multiple runs?

docs/ff.rst

@@ -27,5 +27,5 @@ Downward experiments, we recommend taking a look at the
 
 Here is a simple parser for FF:
 
-.. literalinclude:: ../examples/ff/ff-parser.py
+.. literalinclude:: ../examples/ff/ff_parser.py
    :caption:

docs/lab.concepts.rst

@@ -4,12 +4,13 @@ Concepts
 ========
 
 An **experiment** consists of multiple **steps**. Most experiments will
-have steps for building and executing the experiment:
+have steps for building and executing the experiment, and parsing logs:
 
     >>> from lab.experiment import Experiment
     >>> exp = Experiment()
     >>> exp.add_step("build", exp.build)
     >>> exp.add_step("start", exp.start_runs)
+    >>> exp.add_step("parse", exp.parse)
 
 Moreover, there are usually steps for **fetching** the results and making
 **reports**:
@@ -18,11 +19,11 @@ Moreover, there are usually steps for **fetching** the results and making
 >>> exp.add_fetcher(name="fetch")
 >>> exp.add_report(Report(attributes=["error"]))
 
-The "build" step creates all necessary files for running the experiment in
-the **experiment directory**. After the "start" step has finished running
-the experiment, we can fetch the result from the experiment directory to
-the **evaluation directory**. All reports only operate on evaluation
-directories.
+The "build" step creates all necessary files for running the experiment in the
+**experiment directory**. After the "start" step has finished running the
+experiment, we can parse data from logs and generated files into "properties"
+files, and then fetch all properties files from the experiment directory to the
+**evaluation directory**. All reports only operate on evaluation directories.
 
 An experiment usually also has multiple **runs**, one for each pair of
 algorithm and benchmark.

docs/lab.tutorial.rst

@@ -35,12 +35,7 @@ Select steps by name or index::
 
     ./exp.py build
     ./exp.py 2
-    ./exp.py 3 4
-
-Here is the parser that the experiment uses:
-
-.. literalinclude:: ../examples/vertex-cover/parser.py
-   :caption:
+    ./exp.py 3 4 5
 
 Find out how to create your own experiments by browsing the `Lab API
 <lab.experiment.html>`_.

docs/news.rst

@@ -1,17 +1,30 @@
 Changelog
 =========
 
-next (unreleased)
+v8.0 (2023-10-21)
 -----------------
 
 Lab
 ^^^
-* Provide support for HTCondor clusters in external repo and add link to docs (Martín Pozo).
+* Make parsing a separate experiment step, see :ref:`FAQs <portparsers>` for motivation and upgrade instructions (Jendrik Seipp).
 
+Downward Lab
+^^^^^^^^^^^^
+* None.
+
+
+v7.5 (2023-10-21)
+-----------------
+
+Lab
+^^^
+* Provide support for `HTCondor <https://htcondor.org/>`_ clusters in a `third-party repository <https://github.com/Martin1887/lab-htcondor-environment>`_ and add link to docs (Martín Pozo).
+* Add documentation for AI Basel's infai_3 partition (Silvan Sievers).
+* Don't rely on the existence of the 'runs-00001-00100' dir when fetching results (Jendrik Seipp).
 
 Downward Lab
 ^^^^^^^^^^^^
-* no changes so far
+* None.
 
 
 v7.4 (2023-08-18)

docs/requirements.in

@@ -0,0 +1,3 @@
+readthedocs-sphinx-search
+sphinx
+sphinx_rtd_theme

docs/requirements.txt

@@ -0,0 +1,64 @@
+#
+# This file is autogenerated by pip-compile with Python 3.10
+# by the following command:
+#
+#    pip-compile
+#
+alabaster==0.7.13
+    # via sphinx
+babel==2.13.0
+    # via sphinx
+certifi==2023.7.22
+    # via requests
+charset-normalizer==3.3.0
+    # via requests
+docutils==0.18.1
+    # via
+    #   sphinx
+    #   sphinx-rtd-theme
+idna==3.4
+    # via requests
+imagesize==1.4.1
+    # via sphinx
+jinja2==3.1.2
+    # via sphinx
+markupsafe==2.1.3
+    # via jinja2
+packaging==23.2
+    # via sphinx
+pygments==2.16.1
+    # via sphinx
+readthedocs-sphinx-search==0.3.1
+    # via -r requirements.in
+requests==2.31.0
+    # via sphinx
+snowballstemmer==2.2.0
+    # via sphinx
+sphinx==7.2.6
+    # via
+    #   -r requirements.in
+    #   sphinx-rtd-theme
+    #   sphinxcontrib-applehelp
+    #   sphinxcontrib-devhelp
+    #   sphinxcontrib-htmlhelp
+    #   sphinxcontrib-jquery
+    #   sphinxcontrib-qthelp
+    #   sphinxcontrib-serializinghtml
+sphinx-rtd-theme==1.3.0
+    # via -r requirements.in
+sphinxcontrib-applehelp==1.0.7
+    # via sphinx
+sphinxcontrib-devhelp==1.0.5
+    # via sphinx
+sphinxcontrib-htmlhelp==2.0.4
+    # via sphinx
+sphinxcontrib-jquery==4.1
+    # via sphinx-rtd-theme
+sphinxcontrib-jsmath==1.0.1
+    # via sphinx
+sphinxcontrib-qthelp==1.0.6
+    # via sphinx
+sphinxcontrib-serializinghtml==1.1.9
+    # via sphinx
+urllib3==2.0.7
+    # via requests

docs/singularity.rst

@@ -11,7 +11,7 @@ Downward Lab.
 
 The experiment script needs a parser and a helper script:
 
-.. literalinclude:: ../examples/singularity/singularity-parser.py
+.. literalinclude:: ../examples/singularity/singularity_parser.py
    :caption:
 
 .. literalinclude:: ../examples/singularity/run-singularity.sh

downward/experiment.py

@@ -9,14 +9,15 @@
 
 from downward import suites
 from downward.cached_revision import CachedFastDownwardRevision
+from downward.parsers.anytime_search_parser import AnytimeSearchParser
+from downward.parsers.exitcode_parser import ExitcodeParser
+from downward.parsers.planner_parser import PlannerParser
+from downward.parsers.single_search_parser import SingleSearchParser
+from downward.parsers.translator_parser import TranslatorParser
 from lab import tools
 from lab.experiment import Experiment, get_default_data_dir, Run
 
 
-DIR = os.path.dirname(os.path.abspath(__file__))
-DOWNWARD_SCRIPTS_DIR = os.path.join(DIR, "scripts")
-
-
 class FastDownwardAlgorithm:
     """
     A Fast Downward algorithm is the combination of revision, driver options and
@@ -122,32 +123,31 @@ class FastDownwardExperiment(Experiment):
         >>> exp = FastDownwardExperiment()
         >>> exp.add_step("build", exp.build)
         >>> exp.add_step("start", exp.start_runs)
+        >>> exp.add_step("parse", exp.parse)
         >>> exp.add_fetcher(name="fetch")
 
     """
 
     # Built-in parsers that can be passed to exp.add_parser().
 
     #: Parsed attributes: "error", "planner_exit_code", "unsolvable".
-    EXITCODE_PARSER = os.path.join(DOWNWARD_SCRIPTS_DIR, "exitcode-parser.py")
+    EXITCODE_PARSER = ExitcodeParser()
 
     #: Parsed attributes: "translator_peak_memory", "translator_time_done", etc.
-    TRANSLATOR_PARSER = os.path.join(DOWNWARD_SCRIPTS_DIR, "translator-parser.py")
+    TRANSLATOR_PARSER = TranslatorParser()
 
     #: Parsed attributes: "coverage", "memory", "total_time", etc.
-    SINGLE_SEARCH_PARSER = os.path.join(DOWNWARD_SCRIPTS_DIR, "single-search-parser.py")
+    SINGLE_SEARCH_PARSER = SingleSearchParser()
 
     #: Parsed attributes: "cost", "cost:all", "coverage".
-    ANYTIME_SEARCH_PARSER = os.path.join(
-        DOWNWARD_SCRIPTS_DIR, "anytime-search-parser.py"
-    )
+    ANYTIME_SEARCH_PARSER = AnytimeSearchParser()
 
     #: Used attributes: "memory", "total_time",
     #: "translator_peak_memory", "translator_time_done".
     #:
     #: Parsed attributes: "node", "planner_memory", "planner_time",
     #: "planner_wall_clock_time", "score_planner_memory", "score_planner_time".
-    PLANNER_PARSER = os.path.join(DOWNWARD_SCRIPTS_DIR, "planner-parser.py")
+    PLANNER_PARSER = PlannerParser()
 
     def __init__(self, path=None, environment=None, revision_cache=None):
         """

downward/scripts/anytime-search-parser.py → downward/parsers/anytime_search_parser.py

@@ -1,5 +1,3 @@
-#! /usr/bin/env python
-
 """
 Parse anytime-search runs of Fast Downward. This includes iterated
 searches and portfolios.
@@ -54,18 +52,18 @@ def add_memory(content, props):
             props["memory"] = raw_memory
 
 
-def main():
-    parser = Parser()
-    parser.add_pattern("raw_memory", r"Peak memory: (.+) KB", type=int),
-    parser.add_function(find_all_matches("cost:all", r"Plan cost: (.+)\n", type=float))
-    parser.add_function(
-        find_all_matches("steps:all", r"Plan length: (.+) step\(s\).\n", type=float)
-    )
-    parser.add_function(reduce_to_min("cost:all", "cost"))
-    parser.add_function(reduce_to_min("steps:all", "steps"))
-    parser.add_function(coverage)
-    parser.add_function(add_memory)
-    parser.parse()
-
-
-main()
+class AnytimeSearchParser(Parser):
+    def __init__(self):
+        super().__init__()
+
+        self.add_pattern("raw_memory", r"Peak memory: (.+) KB", type=int)
+        self.add_function(
+            find_all_matches("cost:all", r"Plan cost: (.+)\n", type=float)
+        )
+        self.add_function(
+            find_all_matches("steps:all", r"Plan length: (.+) step\(s\).\n", type=float)
+        )
+        self.add_function(reduce_to_min("cost:all", "cost"))
+        self.add_function(reduce_to_min("steps:all", "steps"))
+        self.add_function(coverage)
+        self.add_function(add_memory)