From 8ef2bb922dfa3ebeebbce96956b2291b31569aca Mon Sep 17 00:00:00 2001 From: edennihy Date: Wed, 10 Apr 2024 10:24:53 -0700 Subject: [PATCH] Add new page on writing JSON BLOCK files. --- .../writing-json-block-files.rst | 506 ++++++++++++++++++ index.rst | 20 + 2 files changed, 526 insertions(+) create mode 100644 Control-User-Interfaces/writing-json-block-files.rst diff --git a/Control-User-Interfaces/writing-json-block-files.rst b/Control-User-Interfaces/writing-json-block-files.rst new file mode 100644 index 0000000..cc90396 --- /dev/null +++ b/Control-User-Interfaces/writing-json-block-files.rst @@ -0,0 +1,506 @@ +.. This is a template top-level index file for a directory in the procedure's arm of the documentation + +.. This is the label that can be used as for cross referencing in the given area +.. Recommended format is "Directory Name"-"Title Name" -- Spaces should be replaced by hypens +.. _Writing-JSON-BLOCK-Files: + +######################## +Writing JSON BLOCK Files +######################## + +This page contain instructions and an example of how to write JSON BLOCK files for use with the Scheduler. +Fundamentally, JSON BLOCKS are collections of observing scripts and their associated configurations serialized into JSON format. +They provide users a way to develop complex observing sequences using our standard SAL Script library, +without having to develop a new SAL Script. + +If this is the first time you are contributing to the observatory control software, it is highly recommend that you take a look at the `development guidelines`_. + +.. _development guidelines: https://tssw-developer.lsst.io/ + + +.. _Writing-JSON-BLOCK-Files-JIRA-Ticket: + +Getting started with a BLOCK Ticket JIRA +---------------------------------------- + +All JSON BLOCKS should have an associated BLOCK Ticket in JIRA. +This allow us to track progress and provides traceability for JSON BLOCKS to be added to night plans. +Users should start by generating a new JIRA ticket. +This JIRA ticket number provides a BLOCK-ID for reference that can be used to uniquely identify data and individual executions. + +.. _Writing-JSON-BLOCK-Files-Block-Repository: + +The BLOCK Repository in `ts config ocs`_ +---------------------------------------- + +.. _ts config ocs: https://github.com/lsst-ts/ts_config_ocs + +The first step in creating a new JSON BLOCK is to determine which Scheduler will execute the BLOCK. +At current, there are three Scheduler instances available for executing BLOCKS: MTScheduler, ATScheduler, and the OCS Scheduler. +Each Scheduler instance maintains its own repository of JSON BLOCKS, and can only execute valid JSON BLOCKS within its own repository. + +To get started, clone this repository and open a ticket branch to develop your JSON BLOCK: + +.. prompt:: bash + + git clone https://github.com/lsst-ts/ts_config_ocs.git + +Once the JSON BLOCK file is ready, open a new pull request and request review by one of the Commissioning Scientists to get your JSON BLOCK merged into +`ts config ocs`_ and deployed at the summit for execution. + +.. _Writing-JSON-BLOCK-Files-Block-Requirements: + +JSON BLOCK Requirements +----------------------- + +A JSON BLOCK requires three fields: + +.. code-block:: python + + { + "name": str, + "program": str, + "scripts": list[ObservingScript] + } + + +``name`` +Used as an human-readable identifier for the BLOCK. +It is not required to be unique and is often set to the be the same as ``program``. + +``program`` +Used iternally by the Scheduler as the identifier or ``BLOCK-ID``. +It must be unique (i.e. two JSON BLOCKS cannot have the same ``program``), +and it will be used to load the BLOCK into the scheduler. +A good choice for the ``program`` field is the BLOCK JIRA ticket name. + +.. note:: + + There is also a ``program`` keyword specified in most scripts that corresponds to the ``science_program`` keyword written to the Butler. + Unless specified as a script configuration, the ``program`` used in the metadata of the JSON BLOCK will NOT translate to the ``science_program``. + +``scripts`` +Will contain the list of SAL scripts and configurations that will make up our BLOCK. + +Some additional, optional fields that can be passed include: + +``constraints`` +Currently ignored by the Scheduler. +Provides a set of observing constraints for conditions on which to schedule the JSON BLOCK for observation. + +``id`` +A unique uuid that can be used for tracing the JSON BLOCK. If not provided, this field is automatically generated by the Scheduler CSC. + +.. _Writing-JSON-BLOCK-Files-Full-Example: + +A Full Example JSON BLOCK +------------------------- + +Here we will build a JSON BLOCK to take a series of images with the Auxiliary Telescope to determine the focus offset of a new optical element. +This full example is based on the test described in https://rubinobs.atlassian.net/browse/SITCOM-1012 and corresponds to BLOCK-91 https://rubinobs.atlassian.net/browse/BLOCK-91, observed in September 2023. + +.. code-block:: json + + { + "name": "BLOCK-91", + "program": "BLOCK-91", + "constraints": [], + "scripts": [ + { + "name": "auxtel/track_target.py", + "standard": true, + "parameters": { + "target_name": "gam Gru" + } + }, + { + "name": "auxtel/latiss_acquire.py", + "standard": false, + "parameters": { + "program": "BLOCK-91", + "reason": "SITCOM-1012", + "acq_grating": "holo4_003", + "do_reacquire": true, + "acq_exposure_time": 0.2, + "acq_filter": "SDSSr_65mm" + } + }, + { + "name": "auxtel/offset_ataos.py", + "standard": true, + "parameters": { + "z": -0.1 + } + }, + { + "name": "auxtel/take_image_latiss.py", + "standard": true, + "parameters": { + "program": "BLOCK-91", + "reason": "SITCOM-1012", + "nimages": 2, + "exp_times": 10, + "image_type": "FOCUS", + "filter": "cyl_lens", + "grating": "holo4_003" + } + }, + { + "name": "auxtel/offset_ataos.py", + "standard": true, + "parameters": { + "z": 0.025 + } + }, + { + "name": "auxtel/take_image_latiss.py", + "standard": true, + "parameters": { + "program": "BLOCK-91", + "reason": "SITCOM-1012", + "nimages": 2, + "exp_times": 10, + "image_type": "FOCUS", + "filter": "cyl_lens", + "grating": "holo4_003" + } + }, + { + "name": "auxtel/offset_ataos.py", + "standard": true, + "parameters": { + "z": 0.025 + } + }, + { + "name": "auxtel/take_image_latiss.py", + "standard": true, + "parameters": { + "program": "BLOCK-91", + "reason": "SITCOM-1012", + "nimages": 2, + "exp_times": 10, + "image_type": "FOCUS", + "filter": "cyl_lens", + "grating": "holo4_003" + } + }, + { + "name": "auxtel/offset_ataos.py", + "standard": true, + "parameters": { + "z": 0.025 + } + }, + { + "name": "auxtel/take_image_latiss.py", + "standard": true, + "parameters": { + "program": "BLOCK-91", + "reason": "SITCOM-1012", + "nimages": 2, + "exp_times": 10, + "image_type": "FOCUS", + "filter": "cyl_lens", + "grating": "holo4_003" + } + }, + { + "name": "auxtel/offset_ataos.py", + "standard": true, + "parameters": { + "z": 0.025 + } + }, + { + "name": "auxtel/take_image_latiss.py", + "standard": true, + "parameters": { + "program": "BLOCK-91", + "reason": "SITCOM-1012", + "nimages": 2, + "exp_times": 10, + "image_type": "FOCUS", + "filter": "cyl_lens", + "grating": "holo4_003" + } + }, + { + "name": "auxtel/offset_ataos.py", + "standard": true, + "parameters": { + "z": 0.025 + } + }, + { + "name": "auxtel/take_image_latiss.py", + "standard": true, + "parameters": { + "program": "BLOCK-91", + "reason": "SITCOM-1012", + "nimages": 2, + "exp_times": 10, + "image_type": "FOCUS", + "filter": "cyl_lens", + "grating": "holo4_003" + } + }, + { + "name": "auxtel/offset_ataos.py", + "standard": true, + "parameters": { + "z": 0.025 + } + }, + { + "name": "auxtel/take_image_latiss.py", + "standard": true, + "parameters": { + "program": "BLOCK-91", + "reason": "SITCOM-1012", + "nimages": 2, + "exp_times": 10, + "image_type": "FOCUS", + "filter": "cyl_lens", + "grating": "holo4_003" + } + }, + { + "name": "auxtel/offset_ataos.py", + "standard": true, + "parameters": { + "z": 0.025 + } + }, + { + "name": "auxtel/take_image_latiss.py", + "standard": true, + "parameters": { + "program": "BLOCK-91", + "reason": "SITCOM-1012", + "nimages": 2, + "exp_times": 10, + "image_type": "FOCUS", + "filter": "cyl_lens", + "grating": "holo4_003" + } + }, + { + "name": "auxtel/offset_ataos.py", + "standard": true, + "parameters": { + "z": 0.025 + } + }, + { + "name": "auxtel/take_image_latiss.py", + "standard": true, + "parameters": { + "program": "BLOCK-91", + "reason": "SITCOM-1012", + "nimages": 2, + "exp_times": 10, + "image_type": "FOCUS", + "filter": "cyl_lens", + "grating": "holo4_003" + } + }, + { + "name": "auxtel/offset_ataos.py", + "standard": true, + "parameters": { + "z": -0.1 + } + }, + { + "name": "auxtel/stop_tracking.py", + "standard": true, + "parameters": {} + } + ] + } + +.. _Writing-JSON-BLOCK-Files-Notebook: + +Generating this BLOCK in a Notebook +----------------------------------- + +To generate this JSON BLOCK, we will make use of a set of convenience classes defined in the `ts observing`_ repository. +The `ts observing`_ repository is also used by the Scheduler CSC when loading JSON BLOCKS, +so by using it to generate our JSON file we can ensure proper formatting. + +.. _ts observing: https://github.com/lsst-ts/ts_observing + +The `ts observing`_ repository can be installed in your local environment by running + +.. prompt:: bash + + git clone https://github.com/lsst-ts/ts_observing.git + cd ts_observing + pip install -e . --no-deps + +and restarting your notebook kernel. + +To setup your notebook, import the ``ObservingBlock`` and ``ObservingScript`` classes from `ts observing`_ + +.. code:: python + + from lsst.ts.observing import ObservingBlock, ObservingScript + +Next, we define some parameters to be used throughout the notebook and instantiate an empty list of scripts. + +.. code:: python + + name = "BLOCK-91" + program = "BLOCK-91" + reason = "SITCOM-1012" + constraints = [] + scripts = [] + filter_to_use = 'cyl_lens' + +Here we setup our first script to slew to and track a target using the ``ObservingScript`` class. +We must pass the script configuration to the ``ObservingScript`` class as a ``parameters`` dictionary, +and then append it to our list of scripts. + +.. code:: python + + track_target_script = ObservingScript( + name = "auxtel/track_target.py", + standard = True, + parameters = dict( + target_name="gam Gru" + ) + ) + scripts.append(track_target_script) + +Now add a script to acquire and re-center the target on the grating hotspot + +.. code:: python + + acquire_script = ObservingScript( + name="auxtel/latiss_acquire.py", + standard=False, + parameters = dict( + program = program, + reason = reason, + acq_grating = "holo4_003", + do_recquire = True + ) + ) + + scripts.append(acquire_script) + +And finally we setup our data taking block, utilizing a loop to successively apply ATAOS offsets and take images. + +.. code:: python + + z_offset_start = -0.1 # mm + z_offset_step = 0.025 # mm + z_offset_end = 0.1 + + offset_script = ObservingScript( + name = "auxtel/offset_ataos.py", + standard = True, + parameters = dict( + z = z_offset_start + ) + ) + + take_images_script = ObservingScript( + name = "auxtel/take_image_latiss.py", + standard = True, + parameters = dict( + program = program, + reason = reason, + nimages = 2, + exp_times = 30, + image_type = "FOCUS", + filter = filter_to_use, + grating = 'holo4_003', + ) + ) + + scripts.append(offset_script) + scripts.append(take_images_script) + + z_offset_total = z_offset_start + + while z_offset_total < z_offset_end: + offset_script = ObservingScript( + name = "auxtel/offset_ataos.py", + standard = True, + parameters = dict( + z = z_offset_step + ) + ) + + + scripts.append(offset_script) + scripts.append(take_images_script) + + z_offset_total+=z_offset_step + +To finish our block, we return the ATAOS to its nominal position and stop tracking the target. + +.. code:: python + + offset_script = ObservingScript( + name = "auxtel/offset_ataos.py", + standard = True, + parameters = dict( + z = -z_offset_end + ) + ) + + + scripts.append(offset_script) + + stop_tracking_script = ObservingScript( + name = "auxtel/stop_tracking.py", + standard = True, + parameters = dict() + ) + + scripts.append(stop_tracking_script) + +Now, we assemble the block using the ``ObservingBlock`` class and write it out to a JSON file. + +.. code:: python + + block = ObservingBlock( + name = "BLOCK-91", + program = "BLOCK-91", + scripts = scripts, + ) + + output_file_path = name+'.json' + + with open(output_file_path, 'w') as fp: + fp.write(block.model_dump_json(indent=2)) + +This will produce the file ``BLOCK-91.json`` in your local directory. +This file is ready to be added to your development branch in `ts config ocs`_. + +.. _Writing-JSON-BLOCK-Best-Practices: + +Best Practices for JSON BLOCK Writing +------------------------------------- + +Here we provide a few tips for effective JSON BLOCK writing. + +- Evaluate the full SAL Script library before deciding which scripts to use. It may be that you can combine two or more steps by choosing a different, more appropriate SAL Script. +- Keep JSON BLOCKS short. Consider both the total execution time and the total number of scripts. In general, you do not want a single BLOCK to take more than 1hr of execution time. You also should avoid having BLOCKS of more than 100 individual scripts, at the time of writing there is no automatic method to recover and execute a BLOCK from an individual step in the case of failure. +- For use cases where execution time is paramount, consider developing a new SAL Script. JSON BLOCKS execute SAL Scripts in sequence so there is some added system latency compared to a single SAL Script executing the same commands. +- Recall that JSON BLOCKS are not configurable, so if your test requires changing multiple parameters consider using individual SAL Scripts or a different method. + +.. _Writing-JSON-BLOCK-Testing-and-Validating: + +Testing and Validating JSON BLOCKS +---------------------------------- + +At the time of enabling the Scheduler, the BLOCK library is validated for both proper JSON format and individual script configurations are checked against their schema. +If there is an error, the JSON BLOCK will not be available to add to the Scheduler for observation. +Therefore, it is required that all JSON BLOCKS be validated using one of the test stands described in the `Operational Environments`_ before deployment at the summit. +Separate documentation will be developed describing this process. + +.. _Operational Environments: https://obs-ops.lsst.io/Observing-Interface-Setup/environments.html diff --git a/index.rst b/index.rst index 3776924..771dde4 100644 --- a/index.rst +++ b/index.rst @@ -72,6 +72,26 @@ Users can interface with them either using the :ref:`LOVE interface