Canary checks for simulations in CI

.github/README.md

@@ -0,0 +1,86 @@
+# Table of Contents
+- [Description](#description)
+- [Directory Structure](#directory-structure)
+- [Workflow jobs](#workflow-jobs)
+- [Synchronous jobs](#1-cron-jobs)
+- [PR verification jobs](#2-pr-verification-jobs)
+- [Flow result checks](#flow-result-verification)
+# Description
+
+OpenFASoC makes use of the Github actions based CI workflow. Through the use of YAML workflow files and pythonic verification scripts (`parse_rpt.py`), the changes made in pull requests are verified.
+
+<b>Disclaimer: specific functionality information is present as docstrings in each file</b>
+
+# Directory Structure 
+
+```
+.github
+├── scripts
+│   ├── dependencies
+│   │   ├── get_tag.py
+│   │   └── tool_metadata.py
+│   ├── expected_drc_reports
+│   │   └── expected_ldo_drc.rpt
+│   ├── expected_sim_outputs
+│   │   ├── cryo-gen
+│   │   ├── ldo-gen
+│   │   └── temp-sense-gen
+│   │       ├── postPEX_sim_result
+│   │       └── prePEX_sim_result
+│   ├── Dockerfile
+│   ├── get_docker_config.py
+│   ├── gh.py
+│   ├── parse_rpt.py
+│   ├── tool_metadata.py
+│   ├── tool.py
+│   └── update_tools.py
+└── workflows
+    ├── cryo_gen.yml
+    ├── ldo_sky130hvl.yml
+    ├── tempSense_sky130hd.yml
+    ├── test_python_api.yml
+    └── verify_latest_tools_version.yml
+```
+# Workflow jobs
+The repo supports 2 kinds of workflow runs
+
+## 1. CRON jobs
+- `workflows/verify_latest_tools_version.yml`
+
+    This job automatically runs at 1 A.M. UTC and is used to verify functionality with the latest version of the toolset that the flow uses. 
+    It makes use of the `workflows/verify_latest_tools_version.yml` file and builds a docker <i>alpha</i> image with the latest version of the toolset. 
+
+    Should the generator workflow runs (temp-sense, cryo-gen and ldo-gen) run to completion, the <i>alpha</i> image is tagged as <i>stable</i> and pushed to the dockerhub, which will thereafter be used for PR verification runs
+
+    The toolset version info is also updated in the required places in the repository with an automated PR, issued by the same `.yml` file.
+- `workflows/test_python_api.yml`
+
+    This job runs at 2 A.M. UTC. Using `pytest`, the common python API developed for the temperature sensor generator. 
+
+    Verilog generation and simulations are run for the temperature sensor generator for python versions `3.8`, `3.9`, and `3.10`.
+
+    Successful completion of all jobs is required to guarantee that the tool is working properly.
+## 2. PR verification jobs
+These workflow jobs, which include - 
+* `cryo_gen.yml`
+* `tempSense_sky130hd.yml`
+* `ldo_sky130hvl.yml`
+
+are run to verify the potential changes made to the repository via pull requests. 
+
+These workflows run the verilog generation and openroad flows for the 3 generators and spice simulations are run specifically for the temperature sensor generator.
+
+# Flow Result Verification
+
+The generator flows run by the CI workflow jobs specified in the `.yml` files produce results which are verified by the `scripts/parse_rpt.py` file.
+
+The script checks - 
+* DRC results
+* LVS results
+* OpenROAD flow results
+    - existence of `.cdl, .sdc, .def, .gds, .v, .spice`, etc. files in the right directories
+    - existence of optimum configuration search result `.csv` files for the temp-sense generator
+* Simulation results for temp-sense
+    - if the generated frequency, power and error results are within an allowable range from the stored simulation results
+    - if the number of failed simulations returned by the run is 0
+    - if the run result files (`.sp, .log and parameters.txt`) exist for all inverter-header configurations

.github/scripts/parse_rpt.py

@@ -1,7 +1,7 @@
 """
 This script performs checks on generated files and reports for different generators based on workflow parameters.
 
-The `_generator_is` variable is a dictionary with keys and values indicating information about the workflow being run.
+The `generator_is` variable is a dictionary with keys and values indicating information about the workflow being run.
 Values for each key are determined by how this script is called by the .yml files in .github/workflows.
 
 The `cryo_library` variable is used to determine which library (sky130hd_cryo, sky130hs_cryo, sky130hvl_cryo) the workflow is targeting.
@@ -31,37 +31,38 @@
 import warnings
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
-from common.get_ngspice_version import check_ngspice_version
+sys.path.append(os.path.join(os.path.dirname(__file__), '../common'))
 from common.check_gen_files import check_gen_files
-from common.check_sim_results import compare_files
+from simulation.classify_sim_error import classify_sim_error
+from tools.compare_files import compare_files
 
 sys.stdout.flush()
 
 cryo_library = ""
-_generator_is = {
+generator_is = {
     'sky130hvl_ldo': 0, 
     'sky130hd_temp': 0, 
     'sky130XX_cryo': 0
 }
 
 if len(sys.argv) == 1:
-    _generator_is['sky130hd_temp'] = 1
+    generator_is['sky130hd_temp'] = 1
 elif len(sys.argv) > 1:
     if sys.argv[1] == 'sky130hvl_ldo':
-        _generator_is['sky130hvl_ldo'] = 1
+        generator_is['sky130hvl_ldo'] = 1
     elif sys.argv[1] == 'sky130hd_temp_full':
-        _generator_is['sky130hd_temp'] = 1
+        generator_is['sky130hd_temp'] = 1
     else:
-        _generator_is['sky130XX_cryo'] = 1
+        generator_is['sky130XX_cryo'] = 1
 
-if _generator_is['sky130XX_cryo']:
+if generator_is['sky130XX_cryo']:
     # check which cryo-gen library's workflow is being run
     dir_path = r'flow/reports'
     lib = os.listdir(dir_path)
     cryo_library = str(lib[0])
 
 ## DRC and LVS Filename Declaration 
-if _generator_is['sky130hd_temp'] or _generator_is['sky130hvl_ldo']:
+if generator_is['sky130hd_temp'] or generator_is['sky130hvl_ldo']:
     drc_filename = "work/6_final_drc.rpt"
     lvs_filename = "work/6_final_lvs.rpt"
 elif len(sys.argv) > 1 and sys.argv[1] == cryo_library:
@@ -70,7 +71,7 @@
 
 
 ## DRC check 
-if _generator_is['sky130hvl_ldo']:
+if generator_is['sky130hvl_ldo']:
     expected_ldo_rpt_filename = "../../../.github/scripts/expected_drc_reports/expected_ldo_drc.rpt"
     with open(drc_filename) as f1, open(expected_ldo_rpt_filename) as f2:
         content1 = f1.readlines()
@@ -107,12 +108,12 @@
             print("LVS is clean!")
 
 ## Result File Check
-if _generator_is['sky130hvl_ldo']:
+if generator_is['sky130hvl_ldo']:
     json_filename = "spec.json"
 else:
     json_filename = "test.json"
 
-if check_gen_files(json_filename, _generator_is, cryo_library):
+if check_gen_files(json_filename, generator_is, cryo_library):
     print("Flow check is clean!")
 else:
     print("Flow check failed!")
@@ -121,20 +122,13 @@
     sim_state_filename = "work/sim_state_file.txt"
     result_filename = "work/prePEX_sim_result" 
     template_filename = "../../../.github/scripts/expected_sim_outputs/temp-sense-gen/prePEX_sim_result"
-    max_allowable_error = 0.5
 
     ### Generated result file check against stored template
-    ngspice_version_flag = check_ngspice_version()
-    file_comp_flag = compare_files(template_filename, result_filename, max_allowable_error)
-
-    if ngspice_version_flag == 1 and file_comp_flag == 0:
-        raise ValueError("Ngspice version matches but sim results do not...sims failed!")
-    elif ngspice_version_flag == 0 and file_comp_flag == 0:
-        warnings.warn("The ngspice version does not match, "
-                       "simulation results might not match! "
-                       "Please contact a maintainer of the repo!", DeprecationWarning)
-    elif ngspice_version_flag == 0 and file_comp_flag == 1:
-        warnings.warn("The ngspice version does not match!", DeprecationWarning)
+    sim_error_type = classify_sim_error(template_filename, result_filename, compare_files)
+    if sim_error_type == 'red':
+        raise ValueError("Simulation results do not match with those in stored template file!")
+    elif sim_error_type == 'amber':
+        warnings.warn("Simulation results are within an allowable error range from template files!")
 
     ### Number of failed simulations returned from simulation state check
     sim_state = json.load(open("work/sim_state_file.txt"))

docs/source/ci.rst

@@ -17,3 +17,5 @@ If the CI works, the tool versions will be  mentioned in the README file beside
 Alpha image of OpenFASoC environment - `docker pull msaligane/openfasoc:alpha`
 
 Stable image of OpenFASoC environment - `docker pull msaligane/openfasoc:stable`
+
+**More detailed information about the CI and its workflows can be found in the README under the `.github` folder in the repository**

docs/source/common-python-api.rst

@@ -208,4 +208,72 @@ Functions
                 runs_dir=f"run/prePEX_inv{num_inv}_header{num_header}/",
                 sim_tool=simTool,
                 netlist_path=dstNetlist
-            )
+            )
+
+3. Generated File Check (``common.check_gen_files.py``)
+#######################################################
+This module is used to check for the presence of the required non-report files that each generator creates. It gets
+the module_name (str) from the .json file present in the generator top-level folder.
+
+Functions
+^^^^^^^^^
+1. check_gen_files(json_filename: str, generator_is: dict, cryo_library: str) -> int:
+
+    Arguments:
+
+        - json_filename (str): String containing the name of the .json filename for each generator
+        - generator_is (dict): Dictionary containing key-value pairs that signify which generator's flow results are being checked
+        - cryo_library (str): String containing which cryo-gen library (sky130hs, sky130hd, sky130hvl) is being checked for
+
+    Uses:
+
+        - work_dir (str): String containing the directory in which to check files
+        - data (str): String containing data from the .json file
+        - module_name (str): String containing the name of module that the check is being done for (eg. tempsenseInst_error)
+        - extension_file_path (str): Contains the extensions of the files which each generator produces for the flows
+
+    Returns:
+
+        - 1: if all checks are successful
+        - else: raises a ValueError: If any of the various checks go wrong (.csv file checks for temp-sense, flow generated files for all generators)
+
+    Specifics: 
+
+        After checking for the existence of the **work/** directory, the function does the following - 
+
+            1. checks for the presence of files generated by the openroad flow with the following piece of code 
+
+            .. code-block:: python
+
+                if os.path.exists(extension_file_path):
+                    with open(extension_file_path) as f:
+
+                        for extension in f:
+                            extension = extension.strip()
+                            if (generator_is['sky130XX_cryo']) and (extension == ".spice" or extension == "_pex.spice" or extension.strip() == "_sim.spice"):
+                                file = "./flow/" + module_name + extension.strip() 
+                            else:
+                                file = "".join([filename, extension])
+                            if (os.path.exists(file) == 0):
+                                raise ValueError(file + " does not exist!")
+                else: 
+                    print("checking flow results with possibly stale list of extensions...")
+                    extensions = [".sdc", ".gds", ".def", ".spice", ".v", "_pex.spice"]
+                    for extension in extensions:
+                            extension = extension.strip()
+                            if (generator_is['sky130XX_cryo']) and (extension == ".spice" or extension == "_pex.spice"):
+                                file = "./flow/" + module_name + extension 
+                            else:
+                                file = "".join([filename, extension])
+
+                            if (os.path.exists(file) == 0):
+                                raise ValueError(file + " does not exist!")
+
+            2. checks for the presence of the **.csv** files generated by the header-inverter configuration power/error optimisation, which contain the best configuration and the sensor power/error outputs for run of the configuration optimisation with the following code snipper
+
+            .. code-block:: python
+
+                if generator_is['sky130hd_temp']:
+                    for file in ("error_within_x.csv", "golden_error_opt.csv", "search_result.csv"):
+                        if os.path.exists(file) == 0:
+                            raise ValueError(file + " does not exist!")

openfasoc/generators/common/README.md

@@ -0,0 +1,57 @@
+# Table of Contents
+- [Description](#description)
+- [File Tree](#file-tree)
+- [Verilog Generation](#verilog-generation)
+- [Simulation runs](#simulations)
+- [Flow result checks](#flow-result-checks)
+- [Simulation result checks](#simulation-result-checks)
+
+# Description
+This folder contains python script files that are common to all generators, used as config or result parse files for generator flows and simulations
+
+### File Tree
+```
+generators
+└── common
+    ├── simulation
+    │   ├── simulation_config.py
+    │   ├── simulation_run.py
+    │   ├── utils.py
+    │   └── __init__.py
+    ├── __init__.py
+    ├── check_gen_files.py
+    ├── classify_sim_error.py
+    ├── get_ngspice_version.py
+    └── verilog_generation.py
+
+```
+### Verilog Generation
+The file `verilog_generation.py` is used to convert verilog files such that they use the mako templating library for simpler and more readable syntax. Specific function descriptions are present in as docstrings in the file.
+
+### Simulations
+The files found in the `simulation/` directory are used as pythonic script files to run simulations for each generator. These files mainly generate configurations and run files for the simulations, which are used by each of the generators. Specific function descriptions are found in the respective files
+
+### Flow result checks
+The file `check_gen_files.py` is used to check if simulations can be run correctly for a generator. Specifically, it is used in the `parse_rpt.py` file found in the `tools/` folder of each generator. This file runs at the end of each generator flow to check for successful completion.       
+
+Only temp-sense-gen, cryo-gen and ldo-gen are currently supported. For these generators, this file checks if the necessary `work/` directory and the simulation generated files are present (such as `.sdc`, `.cdl`, `.gds`, `.def`, among others).  
+
+The file also checks if the necessary optimum inverter-header configuration search results are present for the temp-sense-gen flow (in the form of the error optimisation `.csv` files)
+
+Check the docstrings in each file for specific function definitions.
+### Simulation Result Checks
+The files `classify_sim_error.py` and `get_ngspice_version.py` together, are used to check for errors in the simulation runs for each generator. These files use a dictionary of maximum and minimum allowable deviations of simulation results from an ideal set of result files present in `.github/scripts/expected_sim_outputs/*`. The dictionary of deviations, called "errors" is used for the same
+```python
+errors = {
+        'frequency': { 'max': 1, 'min': 0.5 },
+        'power': { 'max': 1000, 'min': 1000 },
+        'error': { 'max': 100, 'min': 50 },
+    }
+```
+If the deviation  of the current run results (in percentage) is greater than the maximum allowable deviation for any of the results, the file returns an urgent "red" alert, which raises a ValueError in the `parse_rpt.py` file.  
+
+If the deviations lie between the maximum and minimum allowable deviations, the script returns an "amber" alert, which raises a soft warning.  
+
+If the deviations are all less than minimum allowable deviation, the script returns "green", which does not reflect anything in the `parse_rpt.py` file. 
+
+If the current ngspice version does not match with the ngspice version stored at the time the templates were stored, the function `check_ngspice_version()` from `check_ngspice_version.py` returns 0, which leads to a soft warning raised in the `parse_rpt.py` file, to notify the maintainers about the same.

openfasoc/generators/common/check_gen_files.py

@@ -1,10 +1,10 @@
 """
-This script is used to check for the presence of the required non-report files that each generator creates. It gets
+This module is used to check for the presence of the required non-report files that each generator creates. It gets
 the module_name (str) from the .json file present in the generator top-level folder.
 
 Args:
     json_filename (str): String containing the name of the .json filename for each generator
-    _generator_is (dict): Dictionary containing key-value pairs that signify which generator's flow results are being checked
+    generator_is (dict): Dictionary containing key-value pairs that signify which generator's flow results are being checked
     cryo_library (str): String containing which cryo-gen library (sky130hs, sky130hd, sky130hvl) is being checked for
 Uses: 
     work_dir (str): String containing the directory in which to check files
@@ -20,15 +20,15 @@
 import json
 import os
 
-def check_gen_files(json_filename, _generator_is, cryo_library) -> int:
+def check_gen_files(json_filename, generator_is, cryo_library) -> int:
     with open(json_filename) as file:
         data = json.load(file)
 
     # print('Found .json config file...')
 
     module_name = data.get("module_name", "default")
 
-    if _generator_is['sky130XX_cryo']:
+    if generator_is['sky130XX_cryo']:
         work_dir = "./work/" + cryo_library + "/"
     else:
         work_dir = "./work/"
@@ -43,25 +43,27 @@ def check_gen_files(json_filename, _generator_is, cryo_library) -> int:
             with open(extension_file_path) as f:
 
                 for extension in f:
-                    if (_generator_is['sky130XX_cryo']) and (extension.strip() == ".spice" or extension.strip() == "_pex.spice" or extension.strip() == "_sim.spice"):
+                    extension = extension.strip()
+                    if (generator_is['sky130XX_cryo']) and (extension == ".spice" or extension == "_pex.spice" or extension.strip() == "_sim.spice"):
                         file = "./flow/" + module_name + extension.strip() 
                     else:
-                        file = "".join([filename, extension.strip()])
+                        file = "".join([filename, extension])
                     if (os.path.exists(file) == 0):
                         raise ValueError(file + " does not exist!")
         else: 
             print("checking flow results with possibly stale list of extensions...")
             extensions = [".sdc", ".gds", ".def", ".spice", ".v", "_pex.spice"]
             for extension in extensions:
-                    if (_generator_is['sky130XX_cryo']) and (extension.strip() == ".spice" or extension.strip() == "_pex.spice"):
-                        file = "./flow/" + module_name + extension.strip() 
+                    extension = extension.strip()
+                    if (generator_is['sky130XX_cryo']) and (extension == ".spice" or extension == "_pex.spice"):
+                        file = "./flow/" + module_name + extension 
                     else:
-                        file = "".join([filename, extension.strip()])
+                        file = "".join([filename, extension])
 
                     if (os.path.exists(file) == 0):
                         raise ValueError(file + " does not exist!")
     # print("Found necessary work result files!")
-    if _generator_is['sky130hd_temp']:
+    if generator_is['sky130hd_temp']:
         for file in ("error_within_x.csv", "golden_error_opt.csv", "search_result.csv"):
             if os.path.exists(file) == 0:
                 raise ValueError(file + " does not exist!")