Feature internal 46 logging (#403)

* Adding logging framework for STIGS * Made changes to the test yaml files for agg_stat * Syntax edit * Needed indentation * Synatx edits to agg_stat * Chnaged import statement * Changed import statement * Needed a line of code to be added * Added logging statements and modified tests accordingly * Syntax edit to logger variable as a function argument * Syntax edits * Syntax edits based on pytest outputs * Changes to safe_log function * Positional argument changes for logger * Syntax edits * Edit to a function call * Added logger to test_event_equalize * Changed placement of logger argument * Edits to placement of logger argument * Logging added to utils and metcalcpy by using safe_log in util * Syntax edits * Syntax edits wrt ValueError * Syntax edits to NameError * Minor syntax edits * Minor syntax edits * Minor edits with safe_log * Minor edits to safe_log in agg_eclv * Minor edit to event_equalize in utils * Syntax edits to safe_log * Syntax edits * Added missing bracket * New syntax edits * Edits to Aplin * Final edit to agg_stat * Edits based on SonarQube findings * adding new logging directions * first page of google doc added * fixing indenting and code block to python * Removed extra debug statements * Update actions/upload-artifact@v2 to be actions/upload-artifact@v4 * adding second page of documentation * adding spacing for better presentation * Trying to fix Enumerated list ends without a blank line warning * fixing typo * adding 3rd page * adding final page four * fixing formatting * fixing formatting * Changing to 3rd person and removing "you" * replacing "their" and bolding file name. * changing to METplus specifications * Resolving final conflict * Attempt to resolve at least some doc errors. * Moved logging.rst to correct location * Removing What's New header * Removed other references to new * Fixed typo and removed reference to now * Removed duplicate reference in list to log_level and moved up description * Removed periods from incomplete sentence descriptions * Modified the name of the headings to eliminate the name of the script as the script is indicated in the description. This allows for cleaner headers. * Formatting * Removing logging.rst from an area I accidentally added it to --------- Co-authored-by: Ishita Srivastava <[email protected]> Co-authored-by: ishitas <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]>
dtcenter · Oct 7, 2024 · 34cc8e2 · 34cc8e2
1 parent 554d4f6
commit 34cc8e2
Show file tree

Hide file tree

Showing 3 changed files with 97 additions and 44 deletions.
diff --git a/docs/Users_Guide/logging.rst b/docs/Users_Guide/logging.rst
@@ -1,26 +1,22 @@
-****************
-Logging In Guide
-****************
+*******
+Logging
+*******
 
 
-This guide provides a comprehensive overview of the newly integrated logging capabilities 
+This guide provides a comprehensive overview of the integrated logging capabilities 
 within METcalcpy. These enhancements are designed to provide users with valuable insights 
 into the application's execution, aiding in tasks such as debugging, performance monitoring, 
 and understanding the operational flow of the program.
 
+Centralized Logging Configuration
+=================================
 
-What's New
-==========
-
-Centralized Logging Configuration (**logging_config.py**):
-----------------------------------------------------------
-
-A new script, **logging_config.py**, has been introduced to centralize the management of logging 
+A script, **logging_config.py**, has been introduced to centralize the management of logging 
 configurations. This approach ensures consistency and simplifies the maintenance of logging 
 settings across all modules within METcalcpy.
 
 
-* Key Feature: :code:`setup_logging` function
+* **Key Feature:** :code:`setup_logging` **Function**
 
   * The :code:`setup_logging` function is the core of **logging_config.py**. It initializes 
     and configures the logger instance based on parameters specified in a YAML configuration 
@@ -48,65 +44,120 @@ which contains logging configurations from a YAML file such as
 the user's settings.
 
 YAML-Driven Configuration
--------------------------
+=========================
 
-METcalcpy now allows users to customize logging behavior directly within 
-their YAML configuration files, eliminating the need for hardcoding 
+METcalcpy allows users to customize logging behavior directly within 
+the user's YAML configuration files, eliminating the need for hardcoding 
 logging settings in Python scripts.
 
 **Key Parameters in YAML Configuration:**
 
-:code:`log_dir:` Specifies the directory where log files are stored.
-
-:code:`log_filename:` Defines the name of the log file.
-
-:code:`log_level:` Determines the verbosity of the log output. 
-Available levels are :code:`DEBUG, INFO, WARNING, and ERROR:`.
-
-:code:`log_level:` By setting the appropriate log level in your YAML configuration 
-file (e.g., log_level: WARNING), you can control the verbosity of the log output, 
-ensuring that only the necessary information is recorded.
+| :code:`log_dir:` Specifies the directory where log files are stored.
+| :code:`log_filename:` Defines the name of the log file.
+| :code:`log_level:` Determines the verbosity of the log output. 
+  Available levels are DEBUG, INFO, WARNING, and ERROR. By setting the appropriate
+  log level in the YAML configuration file (e.g., log_level: WARNING), the user can
+  control the verbosity of the log output, ensuring that only the necessary
+  information is recorded.
 
 METcalcpy supports the following log levels:
 
-  1. **DEBUG:**
+1. **DEBUG:**
 
-    * **Purpose:** Captures detailed information for diagnosing issues.
-    * **Use Case:** Ideal during development or troubleshooting to see all 
-      the internal workings of the application.
+  * **Purpose:** Captures detailed information for diagnosing issues
+  * **Use Case:** Ideal during development or troubleshooting to see all the internal workings of the application
 
-  2. **INFO:**
+2. **INFO:**
 
-    * **Purpose:** Records general information about the application's execution.
-    * **Use Case:** Suitable for tracking the progress and key events 
-      in the application's workflow without overwhelming detail.
+  * **Purpose:** Records general information about the application's execution
+  * **Use Case:** Suitable for tracking the progress and key events in the application's workflow without overwhelming detail
 
-  3. **WARNING:**
+3. **WARNING:**
 
-    * **Purpose:** Logs potential issues that are not immediately critical but 
-      could lead to problems.
-    * **Use Case:** Useful for highlighting areas that may require attention 
-      but don't stop the application from running.
+  * **Purpose:** Logs potential issues that are not immediately critical but could lead to problems
+  * **Use Case:** Useful for highlighting areas that may require attention but doesn't stop the application from running
 
-  4. **ERROR:**
+4. **ERROR:**
 
-    * **Purpose:** Captures serious issues that prevent parts of the 
-      application from functioning correctly.
-    * **Use Case:** Necessary for logging events that require immediate 
-      attention and could cause the application to fail or produce incorrect results.
+  * **Purpose:** Captures serious issues that prevent parts of the application from functioning correctly
+  * **Use Case:** Necessary for logging events that require immediate attention and could cause the application to fail or produce incorrect results
 
 Informative Log Formatting
---------------------------
+==========================
 
 Log messages in METcalcpy are meticulously formatted to include detailed information, 
 improving readability and facilitating easier analysis of log data.
 
+**Standard Log Format Includes:**
+
+  * **Timestamp (UTC):** Each log message is tagged with a UTC timestamp 
+    (e.g., :code:`2023-12-19 18:20:00 UTC`), ensuring consistent timekeeping across systems.
+  * **User ID:** The User ID of the script initiator is included, aiding traceability, 
+    particularly in multi-user environments.
+  * **Log Level:** Indicates the severity of the message (e.g., DEBUG, INFO, WARNING, ERROR)
+  * **Log Message:** The main content of the log entry, which may provide context 
+    about events or operations within the script
+
+Safe Logging Utility
+====================
+
+A utility function, :code:`safe_log`, is introduced in **safe_log.py** to 
+enhance the robustness of logging operations.
+
+  * **Functionality:**
+
+    * The :code:`safe_log` function ensures that logging does not become a point of failure. 
+      It checks if a logger object is properly configured before logging any message. If a logger 
+      is not available or an error occurs during logging, :code:`safe_log` handles the 
+      situation gracefully without interrupting the application's core functionality.
+
+Example Usage in **agg_stat.py**:
+
+.. code-block:: py
+
+  from metcalcpy.util.safe_log import safe_log
+
+  safe_log(self.logger, "info", "Successfully loaded data from ...")
+
+Signal Handling for Graceful Shutdown
+=====================================
+
+The **logging_config.py** script is equipped to handle unexpected 
+program terminations gracefully by setting up signal handlers.
+
+  * **Supported Signals:**
+
+    * **SIGINT:** Typically triggered by pressing :code:`CTRL+C` to interrupt the program
+    * **SIGTERM:** Sent by other processes to request the program to stop gracefully
+
+When these signals are intercepted, a message like "Received signal ... Shutting down." 
+is logged, providing insight into the cause of the termination. This feature is valuable 
+for debugging and system monitoring.
+
+How to Use Logging in METcalcpy
+===============================
+
+**Step 1: Configure Logging in the YAML File**
 
+Begin by opening the YAML configuration file (e.g., **val1l2_agg_stat.yaml**) 
+and insert the logging parameters at the top level of the YAML file:
 
+| :code:`log_dir: /path/to/your/log/directory`
+| :code:`log_filename: my_application_log.txt`
+| :code:`log_level: INFO`
 
+**Step 2: Execute METcalcpy Scripts**
 
+With logging configured in the YAML file, run the METcalcpy scripts as usual. 
+The logging system will automatically manage log files according to the user's 
+specified settings.
 
+**Additional Notes**
 
+  * **UTC Timestamps:** METcalcpy uses UTC for all log timestamps, ensuring consistency across systems and time zones.
+  * **Log File Appending:** Logs are appended to existing files when scripts are executed multiple times with the same configuration.
 
+**Example Log Entry:**
 
+:code:`2023-12-19 18:20:00 UTC | user123 | INFO | Data loading completed successfully.`
 
diff --git a/metcalcpy/util/mode_2d_arearat_statistics.py b/metcalcpy/util/mode_2d_arearat_statistics.py
@@ -545,6 +545,7 @@ def calculate_2d_arearat_fsa_osa(input_data, columns_names, logger=None):
             calculated statistic
             or None if some of the data values are missing or invalid
     """
+
     try:
         safe_log(logger, "debug", "Filtering data based on TWO_D_DATA_FILTER.")
         filtered_data = column_data_by_name_value(input_data, columns_names, TWO_D_DATA_FILTER)

diff --git a/metcalcpy/util/mode_3d_ratio_statistics.py b/metcalcpy/util/mode_3d_ratio_statistics.py
@@ -563,6 +563,7 @@ def calculate_3d_ratio_osa_fsa(input_data, columns_names, logger=None):
             calculated statistic as float
             or None if some of the data values are missing or invalid
     """
+
     try:
         safe_log(logger, "debug", "Filtering data based on THREE_D_DATA_FILTER.")
         filtered_data = column_data_by_name_value(input_data, columns_names, THREE_D_DATA_FILTER)