compliance operator OSCAL harvest report, with test cases.

Improved doc string, comprising report --details. spit & polish. spit & polish. Support optional config parameters --start and --end copyright remove test fixtures README reduce LOCs README original README report README merge 0.11.0
ComplianceAsCode · Feb 17, 2021 · 170d233 · 170d233
1 parent 2ec0137
commit 170d233
Show file tree

Hide file tree

Showing 8 changed files with 578 additions and 31 deletions.
diff --git a/CHANGES.md b/CHANGES.md
@@ -1,45 +1,90 @@
-# 0.5.1
 
-- [IMPROVED] Updated template layout for Repository/Branch New Commits report.
-- [IMPROVED] Updated template layout for Repository/Branch/Filepath New Commits report.
+# [0.11.0](https://github.com/ComplianceAsCode/auditree-arboretum/releases/tag/v0.11.0)
 
-# 0.5.0
+- [ADDED] Kubernetes resources fetcher added.
 
-- [NEW] Add repository/branch new commits check.
-- [NEW] Add repository/branch/filepath new commits check.
-- [NEW] Add Github repository/branch/filepath recent commits fetcher.
-- [IMPROVED] Github repository/branch recent commits fetcher now fetches since evidence last update.
+# [0.10.0](https://github.com/ComplianceAsCode/auditree-arboretum/releases/tag/v0.10.0)
+
+- [ADDED] Organization repository direct collaborators check added to `permissions`.
+
+# [0.9.0](https://github.com/ComplianceAsCode/auditree-arboretum/releases/tag/v0.9.0)
+
+- [ADDED] Kubernetes resources fetcher added.
+
+# [0.8.1](https://github.com/ComplianceAsCode/auditree-arboretum/releases/tag/v0.8.1)
+
+- [FIXED] Added missing `__init__.py` file to `permissions/fetchers/github` folder.
+
+# [0.8.0](https://github.com/ComplianceAsCode/auditree-arboretum/releases/tag/v0.8.0)
+
+- [ADDED] GitHub org collaborators fetcher added to `Permissions`.
+
+# [0.7.1](https://github.com/ComplianceAsCode/auditree-arboretum/releases/tag/v0.7.1)
+
+- [FIXED] Github issues fetcher now uses `states` option matching the README writeup.
+
+# [0.7.0](https://github.com/ComplianceAsCode/auditree-arboretum/releases/tag/v0.7.0)
+
+- [ADDED] Folder hierarchy for Issue Management related fetchers, checks, and harvest reports added.
+- [ADDED] Github issues fetcher added to Issue Management.
+- [ADDED] Zenhub workspaces fetcher added to Issue Management.
+
+# [0.6.2](https://github.com/ComplianceAsCode/auditree-arboretum/releases/tag/v0.6.2)
+
+- [ADDED] Folder hierarchy for Permissions related fetchers, checks, and harvest reports added.
+
+# [0.6.1](https://github.com/ComplianceAsCode/auditree-arboretum/releases/tag/v0.6.1)
+
+- [FIXED] Auditree Abandoned Evidence check now tracks all evidence used by check.
+- [FIXED] Auditree Locker Repo Integrity check now tracks all evidence used by check.
+- [FIXED] Auditree Python Packages check now tracks all evidence used by check.
+
+# [0.6.0](https://github.com/ComplianceAsCode/auditree-arboretum/releases/tag/v0.6.0)
+
+- [ADDED] IBM Cloud cluster list fetcher added.
+
+# [0.5.1](https://github.com/ComplianceAsCode/auditree-arboretum/releases/tag/v0.5.1)
+
+- [CHANGED] Template layout for Repository/Branch New Commits report updated.
+- [CHANGED] Template layout for Repository/Branch/Filepath New Commits report updated.
+
+# [0.5.0](https://github.com/ComplianceAsCode/auditree-arboretum/releases/tag/v0.5.0)
+
+- [ADDED] Repository/branch new commits check added.
+- [ADDED] Repository/branch/filepath new commits check added.
+- [ADDED] Github repository/branch/filepath recent commits fetcher added.
+- [CHANGED] Github repository/branch recent commits fetcher now fetches since evidence last update.
 - [CHANGED] Github repo recent commits evidence: TTL set to 2 days for locker, 1 day all other repos.
 - [FIXED] Links to the `auditree-framework` in README.md files are correct now.
 
-# 0.4.0
+# [0.4.0](https://github.com/ComplianceAsCode/auditree-arboretum/releases/tag/v0.4.0)
 
-- [NEW] Add Github repository metadata fetcher.
-- [NEW] Add Github repository recent commits fetcher.
-- [NEW] Add Github repository branch protection fetcher.
-- [NEW] Add Evidence locker repository integrity checks.
-- [NEW] Add Evidence locker recent commits integrity checks.
+- [ADDED] Github repository metadata fetcher added.
+- [ADDED] Github repository recent commits fetcher added.
+- [ADDED] Github repository branch protection fetcher added.
+- [ADDED] Evidence locker repository integrity checks added.
+- [ADDED] Evidence locker recent commits integrity checks added.
 
-# 0.3.0
+# [0.3.0](https://github.com/ComplianceAsCode/auditree-arboretum/releases/tag/v0.3.0)
 
-- [BREAKING] Move `auditree` fetchers and checks up to arboretum.auditree.
-- [NEW] Add folder hierarchy for Ansible fetchers, checks, and harvest reports.
-- [NEW] Add folder hierarchy for Chef fetchers, checks, and harvest reports.
-- [NEW] Add folder hierarchy for IBM Cloud fetchers, checks, and harvest reports.
-- [NEW] Add folder hierarchy for Kubernetes fetchers, checks, and harvest reports.
-- [NEW] Add folder hierarchy for Object Storage fetchers, checks, and harvest reports.
-- [NEW] Add folder hierarchy for Pager Duty fetchers, checks, and harvest reports.
-- [NEW] Add folder hierarchy for Splunk fetchers, checks, and harvest reports.
+- [BREAKING] Moved `auditree` fetchers and checks up to arboretum.auditree.
+- [ADDED] Folder hierarchy for Ansible fetchers, checks, and harvest reports added.
+- [ADDED] Folder hierarchy for Chef fetchers, checks, and harvest reports added.
+- [ADDED] Folder hierarchy for IBM Cloud fetchers, checks, and harvest reports added.
+- [ADDED] Folder hierarchy for Kubernetes fetchers, checks, and harvest reports added.
+- [ADDED] Folder hierarchy for Object Storage fetchers, checks, and harvest reports added.
+- [ADDED] Folder hierarchy for Pager Duty fetchers, checks, and harvest reports added.
+- [ADDED] Folder hierarchy for Splunk fetchers, checks, and harvest reports added.
 
-# 0.2.0
+# [0.2.0](https://github.com/ComplianceAsCode/auditree-arboretum/releases/tag/v0.2.0)
 
-- [NEW] Add Python packages fetcher and check.
+- [ADDED] Python packages fetcher and check added.
 
-# 0.1.0
+# [0.1.0](https://github.com/ComplianceAsCode/auditree-arboretum/releases/tag/v0.1.0)
 
-- [NEW] Add compliance configuration fetcher and check.
+- [ADDED] Compliance configuration fetcher and check added.
 
-# 0.0.1
+# [0.0.1](https://github.com/ComplianceAsCode/auditree-arboretum/releases/tag/v0.0.1)
 
-- [NEW] Add abandoned evidence fetcher and check.
-- [NEW] Make the Auditree Arboretum library public.
+- [ADDED] Abandoned evidence fetcher and check added.
+- [ADDED] Made the Auditree Arboretum library public.
diff --git a/arboretum/__init__.py b/arboretum/__init__.py
@@ -14,4 +14,4 @@
 # limitations under the License.
 """Arboretum - Checking your compliance & security posture, continuously."""
 
-__version__ = '0.5.1'
+__version__ = '0.11.0'
diff --git a/arboretum/kubernetes/README.md b/arboretum/kubernetes/README.md
@@ -23,6 +23,29 @@ Fetchers coming soon...
 
 Checks coming soon...
 
+## Reports
+
+### Compliance OSCAL Observations 
+
+* Report: [compliance_oscal_observations][compliance-oscal-observations]
+* Purpose: Create a JSON format report as a [NIST OSCAL Assessment Results][assessment-results] observations list from the kubernetes [OpenShift Compliance Operator][compliance-operator] data in the evidence locker.
+* Behavior: 
+    * A report is generated comprising a collection of observations, one for each [XCCDF][xccdf] rule/result pair discovered in the `cluster_resource.json` files with respect to the optional date range. Each observation may be enhanced in accordance with an optional `oscal_metadata.yaml` file.
+* Data files required:
+    * `raw/kubernetes/cluster_resource.json`, created by the kubernetes provider [ClusterResourceFetcher][fetch-cluster-resource].
+* Data files optional:
+    * `raw/kubernetes/oscal_metadata.json`, planted by the kubernetes provider account administrator.
+* Details/Config:
+
+   ```shell
+   harvest reports arboretum --detail compliance_oscal_observations
+   ```
+
+[compliance-oscal-observations]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/kubernetes/reports/compliance_oscal_observations.py
+[fetch-cluster-resource]: https://github.ibm.com/auditree/auditree-central/blob/master/auditree_central/provider/iks/fetchers/fetch_cluster_resource.py
+[assessment-results]: https://pages.nist.gov/OSCAL/documentation/schema/assessment-results-layer/assessment-results/
+[xccdf]: https://csrc.nist.gov/projects/security-content-automation-protocol/specifications/xccdf
+[compliance-operator]: https://github.com/openshift/compliance-operator/blob/master/README.md
 [auditree-framework]: https://github.com/ComplianceAsCode/auditree-framework
 [auditree-framework documentation]: https://complianceascode.github.io/auditree-framework/
 [usage]: https://github.com/ComplianceAsCode/auditree-arboretum#usage
diff --git a/arboretum/kubernetes/reports/compliance_oscal_observations.py b/arboretum/kubernetes/reports/compliance_oscal_observations.py
@@ -0,0 +1,227 @@
+# -*- mode:python; coding:utf-8 -*-
+# Copyright (c) 2020 IBM Corp. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+The compliance OSCAL observations report.
+
+A json report comprising NIST OSCAL Assessment Results Observations generated
+by processing compliance operator fetcher cluster_resource evidence. The
+embedded XML within the cluster_resource evidence is transformed to produce the
+report. If an optional oscal_metadata file is specified, then the report is
+enhanced accordingly.
+
+Provide the "start" and "end" optional configuration (--config) parameters
+as a JSON string, in "YYYYMMDD" format to define a date range for the evidence
+used to process the report.  If omitted, the default value is the current date.
+
+---------------
+Example usages:
+---------------
+
+> harvest report my-repo arboretum compliance_oscal_observations
+
+> harvest report my-repo arboretum compliance_oscal_observations \
+--config '{ \
+"oscal_metadata":"raw/kubernetes/oscal_metadata.yaml" \
+}'
+
+> harvest report my-repo arboretum compliance_oscal_observations \
+--config '{ \
+"cluster_resource":"raw/kubernetes/cluster_resource.json", \
+"oscal_metadata":"raw/kubernetes/oscal_metadata.yaml", \
+"start":"20200901", \
+"end":"20201231" \
+}'
+
+--------------------
+oscal_metadata.yaml:
+--------------------
+
+The oscal_metadata.yaml file comprises one or more mappings. Below is shown the
+format of a single mapping. The items in angle brackets are to be replaced with
+desired values for augmenting the produced OSCAL.
+
+The mapping whose <name> matches the [metadata][name] in the evidence for the
+corresponding embedded XML, if any, is used for augmenting the produced OSCAL.
+
+<name>:
+   namespace: <namespace>
+   subject-references:
+      component:
+         uuid-ref: <uuid-ref-component>
+         type: <component-type>
+         title: <component-title>
+      inventory-item:
+         uuid-ref: <uuid-ref-inventory-item>
+         type: <inventory-item-type>
+         title: <inventory-item-title>
+         properties:
+            target: <target>
+            cluster-name: <cluster-name>
+            cluster-type: <cluster-type>
+            cluster-region: <cluster-region>
+
+A sample oscal_metadata.yaml file with 2 mappings is shown below.
+
+ssg-ocp4-ds-cis-111.222.333.444-pod:
+   namespace: xccdf
+   subject-references:
+      component:
+         uuid-ref: 56666738-0f9a-4e38-9aac-c0fad00a5821
+         type: component
+         title: Red Hat OpenShift Kubernetes
+      inventory-item:
+         uuid-ref: 46aADFAC-A1fd-4Cf0-a6aA-d1AfAb3e0d3e
+         type: inventory-item
+         title: Pod
+         properties:
+            target: kube-br7qsa3d0vceu2so1a90-roksopensca-0000026b.iks.mycorp
+            cluster-name: ROKS-OpenSCAP-1
+            cluster-type: openshift
+            cluster-region: us-south
+ssg-rhel7-ds-cis-111.222.333.444-pod:
+   namespace: xccdf
+   subject-references:
+      component:
+         uuid-ref: 89cfe7a7-ce6b-4699-aa7b-2f5739c72001
+         type: component
+         title: RedHat Enterprise Linux 7.8
+      inventory-item:
+         uuid-ref: 46aADFAC-A1fd-4Cf0-a6aA-d1AfAb3e0d3e
+         type: inventory-item
+         title: VM
+         properties:
+            target: kube-br7qsa3d0vceu2so1a90-roksopensca-0000026b.iks.mycorp
+            cluster-name: ROKS-OpenSCAP-1
+            cluster-type: openshift
+            cluster-region: us-south
+"""
+
+import json
+from datetime import datetime, timedelta
+
+from harvest.reporter import BaseReporter
+
+from trestle.utils import osco
+
+import yaml
+
+
+class ComplianceOscalObservations(BaseReporter):
+    """The compliance oscal observations class."""
+
+    @property
+    def report_filename(self):
+        """Return the report filename."""
+        return 'compliance_oscal_observations.json'
+
+    def generate_report(self):
+        """
+        Generate the compliance oscal observations report content.
+
+        :returns: stringified OSCAL json content
+        """
+        # get required cluster resource path
+        path_cluster_resource = self.config.get(
+            'cluster_resource', 'raw/kubernetes/cluster_resource.json'
+        )
+        # get optional oscal_metadata path
+        path_oscal_metadata = self.config.get(
+            'oscal_metadata', 'raw/kubernetes/oscal_metadata.yaml'
+        )
+        # get start+end dates
+        start_dt = datetime.strptime(
+            self.config.get('start', datetime.today().strftime('%Y%m%d')),
+            '%Y%m%d'
+        )
+        end_dt = datetime.strptime(
+            self.config.get('end', datetime.today().strftime('%Y%m%d')),
+            '%Y%m%d'
+        )
+        if start_dt > end_dt:
+            raise ValueError('Cannot have start date before end date.')
+        current_dt = start_dt
+        previous = None
+        observation_list = []
+        # examine each day's evidence, if any
+        while current_dt <= end_dt:
+            try:
+                cluster_resource = json.loads(
+                    self.get_file_content(path_cluster_resource, current_dt)
+                )
+                try:
+                    oscal_metadata = yaml.load(
+                        self.get_file_content(path_oscal_metadata, current_dt),
+                        Loader=yaml.FullLoader
+                    )
+                    # add locker info to oscal metadata
+                    for key in oscal_metadata.keys():
+                        entry = oscal_metadata[key]
+                        entry['locker'] = self.repo_url
+                except Exception:
+                    oscal_metadata = None
+                # skip if no new evidence
+                if previous != cluster_resource:
+                    previous = cluster_resource
+                    # examine entries skipping those not relevant
+                    for key in cluster_resource.keys():
+                        for group in cluster_resource[key]:
+                            for cluster in cluster_resource[key][group]:
+                                for resource in cluster.get('resources', []):
+                                    self._update_observations(
+                                        observation_list,
+                                        resource,
+                                        oscal_metadata
+                                    )
+            except Exception:
+                pass
+            current_dt = current_dt + timedelta(days=1)
+        # create report
+        if len(observation_list) == 0:
+            raise RuntimeError('No report content.')
+        observation_dict = json.dumps(
+            {'observations': observation_list}, indent=2
+        )
+        report = str(observation_dict)
+        return report
+
+    def _update_observations(self, observation_list, resource, oscal_metadata):
+        """Update observations list with additional observations."""
+        if resource.get('kind') != 'ConfigMap':
+            return
+        if 'data' not in resource.keys():
+            return
+        if 'results' not in resource['data'].keys():
+            return
+        if 'metadata' not in resource.keys():
+            return
+        if 'name' not in resource['metadata'].keys():
+            return
+        # assemble osco data for transformation
+        data = {'results': resource['data']['results']}
+        osco_data = {
+            'kind': resource['kind'],
+            'data': data,
+            'metadata': resource['metadata']
+        }
+        # get OSCAL Observation objects
+        arp, analysis = osco.get_observations(osco_data, oscal_metadata)
+        # convert Observation objects into json
+        for observation_model in arp.observations:
+            observation_json = json.loads(
+                observation_model.json(
+                    exclude_none=True, by_alias=True, indent=2
+                )
+            )
+            observation_list.append(observation_json)
diff --git a/setup.cfg b/setup.cfg
@@ -23,6 +23,7 @@ packages = find:
 install_requires =
     auditree-framework>=1.2.3
     auditree-harvest>=1.0.0
+	compliance-trestle>=0.7.0
 
 [options.packages.find]
 exclude =