feat: add problog rule inference to build as code check

[sophie-bates]

Priority (v1)

• Update pyproject.toml with ProbLog dependency.
• Refactor build as code run_check method into smaller subchecks.
• Use ProbLog predicates to enable prolog string to access subcheck results.
• Write ProbLog rules to evaluate the check's overall confidence score.
• Update BuildAsCodeTable to receive the information.
• Compare overall confidence score with threshold to make a decision about check passing.
• Update check_result to store confidence score.
• Intermediate problog queries that allow a decision to be made on which deployment method to store data from.
• Store results from each ci_service and compare confidence_scores to produce final result. Currently just using the first CI service.
• Handle ProbLog errors properly.
• Move subchecks to be external to BAC check.
• Add additional subchecks.
• Fetch project name from setup.py file. Timestamp check requires the project name. This is straightforward to fetch for projects where it's specified in the pyproject.toml, but less so for those that use setup.py. I developed an initial implementation for setup.py but later removed it because I realised it was only applicable for very specified scenarios. The other option was to execute setup.py to fetch the name, but for obvious reasons we shouldn't do that.

Subchecks

• Timestamp of pypi package vs pipeline run - just need to finish with the github_actions function to fetch the 5 most recent runs of a workflow file.
• Workflow trigger
• Test deployment
• Secrets

Note: for many of the sub-checks we've had to do a check for deploy action method vs deploy command as there can be multiple methods used in a workflow. Need to figure out a more streamlined way to handle this.

Later

• Update expected outcomes for integration tests.
• Address the assumption that a project must have a deployment method found - this limited the inference. However, the challenge associated with this is that most sub-tasks rely on the existence of a particular workflow or workflow step to check. Perhaps could store information per workflow? and build up evidence in this way.
• We added tox -e release as a deploy command, however release is just a target pointing to a section in a tox.ini file, so potentially need to investigate parsing these files so verify that it contains deploy commands.
• Most of the functionality of Poetry overlaps with pip anyway, and there's little differentiation between the two in terms of project setup - i.e. Poetry projects can still use the same external GHAs etc. to deploy. Could be potential to have generic python class and use the inference to instead make a decision about which is used but not until later in the check.

Notes:

• Branch multiple-deploy-commands-collected-test stores some changes from the proof of concept for performing inference of multiple deployment commands (that was used in the Micronaut case study).

[tromai]

It's okay to use this built in function for now. But we should keep in mind that this function would raise TypeError when __dict__ attribute could not be found within the target object.

[sophie-bates]

Thanks for your comment Nhan, I'm actually about to push a change that will mean I'm no longer using this anyway so hopefully it's all good.

[tromai]

I am a bit confused by this attribute failed_check. Could you document what this attribute is used for and what is the value that it contains?
From a first look, I was expecting a boolean type instead of a float. Is this attribute the value that we should return if a sub check fails?

[sophie-bates]

Yes, it's the value that should be returned if the sub check fails.

[tromai]

We could define the justification inside the if branch so that we don't need to do it when self.ci_info["bash_commands"] is not available.

[behnazh-w]

At this line, if a CI service has a confidence_score larger than the threshold, but smaller than other CI services, wouldn't the check return the result for the smaller confidence_score?

[sophie-bates]

True. I can update this implementation to store all of the required information (i.e. deploy commands) for each ci_service and then populate the BuilasAsCodeTable and AnalyzeContext with whichever ci_service has the highest confidence_score?

[behnazh-w]

Sure, however I don't understand why AnalyzeContext needs to be updated?

[sophie-bates]

I just meant where the ci_info information is updated, i.e. here.

[behnazh-w]

Oh, that's the inferred provenance representation. We don't update ci_info itself.