Compliance job should not push to the repo

[gioccher]

Environment

• Python version: 3.9.X
• Nautobot version: 2.3.13
• nautobot-golden-config version: 2.2.1

Expected Behavior

If the Intended Config and Compliance jobs run around the same time, the compliance job only updates the database based on its findings.

Observed Behavior

If the Intended Config and Compliance jobs run around the same time, the Compliance job might push to the repository the intended config generated by the other job and make the intended config job misbehave.

When the Compliance job steals the commit from the Intended config job, nautobot is not aware that there is an updated intended config: clicking on the "Intended" or "Configuration Postprocessing" links from the "Configuration Types" panel in the device details page in nautobot shows the old configurations.
I had to run the Generate Intended Configurations job again to make Nautobot realize the intended configuration changed. Nothing new got pushed to git but then nautobot was able to show the updated intended and postprocessed configs

Steps to Reproduce

1. the "Generate Intended Configurations" job started at 23:44:56 and ended at 23:45:07. This was started on-demand.
2. the "Perform Configuration Compliance" job started at 23:45:00 and ended at 23:45:08. This is a scheduled job running hourly.
3. The updated config got committed to git with the commit message "PERFORM CONFIGURATION COMPLIANCE JOB 2025-01-02 23:45:05.426900+00:00" instead of the usual "GENERATE INTENDED CONFIGURATIONS JOB "

[itdependsnetworks]

@gioccher I think it is fair to say the steps to reproduce is, you need to add the step "Create only a single git data source for intended and compliance".

[gioccher]

I don't have any repository specified for Compliance.
I have one Intended, one for Backup, and one for Templates - all different.

The part that seems the most odd to me is why the compliance job would need to push to any repository as part of the job. It should pull from Backup and Intended to get the latest configuration files, and perhaps from Compliance to grab the latest rules. But why would it push anything?

[itdependsnetworks]

ok, I think you are saying there is a race condition in which you cannot run the same job until the prior one is complete. We will soon have the ability to mark a job as is_singleton, however, these are technically different jobs.

If that is the case, we would need capabilities in core to extend that.

[gioccher]

Yeah I'm saying that the "Generate Intended Configurations" and "Perform Configuration Compliance" jobs are incompatible with each other.

I can see two solutions:

• make them compatible by preventing "Perform Configuration Compliance" from pushing to the Intended configurations repository (I still don't understand why it would ever need to do that)
• add support to core to extend is_singleton to multiple jobs

[itdependsnetworks]

Would this be more accurate?

the "Generate Intended Configurations" and "Perform Configuration Compliance" jobs are incompatible to run at the same time as each other.

If so, yes, it makes sense that they are not intended to run at the same time, this would be true for anything touching Git, but more likely to happen in this use case than in most use cases.

[glennmatthews]

FWIW I don't know that a core enhancement is necessarily needed - the same approach there could easily be elaborated upon by Golden Config's specific jobs, i.e. having a shared Redis lock that both jobs set and check for.