More concurrent modification exception avoidance

[jimklimov]

See #604, #601 and similar efforts.

This PR adds synchronization for operations done with LRM resources property, because during a Jenkins start with recent plugin version I've still occasionally got that ConcurrentModificationException. My best guess is that freePostMortemResources() (explicitly mentioned in the starting page stack trace) and perhaps deleteNotExistingNodes() locked horns.

The XML mentioned a number of resources, all are ephemeral=true (probably saved with a graceful shutdown, then slated for deletion during start-up as no living jobs referenced them). On that system, there are no lockable resources defined persistently.

Testing done

Deployed into the Jenkins instance that suffered the issue. It started and seems to work correctly (gotta run some builds that lock/unlock stuff, to be more sure), and cleaned away many of the ephemeral resources.

On one restart I saw two remaining (one claimed locked by a job that was aborted by the restart), on another one remained and did not disappear after I claimed and released it via UI.

Proposed upgrade guidelines

N/A

Localizations

• English
• German
• French
• Slovak
• Czech
• ...

Submitter checklist

• The Jira / Github issue, if it exists, is well-described.
• The changelog entries and upgrade guidelines are appropriate for the audience affected by the change (users or developers, depending on the change) and are in the imperative mood (see examples).
  • The changelog generator for plugins uses the pull request title as the changelog entry.
  • Fill in the Proposed upgrade guidelines section only if there are breaking changes or changes that may require extra steps from users during the upgrade.
• There is automated testing or an explanation that explains why this change has no tests.
• New public functions for internal use only are annotated with @NoExternalUse. In case it is used by non java code the Used by {@code <panel>.jelly} Javadocs are annotated.

• New or substantially changed JavaScript is not defined inline and does not call eval to ease the future introduction of Content Security Policy (CSP) directives (see documentation).
• For dependency updates, there are links to external changelogs and, if possible, full differentials.
• For new APIs and extension points, there is a link to at least one consumer.
• Any localizations are transferred to *.properties files.
• Changes in the interface are documented also as examples.

Maintainer checklist

Before the changes are marked as ready-for-merge:

• There is at least one (1) approval for the pull request and no outstanding requests for change.
• Conversations in the pull request are over, or it is explicit that a reviewer is not blocking the change.
• Changelog entries in the pull request title and/or Proposed changelog entries are accurate, human-readable, and in the imperative mood.
• Proper changelog labels are set so that the changelog can be generated automatically. See also release-drafter-labels.
• If the change needs additional upgrade steps from users, the upgrade-guide-needed label is set and there is a Proposed upgrade guidelines section in the pull request title (see example).
• java code changes are tested by automated test.

[mPokornyETM]

Hi, may you pls try this changes
#558
I merged it again with current master now.
That shall fixed all the racec as well.
The proposed changes here makes for me no sense. Because all the functions are private and lock is handled outside as well. That means something else is still not synchronized correctly.
We are running with the 558 over few mounts and we have no one concurent modification issue and no one job stuck.

[jimklimov]

Ok, will give that branch a shot. Not sure if I'd have much to tell until (if ever) it crashes, though :)

[mPokornyETM]

Ok, will give that branch a shot. Not sure if I'd have much to tell until (if ever) it crashes, though :)

That the reason, why it is still not merged. Because it runs in our environment, but it does not mean, that it runs elsewhere too.

[mPokornyETM]

It looks so, that this changes are no more necessary

[jimklimov]

Closing for now (maybe forever) :)