Support assembling and archiving repositories in parallel

[HannesWell]

Assembling and arching repositories is a relatively long running task and repository-projects are usually build as one of the last projects in the reactor. So the chances are relatively high, that (if multiple projects are present) multiple repositories are assembled at the same. Therefore it should be possible to build multiple repositories in parallel and there should not be one global lock for each repository-related mojo.

[github-actions]

Test Results

   561 files  ±0     561 suites  ±0   5h 11m 11s ⏱️ +48s
   364 tests ±0     358 ✔️ ±0    6 💤 ±0  0 ❌ ±0 
1 092 runs  ±0  1 073 ✔️ ±0  19 💤 ±0  0 ❌ ±0 

Results for commit f934b1e. ± Comparison against base commit e154832.

♻️ This comment has been updated with latest results.

[laeubi]

Tycho has a FileLockService can we probably reuse that here?

[HannesWell]

I looked into that, but since the FileLockService creates a lock-file in the directory to lock this is not usable in cases where for example a repository is zipped, because then the lock file ends up in the zipped repository.
Furthermore a inter-process file locking does not seem to be necessary because the resources are within the projects being build and the goal is only to prevent illegal concurrent access in the same build-process. One should not run multiple build processes at the same time for the same or even sub-directories any way. So having a file-locking seems to be too much for me and an in-memory locking is sufficient.

[HannesWell]

But with #2853 we could simply add another method like FileLockService.lockForThisProcess or similar that only does an ordinary in-process/memory locking without creating any files.

[laeubi]

I looked into that, but since the FileLockService creates a lock-file in the directory to lock this is not usable in cases where for example a repository is zipped, because then the lock file ends up in the zipped repository.

Have you tested this? The zipping of repository is taking place in a separate step where the lock file should already be deleted (or can be deleted).

One should not run multiple build processes at the same time for the same or even sub-directories any way

I think you are aware of murphy's law ;-)

So having a file-locking seems to be too much for me and an in-memory locking is sufficient.

If we can reuse something that offers even more we should use that instead of inventing another approach that offers weaker guarantees.

[laeubi]

Now you have refactored the lock do you think it could be used here?

[HannesWell]

I looked into that, but since the FileLockService creates a lock-file in the directory to lock this is not usable in cases where for example a repository is zipped, because then the lock file ends up in the zipped repository.

Have you tested this? The zipping of repository is taking place in a separate step where the lock file should already be deleted (or can be deleted).

I haven't tried it yet. But with this zipping the repo would also be guarded with a lock-file, which would create the file before the zipping starts and deletes it only afterwards. And in between the directory content is zipped. Therefore expect that the lock-file will be included. But I'll try it.

One should not run multiple build processes at the same time for the same or even sub-directories any way

I think you are aware of murphy's law ;-)

It could definitively happen, but one should not do it. So its then the users fault. :)

So having a file-locking seems to be too much for me and an in-memory locking is sufficient.

If we can reuse something that offers even more we should use that instead of inventing another approach that offers weaker guarantees.

In generally that's right. But as written above, I'm not yet sure if this is really suitable. If it is not suitable, I would just add the method as something like lockVirtually() to the FileLockerService.

[laeubi]

I think we do not loose anything using the function as-is ... any complication like "virtual" make it just confusing so if we zip the content it should be easy to exclude the lockfile anyways.

[HannesWell]

I would see the opposite as more complicated since the exclusion then would need to be kept in sync if the lock-file changes one day.
And which file is locked eventually is an implementation detail of the locker and consumer not interested in inter-process locking should not have to care about that.