Check action performance with lot of commits

[josecelano]

The queue loads the messages from the git log output:

private async loadMessagesFromGit(): Promise<void> {
  await this.guardThatGitRepoHasBeenInitialized()

  const noCommits = !(await this.gitRepo.hasCommits()) ? true : false

  if (noCommits) {
    return
  }

  const logCommits = await this.gitRepo.log()

  const allMessages = CommittedMessageLog.fromGitLogCommits(logCommits.all)

  this.committedMessages = allMessages.filterCommitsByQueue(this.name)
}

I suppose SimpleGit loads all commits in memory when you use logCommits.all that could be a problem. I do not expect it to be too slow or too big for very big repositories with a lot of commits but at least we should know the limit. We could improve the process (if needed) with:

• Not loading all commits in memory and processing and filtering them on the fly.
• Stop processing commits once we load the active job (one-job version of the queue) or all active jobs (multiple-active-jobs version of the queue)
• Maybe, if we modify the nextJob() or we even add a new method to get a given job getJob(jobId: JobId) we might need to re-process the git log to find the information for the missing job if we do not have it in memory. In this case, the CommitMessageLog attribute of the queue would be only a cache of the full queue information.

I'm going to add this issue to the roadmap for a future release. This issue is not intended to be the solution for the potential bottleneck problem. If we decide to do something we can discuss the solution and create a new issue for the implementation.

@da2ce7 @yeraydavidrodriguez

[da2ce7]

Related: #81

[josecelano]

We have a test project that has been using git-queue since 2022-01-11. At the moment, It has 8440 commits. I've just run a test this way:

Install git-queue to use it locally

mkdir /tmp/Nautilus-Cyberneering
cd /tmp/Nautilus-Cyberneering
git clone [email protected]:Nautilus-Cyberneering/git-queue.git
cd git-queue
git checkout v1-beta
yarn install && yarn build

I'm using the same version we are using on that test project.

Install the test project

mkdir /tmp/Nautilus-Cyberneering
cd /tmp/Nautilus-Cyberneering
git clone [email protected]:Nautilus-Cyberneering/library-consumer.git

Execute git-queue to get the latest job

TIME='\t%E real,\t%U user,\t%S sys,\t%K amem,\t%M mmem' \
INPUT_QUEUE_NAME="Library Update [library-aaa]" \
INPUT_ACTION="next-job" \
  time node /tmp/Nautilus-Cyberneering/git-queue/dist/index.js

The output:

::group::Debug info
git_repo_dir: /tmp/Nautilus-Cyberneering/git-queue
gnupg_home_dir: /home/josecelano/.gnupg
::endgroup::
::group::Setting outputs
::set-output name=job_found::false
::endgroup::
	0:00.12 real,	0.07 user,	0.02 sys,	0 amem,	37056 mmem

I think the method could be a base idea for a benchmarking workflow mentioned here. But we can use a fixed test project or create it. Using an external fixture project would make the test faster.

[josecelano]

The demo project has more than 13K commits and it's working fine. I guess the bottleneck could be the memory consumption.