The first command invocation on Windows is extraordinarily slow

[ofek]

Description

Between 8-25s to run a help command

Note:

• bash is faster but still affected, powershell is fastest
• Rust-compiled tools like Cargo and Rustup are significantly slower than Git or Python

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 18.04
• Ubuntu 20.04
• Ubuntu 22.04
• macOS 10.15
• macOS 11
• macOS 12
• Windows Server 2019
• Windows Server 2022

Image version and build link

https://github.com/ofek/vector/actions/runs/3456312722

Is it regression?

Unknown

Expected behavior

Minimal start-up penalty

Actual behavior

Extreme start-up penalty

Repro steps

name: Test

on: [push, pull_request]

jobs:
  bash:
    name: ${{ matrix.os }} / bash / ${{ matrix.command }}
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: ["windows-2019", "windows-2022"]
        command: ["cargo --help", "rustup --help", "python --help", "git --help"]

    steps:
    - run: ${{ matrix.command }}
      shell: bash
    - run: ${{ matrix.command }}
      shell: bash

  pwsh:
    name: ${{ matrix.os }} / pwsh / ${{ matrix.command }}
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: ["windows-2019", "windows-2022"]
        command: ["cargo --help", "rustup --help", "python --help", "git --help"]

    steps:
    - run: ${{ matrix.command }}
      shell: pwsh
    - run: ${{ matrix.command }}
      shell: pwsh

  powershell:
    name: ${{ matrix.os }} / powershell / ${{ matrix.command }}
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: ["windows-2019", "windows-2022"]
        command: ["cargo --help", "rustup --help", "python --help", "git --help"]

    steps:
    - run: ${{ matrix.command }}
      shell: powershell
    - run: ${{ matrix.command }}
      shell: powershell

[erik-bershel]

Hello @ofek,
We'll take a look at it.

[ofek]

Any update on this?

[erik-bershel]

@ofek we are working on this issue. I will share all information after I get something interesting.

[ofek]

Do you have any theories? I'm curious 🙂

[ilia-shipitsin]

Dear @ofek , I suppose that Windows Defender might be responsible for build delay.

I created sample repo with your pipeline definition.
After I added Set-MpPreference -DisableRealtimeMonitoring $true before your build steps, seems, timings are good

[ofek]

oh wow nice! note that shell: powershell is required rather than using the default shell or else it is as slow or slower than other commands running for the first time

[ofek]

Actually it looks like the machines are already set up that way https://github.com/al-cheb/virtual-environments/blob/main/images/win/scripts/Installers/Configure-Antivirus.ps1

It would be nice if we could get an official follow-up on this issue as it is globally wasting countless minutes

[myitcv]

FYI @ofek I independently raised #7320. See #7320 (comment) in particular, which points to action/investigation over at actions/checkout#1186.

It would be nice if we could get an official follow-up on this issue as it is globally wasting countless minutes

Note my response is not an official response, just a signpost to others issues.

[ofek]

Thanks for the links! I have (mostly) resolved this in our workflows by always using Bash

[myitcv]

Thanks for the links! I have (mostly) resolved this in our workflows by always using Bash

Interesting. This does not appear to influence our case, because all of our workflows on all platforms are bash, and yet we still see the significant slowness on Windows.

[ofek]

It's not a complete resolution for sure! But it is much faster than powershell.

[ilia-shipitsin]

we were able to find a difference between pwsh and powershell.
under the hood powershell builds modules analysis cache. that cache is kept in ephemeral user profile and created from scratch on provisioning runner.

by our measurement this: #8495 should reduce initial powershell run time from 20s to 8s

[mikhailkoliada]

Deployed now, closing

[ofek]

Thank you!