-
|
In our CI, cloning https://github.com/rust-lang/crates.io-index takes roughly 9 minutes: It seems to spend most of the time in This problem only happens on windows and does not occur on x86-64 Linux or MacOS. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 6 replies
-
|
Thanks a lot for reaching out. While perusing the CI log, I noticed that the log for object resolution doesn't account for any object. This should be fixed with the latest release of Further, what's slow is specifically the checkout portion of the clone operation, which creates one small file per crate at a rate of 249 files per second. It does so with efficiency in mind and can checkout the linux kernel in below a second on a linux machine, and the code for that is the same. Of course on windows there shouldn't be such a huge difference, especially since But why is I am conducting a test on windows now with the latest version from |
Beta Was this translation helpful? Give feedback.
-
|
After finally and seemingly (somewhat) successfully switching the Windows VM to The baseline is set by A local clone seems to take a little longer than a remote clone, fair enough, and overall we have 1m13s to beat. Unfortunately it's hard to see how long files take to checkout, but it's the majority of the time 'by feeling'. Now comes Overall, performance is much lower whenever IO is involved, 2m27s vs 1m14 for git. The When looking at the checkout speed, then it's definitely much slower than what Hypothesis about slow checkoutsI think the reason for this is that I truly admit that optimizing for windows is probably something that would happen at a much, much later stage, after Is there a workaround?Fortunately, all other operations are as fast as git or faster (pack resolution, for instance), and one would have to avoid checking out 120k files. Instead, these should be read directly from the git object database. As this uses a memory map I'd be thinking that the operations there are optimized by the kernel, so performance should mostly be comparable to unix systems. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you very much! Optimizing for windows is indeed hard and I heard that its I/O and fs perf is worse than Linux, so it's ok to be a bit slower on windows, but our windows CI took 9x the unix CI took, which is just a bit too much.
I think maybe IOCP can help? I was able to Also, I heard that Windows now also start to support io-uring, which is something worth considering since it would speedup check out both Linux and Windows, if that is the case.
That will be appreciated!
If there is anyway to lazily read objects as needed, then it would be awesome, we can also use this for |
Beta Was this translation helpful? Give feedback.
-
Indeed, this is why it's probably best to avoid cloning (with worktree) entirely on that platform or use
If you point me to the place where the full checkout is needed, I could probably figure out the rest from there as long as a git repository is present that contains the crates-index. It's not a big deal at all (probably not even famous last words ;)). |
Beta Was this translation helpful? Give feedback.
-
Here is the code we use for cloning and checkout. It is then used in In In |
Beta Was this translation helpful? Give feedback.
-
|
For example, the registry path of |
Beta Was this translation helpful? Give feedback.
After finally and seemingly (somewhat) successfully switching the Windows VM to
aarch64-*msvcI could produce some numbers.The baseline is set by
gititself, and writes are performed to the VM disk which probably isn't the fastest, but it's not sluggish either.