⚡ fetch org repositories in parallel #4970
Conversation
Signed-off-by: Salim Afiune Maya <[email protected]>
Signed-off-by: Salim Afiune Maya <[email protected]>
| break | ||
| } | ||
|
|
||
| // send as many request as workers we have |
There was a problem hiding this comment.
Why do we limit by number of workers? Doesn't this artificially throttle things
There was a problem hiding this comment.
Good point. Let me check that.
There was a problem hiding this comment.
@jaym I updated the PR
This will help us submit as many requests as we want without knowing about the workers. Signed-off-by: Salim Afiune Maya <[email protected]>
afiune
force-pushed
the
afiune/workerpool+github
branch
from
599b1bc to
93ca08d
Compare
internal/workerpool/collector.go
Outdated
| } | ||
|
|
||
| func (c *collector[R]) RequestsRead() int64 { | ||
| return c.requestsRead |
There was a problem hiding this comment.
i think there might be a data race here on requestsRead
There was a problem hiding this comment.
Fixed! Thanks for pointing it out, Jay.
internal/workerpool/pool.go
Outdated
|
|
||
| // PendingRequests returns the number of pending requests. | ||
| func (p *Pool[R]) PendingRequests() int64 { | ||
| return p.requestsSent - p.collector.RequestsRead() |
There was a problem hiding this comment.
i'm not really familiar with the atomics api in go, but i would expect some sort of synchronized read on requestsSent
There was a problem hiding this comment.
atomic helps to do thread safe operations, yes, I missed the read calls, they are now there! 🙌🏽
Signed-off-by: Salim Afiune Maya <[email protected]>
|
Used the data race detector from go to validate. |
Introducing an internal go package called
workerpoolthat we can use to send parallelrequests when needed.
⚡ For this change, I am making the fetching of repositories for an organization faster.
Tested this code with an organization that has around 3k repositories
Before (~2 Minutes)
After (~5 seconds)