-
Notifications
You must be signed in to change notification settings - Fork 57
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: make kurtosis service logs faster (#2525)
## Description
Users were experiencing `kurtosis service logs` taking a long time.
After running tests, I discovered that a majority of execution time
during log processing was spent in the following lines:
```
logLines := []logline.LogLine{*logLine}
userServicesLogLinesMap := map[service.ServiceUUID][]logline.LogLine{
serviceUuid: logLines,
}
logsByKurtosisUserServiceUuidChan <- userServicesLogLinesMap
```
Prior to this change, we were sending logs one at a time on an
unbuffered channel - unbuffered channels block until the receiving
goroutine reads the value. This was causing a lot of time being wasted
waiting to send log lines across the channel.
This change implements a `LogLineSender` that:
1. uses a buffered go channel (won't block on sending line unless buffer
is full)
2. batches log lines (reduces read overhead, receiving goroutine
performs fewer reads/sends)
With this change, the time to read 20 minutes of `cl-lighthouse-geth`
logs with log level set to debug went from `1min53sec` to `30.055`
seconds. The time to read 2 hours 10 minutes worth of `cl-lighthouse`
debug logs (around 3.4 gb of logs) went from `15min1sec` to `3min31`
sec. (As a benchmark, `cat logs.json` on `3.4 gb` of logs takes around
`2min` - on my machine - so much closer) This can likely be improved
further by tuning the buffer size and batch amount.
## Is this change user facing?
YES
## References:
https://discord.com/channels/783719264308953108/1267837033032974467/1267842228072611881- Loading branch information
Showing
9 changed files
with
109 additions
and
52 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -145,7 +145,7 @@ version: 2.1 | |
|
|
||
| orbs: | ||
| npm-publisher: uraway/[email protected] | ||
| kurtosis-docs-checker: kurtosis-tech/[email protected].7 | ||
| kurtosis-docs-checker: kurtosis-tech/[email protected].9 | ||
| slack: circleci/[email protected] | ||
|
|
||
| executors: | ||
|
|
@@ -1475,17 +1475,6 @@ workflows: | |
| name: "Check if CLI builds for all os and arch pairs" | ||
| <<: *filters_ignore_main | ||
|
|
||
| - test_enclave_manager_web_ui: | ||
| name: "Test Basic Web UI Functionality in Docker" | ||
| context: | ||
| - docker-user | ||
| requires: | ||
| - build_cli | ||
| - build_api_container_server | ||
| - build_engine_server | ||
| - build_files_artifacts_expander | ||
| <<: *filters_ignore_main | ||
|
|
||
| - test_basic_cli_functionality: | ||
| name: "Test Basic CLI Functionality in Docker" | ||
| cli-cluster-backend: "docker" | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
65 changes: 65 additions & 0 deletions
65
engine/server/engine/centralized_logs/logline/logline_sender.go
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| package logline | ||
|
|
||
| import ( | ||
| "github.com/kurtosis-tech/kurtosis/container-engine-lib/lib/backend_interface/objects/service" | ||
| "sync" | ||
| ) | ||
|
|
||
| const ( | ||
| batchLogsAmount = 500 | ||
| logsChanBufferSize = 300 | ||
| ) | ||
|
|
||
| type LogLineSender struct { | ||
| logsChan chan map[service.ServiceUUID][]LogLine | ||
|
|
||
| logLineBuffer map[service.ServiceUUID][]LogLine | ||
|
|
||
| mu sync.Mutex | ||
| } | ||
|
|
||
| func NewLogLineSender() *LogLineSender { | ||
| return &LogLineSender{ | ||
| logsChan: make(chan map[service.ServiceUUID][]LogLine, logsChanBufferSize), | ||
| logLineBuffer: map[service.ServiceUUID][]LogLine{}, | ||
| mu: sync.Mutex{}, | ||
| } | ||
| } | ||
|
|
||
| func (sender *LogLineSender) Send(serviceUuid service.ServiceUUID, logLine LogLine) { | ||
| sender.mu.Lock() | ||
| defer sender.mu.Unlock() | ||
|
|
||
| sender.logLineBuffer[serviceUuid] = append(sender.logLineBuffer[serviceUuid], logLine) | ||
|
|
||
| if len(sender.logLineBuffer[serviceUuid])%batchLogsAmount == 0 { | ||
| userServicesLogLinesMap := map[service.ServiceUUID][]LogLine{ | ||
| serviceUuid: sender.logLineBuffer[serviceUuid], | ||
| } | ||
| sender.logsChan <- userServicesLogLinesMap | ||
|
|
||
| // clear buffer after flushing it through the channel | ||
| sender.logLineBuffer[serviceUuid] = []LogLine{} | ||
| } | ||
| } | ||
|
|
||
| func (sender *LogLineSender) GetLogsChannel() chan map[service.ServiceUUID][]LogLine { | ||
| return sender.logsChan | ||
| } | ||
|
|
||
| // sends all logs remaining in the buffers through the channel | ||
| // this should be called at the end of processing to send the remainder of logs | ||
| func (sender *LogLineSender) Flush() { | ||
| sender.mu.Lock() | ||
| defer sender.mu.Unlock() | ||
|
|
||
| for uuid, logLines := range sender.logLineBuffer { | ||
| serviceUuid := uuid | ||
| userServiceLogLinesMap := map[service.ServiceUUID][]LogLine{ | ||
| serviceUuid: logLines, | ||
| } | ||
| sender.logsChan <- userServiceLogLinesMap | ||
|
|
||
| sender.logLineBuffer[serviceUuid] = []LogLine{} | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters