Configure rbd-mirror debug logs dynamically #1346
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nirs
force-pushed
the
drenv-rbd-mirror-debug
branch
5 times, most recently
from
a9a69bc to
7ca1d82
Compare
This was referenced Apr 17, 2024
Closed
nirs
force-pushed
the
drenv-rbd-mirror-debug
branch
2 times, most recently
from
137c1d8 to
5ebca81
Compare
nirs
commented
Apr 18, 2024
nirs
force-pushed
the
drenv-rbd-mirror-debug
branch
from
5ebca81 to
0d23498
Compare
This makes testing changes much easier, for example, to test that the
kustomization creates a correct yaml, you can use:
kustomize build addon/rook-cluster
And since we support caching kustomization results, it is easy to cache
now all rook resources.
The downside of this change is increasing rook version duplication.
Before it was duplicated in 3 times in 3 files, now it is duplicated 5
times in 3 files. We need to improve this later.
Signed-off-by: Nir Soffer <[email protected]>
Enabling debug logs creates huge logs that can can fill up the node disk
if you leave a test environment running for long time. The log collector
rotates log files to avoid this issue.
With this change all ceph daemons log to file:
$ minikube ssh -p dr1 'ls -lh /data/rook/rook-ceph/log'
total 3.9M
-rw-r--r-- 1 167 167 453K Apr 18 14:08 20464565-c3dd-430f-830f-297dd03864bd-client.rbd-mirror-peer.log
-rw-r--r-- 1 167 167 91K Apr 18 14:08 ceph-client.ceph-exporter.log
-rw-r--r-- 1 167 167 1.8M Apr 18 14:08 ceph-client.rbd-mirror.a.log
-rw-r--r-- 1 167 167 326K Apr 18 14:08 ceph-mgr.a.log
-rw-r--r-- 1 167 167 234K Apr 18 14:08 ceph-mon.a.log
-rw-r--r-- 1 167 167 1.1M Apr 18 14:07 ceph-osd.0.log
-rw-r--r-- 1 167 167 17K Apr 18 13:57 ceph-volume.log
This is probably not needed, but it may help debugging issue in ceph.
Sine all logs expect rbd-mirror logs are not using debug level it
should not be a problem, and since logs are rotated the amount of data
should be limited.
Unfinished:
- Check how many logs files are kept after rotation
- Update gather tool to gather also rotated logs files
Signed-off-by: Nir Soffer <[email protected]>
This reverts commit c5e221b. Signed-off-by: Nir Soffer <[email protected]>
This reverts commit 832a5e1. Signed-off-by: Nir Soffer <[email protected]>
With our current configuration, the rbd-mirror-peer log is not created.
Iilya thinks[1] that this may be caused by the way we configure using
rook-config-override. Let's try to configure in the same way as
ocs-operator, using `ceph config set` commands[2].
With this configuration, the rbd-mirror-peer log is created:
$ minikube ssh -p dr1 'ls -1 /data/rook/rook-ceph/log/*rbd-mirror*.log'
/data/rook/rook-ceph/log/20464565-c3dd-430f-830f-297dd03864bd-client.rbd-mirror-peer.log
/data/rook/rook-ceph/log/ceph-client.rbd-mirror.a.log
[1] https://tracker.ceph.com/issues/65487?issue_count=296&issue_position=1&next_issue_id=65481#note-12
[2] red-hat-storage/ocs-operator@e39bb41
Signed-off-by: Nir Soffer <[email protected]>
nirs
force-pushed
the
drenv-rbd-mirror-debug
branch
from
0d23498 to
7b93c77
Compare
For testing the fix to bug[1], Ilia asked to run without debug logs,
since they make the issue much harder to reproduce. Add an environment
variable RBD_MIRROR_DEBUG=0 to disable debug logs. We enable debug logs
by default so issues in CI or developers environment will have useful
logs by default.
The new environment variable may also be useful for developers keeping
an environment for long time and want to avoid collecting hundreds of
megabytes of debug logs.
Example usage:
# Starting environment without debug logs
$ RBD_MIRROR_DEBUG=0 drenv start envs/rook.yaml
...
# Disabling debug logs on running environment
$ RBD_MIRROR_DEBUG=0 addons/rbd-mirror/start dr1 dr2
Disable rbd-mirror debug logs in cluster 'dr1'
Disable rbd-mirror debug logs in cluster 'dr2'
Getting mirroring info site name for cluster 'dr1'
...
# Enabling debug logs on running environment
$ addons/rbd-mirror/start dr1 dr2
$ addons/rbd-mirror/start dr1 dr2
Enable rbd-mirror debug logs in cluster 'dr1'
Enable rbd-mirror debug logs in cluster 'dr2'
Getting mirroring info site name for cluster 'dr1'
...
[1] https://tracker.ceph.com/issues/65487#note-20
Signed-off-by: Nir Soffer <[email protected]>
nirs
force-pushed
the
drenv-rbd-mirror-debug
branch
from
7b93c77 to
4705692
Compare
raghavendra-talur
approved these changes
Apr 30, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Configure rbd-mirror debug logs dynamically
With our current configuration, the rbd-mirror-peer log is not created.
Iilya thinks[1] that this may be caused by the way we configure using
rook-config-override. Let's try to configure in the same way as
ocs-operator, using
ceph config setcommands[2].[1] https://tracker.ceph.com/issues/65487?issue_count=296&issue_position=1&next_issue_id=65481#note-12
[2] red-hat-storage/ocs-operator@e39bb41
Status