Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improvement] Default size of /dev/shm is too small for PostgreSQL #6014

Closed
nayutah opened this issue Dec 7, 2023 · 0 comments · Fixed by #6016 or #6022
Closed

[Improvement] Default size of /dev/shm is too small for PostgreSQL #6014

nayutah opened this issue Dec 7, 2023 · 0 comments · Fixed by #6016 or #6022
Assignees
Labels
kind/enhancement New feature or request
Milestone

Comments

@nayutah
Copy link
Collaborator

nayutah commented Dec 7, 2023

Is your improvement request related to a problem? Please describe.
Default size of /dev/shm in docker & containerd is 64MB, too small for databases relying on shared memory like PostgreSQL
Also referred to bug #6013

The problem has been solved in:
https://github.com/apecloud/kubeblocks-addons/blob/main/addons/apecloud-postgresql/templates/clusterdefinition.yaml

One can test it with kubeblocks 0.7.* series:
slc@slcmac aws % kubectl exec -it poplar07-postgresql-0 -- /bin/sh
Defaulted container "postgresql" out of: postgresql, pgbouncer, metrics, kb-checkrole, config-manager, pg-init-container (init)
# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 457G 128G 307G 30% /
tmpfs 64M 0 64M 0% /dev
/dev/sda2 457G 128G 307G 30% /etc/hosts
tmpfs 126G 8.0K 126G 1% /kb-podinfo
tmpfs 126G 40K 126G 1% /dev/shm
tmpfs 126G 12K 126G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 63G 0 63G 0% /proc/acpi
tmpfs 63G 0 63G 0% /proc/scsi
tmpfs 63G 0 63G 0% /sys/firmware

root@node5:~# crictl exec -it 2b9e18c1cd576 /bin/sh
# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 457G 47G 387G 11% /
tmpfs 64M 0 64M 0% /dev
/dev/sda2 457G 47G 387G 11% /etc/hosts
tmpfs 126G 8.0K 126G 1% /kb-podinfo
tmpfs 126G 40K 126G 1% /dev/shm
tmpfs 126G 12K 126G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 63G 0 63G 0% /proc/acpi
tmpfs 63G 0 63G 0% /proc/scsi
tmpfs 63G 0 63G 0% /sys/firmware

This solution is adopted by several PostgreSQL providers, but this solution also has a safety risk:
When you dont set the 'sizeLimit', the default size of /dev/shm is set up with 'node physical memory capacity'.
It is a dangerous behavior for containers & k8s, a malicious container can congest all node memory with:
dd if=/dev/zero of=/dev/shm/output bs=4096 count=1M (or other counts)
So other containers on the same node will be affected.

So it is better to :
1, explicitly set 'sizeLimit' in the pod spec
2, if 'sizeLimit' is unset, the Operator should give it a default value with pod memory 'request' @leon-inf

@nayutah nayutah added the kind/enhancement New feature or request label Dec 7, 2023
@leon-inf leon-inf linked a pull request Dec 7, 2023 that will close this issue
@github-actions github-actions bot added this to the 5986 milestone Dec 8, 2023
@leon-inf leon-inf linked a pull request Dec 8, 2023 that will close this issue
@ahjing99 ahjing99 modified the milestones: 5986, Release 0.8.0 Dec 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment