Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: add DB keepalives #358

Merged
merged 1 commit into from
Jan 14, 2025
Merged

fix: add DB keepalives #358

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions .env
View file
Open in desktop

This file was deleted.

31 changes: 31 additions & 0 deletions .env-example
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Used for local development, when running in docker or on command line

DB_USER=
DB_PASS=
DB_HOST=
DB_NAME=
DB_PORT=
DB_SSL_MODE=
DB_KEEPALIVE_IDLE=60
DB_KEEPALIVE_INTERVAL=15
DB_KEEPALIVE_COUNT=5

AZURE_STORAGE_CONNECTION_STRING=
AZURE_STORAGE_CONTAINER_SOURCE=source
AZURE_STORAGE_CONTAINER_CLEAN=clean
ACTIVITIES_LAKE_CONTAINER_NAME=lake

SCHEMA_VALIDATION_API_URL=
SCHEMA_VALIDATION_KEY_NAME=x-functions-key
SCHEMA_VALIDATION_KEY_VALUE=

VALIDATOR_API_URL=
VALIDATOR_API_KEY_NAME=x-functions-key
VALIDATOR_API_KEY_VALUE=

SOLR_API_URL=
SOLR_USER=
SOLR_PASSWORD=
LOG_LEVEL=
SOLR_PARALLEL_PROCESSES=
SOLR_500_SLEEP=
6 changes: 6 additions & 0 deletions .github/workflows/develop.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ env:
DB_HOST: ${{ secrets.DB_HOST }}
DB_NAME: ${{ secrets.DB_NAME }}
DB_PORT: ${{ secrets.DB_PORT }}
DB_KEEPALIVE_IDLE: ${{ vars.DEV_DB_KEEPALIVE_IDLE }}
DB_KEEPALIVE_INTERVAL: ${{ vars.DEV_DB_KEEPALIVE_INTERVAL }}
DB_KEEPALIVE_COUNT: ${{ vars.DEV_DB_KEEPALIVE_COUNT }}
LOG_WORKSPACE_ID: ${{ secrets.DEV_LOG_WORKSPACE_ID }}
LOG_WORKSPACE_KEY: ${{ secrets.DEV_LOG_WORKSPACE_KEY }}
COMMSHUB_URL: ${{ secrets.DEV_COMMSHUB_URL }}
Expand Down Expand Up @@ -115,6 +118,9 @@ jobs:
sed -i ''s^#DB_USER#^$DB_USER^g'' ./deployment/deployment.yml
sed -i ''s^#DB_PASS#^$DB_PASS^g'' ./deployment/deployment.yml
sed -i ''s^#DB_NAME#^$DB_NAME^g'' ./deployment/deployment.yml
sed -i ''s^#DB_KEEPALIVE_IDLE#^$DB_KEEPALIVE_IDLE^g'' ./deployment/deployment.yml
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't these strings needed to be added to deployment/deployment.yml too?

And can we add them as values not secureValues, I think that means we can actually see what they are in Azure? (as we won't be able to in GitHub)

Copy link
Collaborator Author

@simon-20 simon-20 Dec 5, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, thanks, good spot.

It's possible to see any secret value sent to an Azure container instance by connecting to the container and echoing the value to the terminal, but I agree it would make sense to have these as values too, so that extra step is not needed.

It would also be good to have them in Github as variables, not secrets. What do you think? I think last time I suggested this you weren't sure because the existing non-secret values had been entered as secrets, but it would be good to start fixing this, especially as we'll have more opportunity to do so with all the upcoming work on the pipeline.

sed -i ''s^#DB_KEEPALIVE_INTERVAL#^$DB_KEEPALIVE_INTERVAL^g'' ./deployment/deployment.yml
sed -i ''s^#DB_KEEPALIVE_COUNT#^$DB_KEEPALIVE_COUNT^g'' ./deployment/deployment.yml

sed -i ''s^#SOLR_API_URL#^$SOLR_API_URL^g'' ./deployment/deployment.yml
sed -i ''s^#SOLR_USER#^$SOLR_USER^g'' ./deployment/deployment.yml
Expand Down
6 changes: 6 additions & 0 deletions .github/workflows/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@ env:
DB_HOST: ${{ secrets.PROD_DB_HOST }}
DB_NAME: ${{ secrets.PROD_DB_NAME }}
DB_PORT: ${{ secrets.PROD_DB_PORT }}
DB_KEEPALIVE_IDLE: ${{ vars.PROD_DB_KEEPALIVE_IDLE }}
DB_KEEPALIVE_INTERVAL: ${{ vars.PROD_DB_KEEPALIVE_INTERVAL }}
DB_KEEPALIVE_COUNT: ${{ vars.PROD_DB_KEEPALIVE_COUNT }}
LOG_WORKSPACE_ID: ${{ secrets.PROD_LOG_WORKSPACE_ID }}
LOG_WORKSPACE_KEY: ${{ secrets.PROD_LOG_WORKSPACE_KEY }}
COMMSHUB_URL: ${{ secrets.PROD_COMMSHUB_URL }}
Expand Down Expand Up @@ -111,6 +114,9 @@ jobs:
sed -i ''s^#DB_USER#^$DB_USER^g'' ./deployment/deployment.yml
sed -i ''s^#DB_PASS#^$DB_PASS^g'' ./deployment/deployment.yml
sed -i ''s^#DB_NAME#^$DB_NAME^g'' ./deployment/deployment.yml
sed -i ''s^#DB_KEEPALIVE_IDLE#^$DB_KEEPALIVE_IDLE^g'' ./deployment/deployment.yml
sed -i ''s^#DB_KEEPALIVE_INTERVAL#^$DB_KEEPALIVE_INTERVAL^g'' ./deployment/deployment.yml
sed -i ''s^#DB_KEEPALIVE_COUNT#^$DB_KEEPALIVE_COUNT^g'' ./deployment/deployment.yml

sed -i ''s^#SOLR_API_URL#^$SOLR_API_URL^g'' ./deployment/deployment.yml
sed -i ''s^#SOLR_USER#^$SOLR_USER^g'' ./deployment/deployment.yml
Expand Down
36 changes: 36 additions & 0 deletions deployment/deployment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,12 @@ properties: # Properties of container group
secureValue: '#DB_PASS#'
- name: DB_NAME
secureValue: '#DB_NAME#'
- name: DB_KEEPALIVE_IDLE
value: '#DB_KEEPALIVE_IDLE#'
- name: DB_KEEPALIVE_INTERVAL
value: '#DB_KEEPALIVE_INTERVAL#'
- name: DB_KEEPALIVE_COUNT
value: '#DB_KEEPALIVE_COUNT#'
- name: SOLR_API_URL
secureValue: '#SOLR_API_URL#'
- name: SOLR_USER
Expand Down Expand Up @@ -84,6 +90,12 @@ properties: # Properties of container group
secureValue: '#DB_PASS#'
- name: DB_NAME
secureValue: '#DB_NAME#'
- name: DB_KEEPALIVE_IDLE
value: '#DB_KEEPALIVE_IDLE#'
- name: DB_KEEPALIVE_INTERVAL
value: '#DB_KEEPALIVE_INTERVAL#'
- name: DB_KEEPALIVE_COUNT
value: '#DB_KEEPALIVE_COUNT#'
- name: SOLR_API_URL
secureValue: '#SOLR_API_URL#'
- name: SOLR_USER
Expand Down Expand Up @@ -137,6 +149,12 @@ properties: # Properties of container group
secureValue: '#DB_PASS#'
- name: DB_NAME
secureValue: '#DB_NAME#'
- name: DB_KEEPALIVE_IDLE
value: '#DB_KEEPALIVE_IDLE#'
- name: DB_KEEPALIVE_INTERVAL
value: '#DB_KEEPALIVE_INTERVAL#'
- name: DB_KEEPALIVE_COUNT
value: '#DB_KEEPALIVE_COUNT#'
- name: SOLR_API_URL
secureValue: '#SOLR_API_URL#'
- name: SOLR_USER
Expand Down Expand Up @@ -190,6 +208,12 @@ properties: # Properties of container group
secureValue: '#DB_PASS#'
- name: DB_NAME
secureValue: '#DB_NAME#'
- name: DB_KEEPALIVE_IDLE
value: '#DB_KEEPALIVE_IDLE#'
- name: DB_KEEPALIVE_INTERVAL
value: '#DB_KEEPALIVE_INTERVAL#'
- name: DB_KEEPALIVE_COUNT
value: '#DB_KEEPALIVE_COUNT#'
- name: SOLR_API_URL
secureValue: '#SOLR_API_URL#'
- name: SOLR_USER
Expand Down Expand Up @@ -239,6 +263,12 @@ properties: # Properties of container group
secureValue: '#DB_PASS#'
- name: DB_NAME
secureValue: '#DB_NAME#'
- name: DB_KEEPALIVE_IDLE
value: '#DB_KEEPALIVE_IDLE#'
- name: DB_KEEPALIVE_INTERVAL
value: '#DB_KEEPALIVE_INTERVAL#'
- name: DB_KEEPALIVE_COUNT
value: '#DB_KEEPALIVE_COUNT#'
- name: SOLR_API_URL
secureValue: '#SOLR_API_URL#'
- name: COMMSHUB_URL
Expand Down Expand Up @@ -280,6 +310,12 @@ properties: # Properties of container group
secureValue: '#DB_PASS#'
- name: DB_NAME
secureValue: '#DB_NAME#'
- name: DB_KEEPALIVE_IDLE
value: '#DB_KEEPALIVE_IDLE#'
- name: DB_KEEPALIVE_INTERVAL
value: '#DB_KEEPALIVE_INTERVAL#'
- name: DB_KEEPALIVE_COUNT
value: '#DB_KEEPALIVE_COUNT#'
- name: SOLR_API_URL
secureValue: '#SOLR_API_URL#'
- name: SOLR_USER
Expand Down
5 changes: 4 additions & 1 deletion src/constants/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,14 @@
DB_PORT=os.getenv("DB_PORT"),
DB_NAME=os.getenv("DB_NAME"),
DB_SSL_MODE=os.getenv("DB_SSL_MODE") or "require",
# Database retry/timeout constants
# Database retry/timeout/keepalive
DB_CONN_RETRY_LIMIT=8,
DB_CONN_SLEEP_START=5,
DB_CONN_SLEEP_MAX=60,
DB_CONN_TIMEOUT=5,
DB_KEEPALIVE_IDLE=int(os.getenv("DB_KEEPALIVE_IDLE", default=60)),
DB_KEEPALIVE_INTERVAL=int(os.getenv("DB_KEEPALIVE_INTERVAL", default=15)),
DB_KEEPALIVE_COUNT=int(os.getenv("DB_KEEPALIVE_COUNT", default=5)),
# Azure Storage Account Connection String
# This can be found in the Azure Portal > Storage Account > Access Keys
STORAGE_CONNECTION_STR=os.getenv("AZURE_STORAGE_CONNECTION_STRING"),
Expand Down
4 changes: 4 additions & 0 deletions src/library/db.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,10 @@ def getDirectConnection(retry_counter=0):
port=config["DB_PORT"],
sslmode=config["DB_SSL_MODE"],
connect_timeout=config["DB_CONN_TIMEOUT"],
keepalives=1,
keepalives_idle=config["DB_KEEPALIVE_IDLE"],
keepalives_interval=config["DB_KEEPALIVE_INTERVAL"],
keepalives_count=config["DB_KEEPALIVE_COUNT"],
)
retry_counter = 0
return connection
Expand Down
Loading