Skip to content

Commit

Permalink
Move dbscripts SQL to koios-artifacts repo, and prep v1.1.0 (#1708)
Browse files Browse the repository at this point in the history
## Description
<!--- Describe your changes -->
- [x] Move /scripts/gnode-helper-scripts/dbscripts folder to
koios-artifacts repo (v1.1.0 tag), as it only contains SQL files.
Accordingly update setup-grest.sh references.
- [x] Some left-over hard-coded entries (other than documentation ones)
for `cardano-community` replaced by `${G_ACCOUNT}`
- [x] Update dbsync.sh to also pre-fill conway genesis hash
- [x] Install cron job for asset_txo_cache_update
- [x] Bump setup-grest.sh tag reference to v1.1.0

## How has this been tested?
<!--- Describe how you tested changes -->
Tested on v1 instance
  • Loading branch information
rdlrt committed Nov 27, 2023
1 parent 30704c1 commit 916af3b
Show file tree
Hide file tree
Showing 9 changed files with 17 additions and 241 deletions.
22 changes: 2 additions & 20 deletions docs/Scripts/gliveview.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,27 +3,9 @@

**Koios gLiveView** is a local monitoring tool to use in addition to remote monitoring tools like Prometheus/Grafana, Zabbix or IOG's RTView. This is especially useful when moving to a systemd deployment - if you haven't done so already - as it offers an intuitive UI to monitor the node status.

The tool is independent from other files and can run as a standalone utility that can be stopped/started without affecting the status of `cardano-node`.

##### Download

If you've used [guild-deploy.sh](../basics.md#pre-requisites), you can skip this part, as this is already set up for you. The tool relies on the common `env` configuration file.
To get current epoch blocks, the [logMonitor.sh](../Scripts/logmonitor.md) script is needed (and can be combined with [CNCLI](../Scripts/cncli.md)). This is optional and **Koios gLiveView** will function without it.

!!! info "Note"
For those who follow the [folder structure](../basics.md#folder-structure) in this repo and do not wish to run `guild-deploy.sh`, you can run the below in `$CNODE_HOME/scripts` folder

To download the script:

```bash
curl -s -o gLiveView.sh https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/gLiveView.sh
curl -s -o env https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/env
chmod 755 gLiveView.sh
```

##### Configuration & Startup

For most setups, it's enough to set `CNODE_PORT` in the `env` file. The rest of the variables should automatically be detected. If required, modify User Variables in `env` and `gLiveView.sh` to suit your environment (if folder structure you use is different). This should lead you to a stage where you can now start running `./gLiveView.sh` in the folder you downloaded the script (the default location would be `$CNODE_HOME/scripts`). Note that the script is smart enough to automatically detect when you're running as a Core or Relay and will show fields accordingly.
For most setups, it's enough to set `CNODE_PORT` in the `env` file. The rest of the variables should automatically be detected. If required, modify User Variables in `env` and `gLiveView.sh` to suit your environment (if the environment is customised). This should lead you to a stage where you can now start running `./gLiveView.sh` in the folder you downloaded the script (the default location would be `$CNODE_HOME/scripts`). Note that the script is smart enough to automatically detect when you're running as a Core or Relay and will show fields accordingly.

The tool can be run in legacy mode with only standard ASCII characters for terminals with trouble displaying the box-drawing characters. Run `./gLiveView.sh -h` to show available command-line parameters or permanently set it directly in script.

Expand Down Expand Up @@ -65,7 +47,7 @@ Displays live metrics from cardano-node gathered through the nodes EKG/Prometheu
- **Mem (RSS)** - RSS is the Resident Set Size and shows how much memory is allocated to cardano-node and that is in RAM. It does not include memory that is swapped out. It does include memory from shared libraries as long as the pages from those libraries are actually in memory. It does include all stack and heap memory.
- **Mem (Live) / (Heap)** - GC (Garbage Collector) values that show how much memory is used for live/heap data. A large difference between them (or the heap approaching the physical memory limit) means the node is struggling with the garbage collector and/or may begin swapping.
- **GC Minor / Major** - Collecting garbage from "Young space" is called a Minor GC. Major (Full) GC is done more rarily and is a more expensive operation. Explaining garbage collection is a topic outside the scope of this documentation and google is your friend for this.
- **Block propagation** - Last delay measures the duration between when the last block was scheduled to be produced and when the node learned about it. Late blocks are blocks whose delay is larger than 5s. If the node is not synching, the number of late blocks needs to stay low. Within 1/3/5s estimates the chance of observing a delay of 1/3/5s (based on the delays observed for previous blocks). A healthy node needs to stay above 95% of blocks within 3s. Finally, served blocks counts how many blocks were fetched by "in" peers. If this does not increase for a long time, it means the "in" peers are learning about new blocks from somewhere else (and therefore this node is not contributing towards accelerating the propagation). Overall, these metrics are helpful in tweaking the topology and/or performance of the network links.
- **Block propagation** - Last Block measures the duration between when the last block was scheduled to be produced and when the node learned about it. Late blocks are blocks whose delay is larger than 5s. If the node is not synching, the number of late blocks needs to stay low. Within 1/3/5s estimates the chance of observing a delay of 1/3/5s (based on the delays observed for previous blocks). A healthy node needs to stay above 95% of blocks within 3s. Finally, served blocks counts how many blocks were fetched by "in" peers. If this does not increase for a long time, it means the "in" peers are learning about new blocks from somewhere else (and therefore this node is not contributing towards accelerating the propagation). Overall, these metrics are helpful in tweaking the topology and/or performance of the network links.

###### Core section

Expand Down
6 changes: 4 additions & 2 deletions scripts/cnode-helper-scripts/dbsync.sh
Original file line number Diff line number Diff line change
Expand Up @@ -70,10 +70,12 @@ check_config_sanity() {
SHGENHASHCFG=$(jq '.ShelleyGenesisHash' <"${CONFIG}" 2>/dev/null)
ALGENHASH=$("${CCLI}" genesis hash --genesis "${ALONZO_GENESIS_JSON}" 2>/dev/null)
ALGENHASHCFG=$(jq '.AlonzoGenesisHash' <"${CONFIG}" 2>/dev/null)
CWGENHASH=$("${CCLI}" genesis hash --genesis "${CONWAY_GENESIS_JSON}" 2>/dev/null)
CWGENHASHCFG=$(jq '.ConwayGenesisHash' <"${CONFIG}" 2>/dev/null)
# If hash are missing/do not match, add that to the end of config. We could have sorted it based on logic, but that would mess up sdiff comparison outputs
if [[ "${BYGENHASH}" != "${BYGENHASHCFG}" ]] || [[ "${SHGENHASH}" != "${SHGENHASHCFG}" ]] || [[ "${ALGENHASH}" != "${ALGENHASHCFG}" ]]; then
if [[ "${BYGENHASH}" != "${BYGENHASHCFG}" ]] || [[ "${SHGENHASH}" != "${SHGENHASHCFG}" ]] || [[ "${ALGENHASH}" != "${ALGENHASHCFG}" ]] || [[ "${CWGENHASH}" != "${CWGENHASHCFG}" ]]; then
cp "${CONFIG}" "${CONFIG}".tmp
jq --arg BYGENHASH ${BYGENHASH} --arg SHGENHASH ${SHGENHASH} --arg ALGENHASH ${ALGENHASH} '.ByronGenesisHash = $BYGENHASH | .ShelleyGenesisHash = $SHGENHASH | .AlonzoGenesisHash = $ALGENHASH' <"${CONFIG}" >"${CONFIG}".tmp
jq --arg BYGENHASH ${BYGENHASH} --arg SHGENHASH ${SHGENHASH} --arg ALGENHASH ${ALGENHASH} --arg CWGENHASH ${CWGENHASH} '.ByronGenesisHash = $BYGENHASH | .ShelleyGenesisHash = $SHGENHASH | .AlonzoGenesisHash = $ALGENHASH | .ConwayGenesisHash = $CWGENHASH' <"${CONFIG}" >"${CONFIG}".tmp
[[ -s "${CONFIG}".tmp ]] && mv -f "${CONFIG}".tmp "${CONFIG}"
fi
}
Expand Down
1 change: 0 additions & 1 deletion scripts/cnode-helper-scripts/env
Original file line number Diff line number Diff line change
Expand Up @@ -781,7 +781,6 @@ set_default_vars() {
[[ -z ${G_ACCOUNT} ]] && G_ACCOUNT="cardano-community"
URL_RAW="https://raw.githubusercontent.com/${G_ACCOUNT}/guild-operators/${BRANCH}"
DB_SCRIPTS_URL="${URL_RAW}/scripts/grest-helper-scripts/db-scripts"
export LC_ALL=C.UTF-8
# special mapping of coreutils gdate to date for MacOS
if [[ $(uname) == Darwin ]]; then
Expand Down
2 changes: 1 addition & 1 deletion scripts/cnode-helper-scripts/gLiveView.sh
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ if [[ ${UPDATE_CHECK} = Y && ${SKIP_UPDATE} != Y ]]; then

if command -v cncli >/dev/null && command -v systemctl >/dev/null && systemctl is-active --quiet ${CNODE_VNAME}-cncli-sync.service; then
vcur=$(cncli -V | sed 's/cncli /v/g')
vrem=$(curl -s https://api.github.com/repos/cardano-community/cncli/releases/latest | jq -r .tag_name)
vrem=$(curl -s https://api.github.com/repos/${G_ACCOUNT}/cncli/releases/latest | jq -r .tag_name)
[[ ${vcur} != ${vrem} ]] && printf "${FG_MAGENTA}CNCLI current version (${vcur}) different from repo (${vrem}), consider upgrading!.${NC}" && waitToProceed
fi

Expand Down
4 changes: 2 additions & 2 deletions scripts/cnode-helper-scripts/guild-deploy.sh
Original file line number Diff line number Diff line change
Expand Up @@ -356,11 +356,11 @@ download_cncli() {
[[ -z ${ARCH##*aarch64*} ]] && err_exit " The cncli pre-compiled binary is not available for ARM, you might need to build them!"
echo -e "\nInstalling CNCLI.."
if command -v cncli >/dev/null; then cncli_version="v$(cncli -V 2>/dev/null | cut -d' ' -f2)"; else cncli_version="v0.0.0"; fi
cncli_git_version="$(curl -s https://api.github.com/repos/cardano-community/cncli/releases/latest | jq -r '.tag_name')"
cncli_git_version="$(curl -s https://api.github.com/repos/${G_ACCOUNT}/cncli/releases/latest | jq -r '.tag_name')"
echo -e "\n Downloading CNCLI..."
rm -rf /tmp/cncli-bin && mkdir /tmp/cncli-bin
pushd /tmp/cncli-bin >/dev/null || err_exit
cncli_asset_url="$(curl -s https://api.github.com/repos/cardano-community/cncli/releases/latest | jq -r '.assets[].browser_download_url' | grep 'linux-musl.tar.gz')"
cncli_asset_url="$(curl -s https://api.github.com/repos/${G_ACCOUNT}/cncli/releases/latest | jq -r '.assets[].browser_download_url' | grep 'linux-musl.tar.gz')"
if curl -sL -f -m ${CURL_TIMEOUT} -o cncli.tar.gz ${cncli_asset_url}; then
tar zxf cncli.tar.gz &>/dev/null
rm -f cncli.tar.gz
Expand Down
173 changes: 0 additions & 173 deletions scripts/grest-helper-scripts/db-scripts/basics.sql

This file was deleted.

18 changes: 0 additions & 18 deletions scripts/grest-helper-scripts/db-scripts/genesis_table.sql

This file was deleted.

21 changes: 0 additions & 21 deletions scripts/grest-helper-scripts/db-scripts/reset_grest.sql

This file was deleted.

11 changes: 8 additions & 3 deletions scripts/grest-helper-scripts/setup-grest.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
# Do NOT modify code below #
######################################

SGVERSION=v1.1.0rc
SGVERSION=v1.1.0

######## Functions ########
usage() {
Expand Down Expand Up @@ -91,7 +91,7 @@ SGVERSION=v1.1.0rc
get_cron_job_executable() {
local job=$1
local job_path="${CRON_SCRIPTS_DIR}/${job}.sh"
local job_url="https://raw.githubusercontent.com/cardano-community/koios-artifacts/${SGVERSION}/files/grest/cron/jobs/${job}.sh"
local job_url="https://raw.githubusercontent.com/${G_ACCOUNT}/koios-artifacts/${SGVERSION}/files/grest/cron/jobs/${job}.sh"
if curl -s -f -m "${CURL_TIMEOUT}" -o "${job_path}" "${job_url}"; then
printf "\n Downloaded \e[32m${job_path}\e[0m"
chmod +x "${job_path}"
Expand Down Expand Up @@ -172,7 +172,11 @@ SGVERSION=v1.1.0rc

get_cron_job_executable "asset-info-cache-update"
set_cron_variables "asset-info-cache-update"
install_cron_job "asset-info-cache-update" "* * * * *"
install_cron_job "asset-info-cache-update" "*/2 * * * *"

get_cron_job_executable "asset-txo-cache-update"
set_cron_variables "asset-txo-cache-update"
install_cron_job "asset-txo-cache-update" "*/5 * * * *"

get_cron_job_executable "epoch-summary-corrections-update"
set_cron_variables "epoch-summary-corrections-update"
Expand Down Expand Up @@ -205,6 +209,7 @@ SGVERSION=v1.1.0rc
[[ -z "${CRON_DIR}" ]] && CRON_DIR="/etc/cron.d"
[[ -z "${PGDATABASE}" ]] && PGDATABASE="cexplorer"
[[ -z "${HAPROXY_CFG}" ]] && HAPROXY_CFG="${CNODE_HOME}/files/haproxy.cfg"
[[ -z "${DB_SCRIPTS_URL}" ]] && DB_SCRIPTS_URL="https://raw.githubusercontent.com/${G_ACCOUNT}/koios-artifacts/${SGVERSION}/files/grest/rpc/db-scripts"
DOCS_URL="https://cardano-community.github.io/guild-operators"
[[ -z "${PGPASSFILE}" ]] && export PGPASSFILE="${CNODE_HOME}"/priv/.pgpass
case ${NWMAGIC} in
Expand Down

0 comments on commit 916af3b

Please sign in to comment.