Sam/oriole17 (#1298)

* feat: build and flake check of pg 16.3 with exts/wrappers

* pg_partman test 15/16 compat

* merge sql interface test

* tests: build test and cache both versions

* chore: run checks individually

* pg 15 and 16 packer/ansible/ghactions (#1268)

* fix: reformat ec2 cleanup commands (#1267)

Co-authored-by: Sam Rose <[email protected]>

* feat: build and flake check of pg 16.3 with exts/wrappers

* pg_partman test 15/16 compat

* merge sql interface test

* tests: build test and cache both versions

* chore: run checks individually

* feat: realease 15 and 16 to staging

* chore: update versions

* chore: make yq available

* chore: run yq from nix

* chore: more setup for staging AMI

* fix: yq usage

* chore: shell vars

* fix: When --init none is used, only users who can elevate to sudo privileges can run Nix

* fix: no -i

* fix: quote correction

* fix: newline extra quotes

* fix: no need for pg major version on packer

* fix: postgresql_major

* fix: ql

* fix: no ansible args in stage to invocation

* fix: unique val

* fix: adjustments to build scripts

* chore: env var handling

* fix: bump to build

* chore: set up more required vars

* chore: bump var

* feat: pg 16 debug symbols

* feat: matrix pg versions build on testinfra

* feat: matrix on Test Database

* chore: running nix in the right context

* feat: just use existing Dockerfile + pg version

* chore: refer to var

* fix: read name without including quotes

* chore: try format function

* fix: strip quotes from version number

* chore: env var

* fix: pg client

* fix

* fix: try to use psql from our own corresponding pkg

* fix: try psql from ppa

* fix: dbmate per pg version

* build dbmate and then install client

* fix: account for architecture

* chore: limit changes detection migrations/schema.sql

* missing docker compose call

* ore: drop tests while investigating

* test: try on pg15 only

* chore: schema needs update

* chore: now run on all versions in matrix

* test: trying a version of schema per major pg version as there are type diffs

---------

Co-authored-by: Sam Rose <[email protected]>
Co-authored-by: Oliver Rice <[email protected]>

* chore: rm dead code

* chore: resolve versions from ansible/vars.yml instead of packer file
create a matrix build for 15 16 (or how ever many versions there are)

* feat: WIP orioledb 16

* feat: intro into CI building psql16 bundle oriole version

* test: working through build compat for oriole

* working orioledb build

* fix: update value

* chore: cut staging only release for oriole16

* narrow down to just oriole staging ami here

* fix: make sure name lines up with gh action convention

* fix: get correct naming convention in here too for oriole bundle

* chore: bump to cut release

* fix: make sure src and debug builds available for oriole too

* chore: bump to staging release

* fix: when oriole16 rm timescale from supautils conf

* fix: also remove from postgresql conf

* fix: find all instances and remove in this condition for tiemscaledb

* fix: maybe broke pgsodium with too many spaces

* fix: create a symlink for missing path

* fix: try a more universal way to deactivate pgsodium at this stage

* chore: when oriole16 rm timescale plv8 postgis pgvector pgrouting

* fix: more handling of oriole16 differences

* chore more oriole16 handling

* fix: refine sed regex

* fix: also cond rm actual files

* fix: correct dir

* chore: bump to release

* feat: parse and handle oropledb-16 in start-server

* feat: start-client for oriole16

* wip for orioledb push to build on linux

* fix: running server

* chore: consolidate nix code that handles building of postgres versions

* fix: update name for bundle

* chore: small cleanup

* chore: bump version release

* chore: cleanup names

* chore: correct names

* fix: var name

* fix: one more var name

* chore: bump to release

* fix: reformatting metadata for clikchouse dep as git was not able to fetch

* feat: wip refactor to pg 17 for orioledb

* feat: working orioledb-17

* fix: increase role duration to avoid expiration

* fix: correct version

* fix: clickhouse needs git in build inputs

* fix: install and then remove git for clickhouse dep

* fix: correct rules for version

* fix: db_user_namespace was deprecated starting in pg 16.4 and higher

* apply fix for wrappers build, deactivate ext in unit test

* chore: deactivate more tests due to deactivated extensions

* fix: also remove from sql file

* fix: regex

* fix: also remove ref here

* fix: graphql_public schema too

* chore: correct dir

* chore: staging release

* feat: re-introduce wal2json rum and pgvector

* fix: also restore pgvector in start-server

* chore: bump staging release

* feat: oriole specific default settings

* feat: use icu for locales only if building orioledb

* fix: need var in stage 1 too

* fix: settings must be in db init

* fix: ansible args

* fix: PostgreSQL uses ICU for all locale-related operations, so we don't need to specify the collation and character type settings separately.

* chore: add a migration for orioledb activation by default

* feat: turn on oriole if oriole ext exists

* test: conditional on test for orioledb

* fix: salt changed repo and method for adding apt package

* fix: udpate salt minion setup

* fix: silent skip if oriole not in the install

* test: check if oriole is available and if so then enable

* chore: build and cache src and debug pkgs as well

* fix: tmp supress build of src and debug

* chore: tmp disable src and debug

* fix: activate oriole first in the oriole context

* fix: pgroonga build on macos

* fix: restore flake check for now

* fixing flake check for darwin + passing check until pg_net

* fix: start-server macos

* fix: src yq from apt

* fix: all instances yq

* fix: jq invoke

* fix: nix profile install

* fix: nix run

* fix: sudo nix run

* chore: cut staging release

* chore: rm debugging

* fix: still need CURRENT_SYSTEM

* test: re-introduce flake checks for 15 and 16

* chore: bump versions of patched pg for oriole to 17_4 and ext to beta6

* chore: bump var to release ami

* chore: break down into functions

* fix: skip x86 darwin for oriole nix builds

* chore: cleanup repo for final review, rebase and merge

* fix: try to properly init db in non oriole context

* chore: restore installation of debug and src for all versions`

* chore: newline

* chore: cleaning up and restoring missing code

* chore: cleanup

* chore: fix gh action conditonal for oriole

* fix: filter orioledb-17 from test on read

---------

Co-authored-by: Oliver Rice <[email protected]>

.github/workflows/test.yml

@@ -1,12 +1,10 @@
 name: Test Database
-
 on:
   push:
     branches:
       - develop
   pull_request:
   workflow_dispatch:
-
 jobs:
   prepare:
     runs-on: ubuntu-latest
@@ -21,13 +19,11 @@ jobs:
           extra-conf: |
             substituters = https://cache.nixos.org https://nix-postgres-artifacts.s3.amazonaws.com
             trusted-public-keys = nix-postgres-artifacts:dGZlQOvKcNEjvT7QEAJbcV6b6uk7VF/hWMjhYleiaLI=% cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
-
       - name: Set PostgreSQL versions
         id: set-versions
         run: |
-          VERSIONS=$(nix run nixpkgs#yq --  '.postgres_major[]' ansible/vars.yml | nix run nixpkgs#jq -- -R -s -c 'split("\n")[:-1]')
+          VERSIONS=$(nix run nixpkgs#yq -- '.postgres_major[] | select(. != "orioledb-17")' ansible/vars.yml | nix run nixpkgs#jq -- -R -s -c 'split("\n")[:-1]')
           echo "postgres_versions=$VERSIONS" >> $GITHUB_OUTPUT
-
   build:
     needs: prepare
     strategy:
@@ -45,49 +41,38 @@ jobs:
       POSTGRES_PASSWORD: password
     steps:
       - uses: actions/checkout@v4
-
       - uses: DeterminateSystems/nix-installer-action@main
         with:
           extra-conf: |
             substituters = https://cache.nixos.org https://nix-postgres-artifacts.s3.amazonaws.com
             trusted-public-keys = nix-postgres-artifacts:dGZlQOvKcNEjvT7QEAJbcV6b6uk7VF/hWMjhYleiaLI=% cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
-
-
       - name: Set PostgreSQL version environment variable
         run: echo "POSTGRES_MAJOR_VERSION=${{ matrix.postgres_version }}" >> $GITHUB_ENV
-
       - name: Strip quotes from pg major and set env var
         run: |
           stripped_version=$(echo ${{ matrix.postgres_version }} | sed 's/^"\(.*\)"$/\1/')
           echo "PGMAJOR=$stripped_version" >> $GITHUB_ENV
-
       - name: Generate common-nix.vars.pkr.hcl
         run: |
           PG_VERSION=$(nix run nixpkgs#yq -- '.postgres_release["postgres'${{ matrix.postgres_version }}'"]' ansible/vars.yml)
           PG_VERSION=$(echo $PG_VERSION | tr -d '"')  # Remove any surrounding quotes
           echo 'postgres-version = "'$PG_VERSION'"' > common-nix.vars.pkr.hcl
-          # Ensure there's a newline at the end of the file
           echo "" >> common-nix.vars.pkr.hcl
-
       - id: settings
-        # Remove spaces and quotes to get the raw version string
         run: sed -r 's/(\s|\")+//g' common-nix.vars.pkr.hcl >> $GITHUB_OUTPUT
-
       - name: Generate args
         id: args
         run: |
           ARGS=$(nix run nixpkgs#yq -- 'to_entries | map(select(.value|type == "!!str")) | map(.key + "=" + .value) | join("\n")' ansible/vars.yml)
           echo "result<<EOF" >> $GITHUB_OUTPUT
           echo "$ARGS" >> $GITHUB_OUTPUT
           echo "EOF" >> $GITHUB_OUTPUT
-
-      #TODO PR Convert to develop branch flakeurl
       - name: verify schema.sql is committed
         run: |
           GIT_SHA=${{github.sha}}
-          nix run github:supabase/postgres/${GIT_SHA}#dbmate-tool -- --version ${{ env.PGMAJOR }}
+          nix run github:supabase/postgres/develop#dbmate-tool -- --version ${{ env.PGMAJOR }}
           if ! git diff --exit-code --quiet migrations/schema-${{ env.PGMAJOR }}.sql; then
             echo "Detected changes in schema.sql:"
             git diff migrations/schema-${{ env.PGMAJOR }}.sql
             exit 1
-          fi
+          fi

.github/workflows/testinfra-nix.yml

@@ -18,7 +18,7 @@ jobs:
       - name: Set PostgreSQL versions
         id: set-versions
         run: |
-          VERSIONS=$(nix run nixpkgs#yq --  '.postgres_major[]' ansible/vars.yml | nix run nixpkgs#jq -- -R -s -c 'split("\n")[:-1]')
+          VERSIONS=$(nix run nixpkgs#yq --  '.postgres_major[]' ansible/vars.yml | nix run nixpkgs#jq --  -R -s -c 'split("\n")[:-1]')
           echo "postgres_versions=$VERSIONS" >> $GITHUB_OUTPUT
 
   test-ami-nix:
@@ -61,6 +61,7 @@ jobs:
 
       - name: Set PostgreSQL version environment variable
         run: echo "POSTGRES_MAJOR_VERSION=${{ matrix.postgres_version }}" >> $GITHUB_ENV
+
 
       - name: Generate common-nix.vars.pkr.hcl
         run: |

.gitignore

@@ -23,4 +23,4 @@ result*
 .idea/
 .vscode/
 
-db
+db

ansible/files/postgres_prestart.sh.j2

@@ -1,9 +1,49 @@
 #!/bin/bash
 
+check_orioledb_enabled() {
+   local pg_conf="/etc/postgresql/postgresql.conf"
+   if [ ! -f "$pg_conf" ]; then
+       return 0
+    fi
+   grep "^shared_preload_libraries" "$pg_conf" | grep -c "orioledb" || return 0
+}
+
+get_shared_buffers() {
+    local opt_conf="/etc/postgresql-custom/generated-optimizations.conf"
+    if [ ! -f "$opt_conf" ]; then
+        return 0
+    fi
+    grep "^shared_buffers = " "$opt_conf" | cut -d "=" -f2 | tr -d ' ' || return 0
+}
+
+update_orioledb_buffers() {
+   local pg_conf="/etc/postgresql/postgresql.conf"
+   local value="$1"
+   if grep -q "^orioledb.main_buffers = " "$pg_conf"; then
+       sed -i "s/^orioledb.main_buffers = .*/orioledb.main_buffers = $value/" "$pg_conf"
+   else
+       echo "orioledb.main_buffers = $value" >> "$pg_conf"
+   fi
+}
+
+main() {
+   local has_orioledb=$(check_orioledb_enabled)
+   if [ "$has_orioledb" -lt 1 ]; then
+       return 0
+   fi
+   local shared_buffers_value=$(get_shared_buffers)
+   if [ ! -z "$shared_buffers_value" ]; then
+       update_orioledb_buffers "$shared_buffers_value"
+   fi
+}
+
+# Initial locale setup
 if [ $(cat /etc/locale.gen | grep -c en_US.UTF-8) -eq 0 ]; then
-    echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
+   echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
 fi
 
 if [ $(locale -a | grep -c en_US.utf8) -eq 0 ]; then
-    locale-gen
+   locale-gen
 fi
+
+main

ansible/tasks/setup-postgres.yml

@@ -204,23 +204,49 @@
     ansible_command_timeout: 60
   when: debpkg_mode
 
-- name: Initialize the database stage2_nix
-  become: yes
-  become_user: postgres
-  shell: source /var/lib/postgresql/.bashrc && /usr/lib/postgresql/bin/pg_ctl -D /var/lib/postgresql/data initdb -o "--allow-group-access" -o "--username=supabase_admin"
-  args:
-    executable: /bin/bash
-  environment:
-    LANG: en_US.UTF-8
-    LANGUAGE: en_US.UTF-8
-    LC_ALL: en_US.UTF-8
-    LC_CTYPE: en_US.UTF-8
-    LOCALE_ARCHIVE: /usr/lib/locale/locale-archive
-  vars:
-    ansible_command_timeout: 60
-    # Circumvents the following error:
-    # "Timeout (12s) waiting for privilege escalation prompt"
-  when: stage2_nix 
+- name: Check psql_version and modify supautils.conf and postgresql.conf if necessary
+  block:
+    - name: Check if psql_version is psql_orioledb
+      set_fact:
+        is_psql_oriole: "{{ psql_version in ['psql_orioledb-16', 'psql_orioledb-17'] }}"
+
+    - name: Initialize the database stage2_nix (non-orioledb)
+      become: yes
+      become_user: postgres
+      shell: source /var/lib/postgresql/.bashrc && /usr/lib/postgresql/bin/pg_ctl -D /var/lib/postgresql/data initdb -o "--allow-group-access" -o "--username=supabase_admin"
+      args:
+        executable: /bin/bash
+      environment:
+        LANG: en_US.UTF-8
+        LANGUAGE: en_US.UTF-8
+        LC_ALL: en_US.UTF-8
+        LC_CTYPE: en_US.UTF-8
+        LOCALE_ARCHIVE: /usr/lib/locale/locale-archive
+      vars:
+        ansible_command_timeout: 60
+      when: stage2_nix and not is_psql_oriole
+
+    - name: Initialize the database stage2_nix (orioledb)
+      become: yes
+      become_user: postgres
+      shell: >
+        source /var/lib/postgresql/.bashrc && initdb -D /var/lib/postgresql/data 
+        --allow-group-access 
+        --username=supabase_admin 
+        --locale-provider=icu 
+        --encoding=UTF-8 
+        --icu-locale=en_US.UTF-8 
+      args:
+        executable: /bin/bash
+      environment:
+        LANG: en_US.UTF-8
+        LANGUAGE: en_US.UTF-8
+        LC_ALL: en_US.UTF-8
+        LC_CTYPE: en_US.UTF-8
+        LOCALE_ARCHIVE: /usr/lib/locale/locale-archive
+      vars:
+        ansible_command_timeout: 60
+      when: stage2_nix and is_psql_oriole
 
 - name: copy PG systemd unit
   template:

ansible/tasks/stage2-setup-postgres.yml

@@ -4,11 +4,56 @@
 #     sudo -u postgres bash -c ". /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh && nix profile install nixpkgs#openjdk11"
 # It was decided to leave pljava disabled at https://github.com/supabase/postgres/pull/690 therefore removing this task
 
+- name: Check psql_version and modify supautils.conf and postgresql.conf if necessary
+  block:
+    - name: Check if psql_version is psql_orioledb-16
+      set_fact:
+        is_psql_oriole: "{{ psql_version in ['psql_orioledb-16', 'psql_orioledb-17'] }}"
+
+    - name: Remove specified extensions from postgresql.conf if oriole-16 build
+      ansible.builtin.command:
+        cmd: >
+          sed -i 's/ timescaledb,//g' 
+          /etc/postgresql/postgresql.conf
+      when: is_psql_oriole and stage2_nix
+      become: yes
+
+    - name: Remove specified extensions from supautils.conf if oriole-16 build
+      ansible.builtin.command:
+        cmd: >
+          sed -i 's/ timescaledb,//g; s/ vector,//g; s/ plv8,//g; s/ postgis,//g; s/ pgrouting,//g' 
+          /etc/postgresql-custom/supautils.conf
+      when: is_psql_oriole and stage2_nix
+      become: yes
+
+    - name: Remove db_user_namespace from postgresql.conf if oriole-xx build
+      ansible.builtin.command:
+        cmd: >
+          sed -i 's/db_user_namespace = off/#db_user_namespace = off/g;' 
+          /etc/postgresql/postgresql.conf
+      when: is_psql_oriole and stage2_nix
+      become: yes
+
+    - name: Append orioledb to shared_preload_libraries append within closing quote
+      ansible.builtin.command:
+        cmd: >
+          sed -i 's/\(shared_preload_libraries.*\)'\''\(.*\)$/\1, orioledb'\''\2/'
+          /etc/postgresql/postgresql.conf
+      when: is_psql_oriole and stage2_nix
+      become: yes
+
+    - name: Add default_table_access_method setting
+      ansible.builtin.lineinfile:
+        path: /etc/postgresql/postgresql.conf
+        line: "default_table_access_method = 'orioledb'"
+        state: present
+      when: is_psql_oriole and stage2_nix
+      become: yes
+
 - name: Install Postgres from nix binary cache
   become: yes
   shell: |
     sudo -u postgres bash -c ". /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh && nix profile install github:supabase/postgres/{{ git_commit_sha }}#{{psql_version}}/bin"
-#TODO (samrose) switch pg_prove sourcing to develop branch once PR is merged
   when: stage2_nix
 
 - name: Install pg_prove from nix binary cache
@@ -199,13 +244,19 @@
     recurse: yes
   when: stage2_nix
 
-- name: Recursively create symbolic links and set permissions for the contrib/postgis-* dir
-  shell: >
-    sudo mkdir -p /usr/lib/postgresql/share/postgresql/contrib && \
-    sudo find /var/lib/postgresql/.nix-profile/share/postgresql/contrib/ -mindepth 1 -type d -exec sh -c 'for dir do sudo ln -s "$dir" "/usr/lib/postgresql/share/postgresql/contrib/$(basename "$dir")"; done' sh {} + \
-    && chown -R postgres:postgres "/usr/lib/postgresql/share/postgresql/contrib/"
-  become: yes
-  when: stage2_nix
+- name: Check psql_version and run postgis linking if not oriole-xx
+  block:
+    - name: Check if psql_version is psql_orioledb-17
+      set_fact:
+        is_psql_oriole: "{{ psql_version == 'psql_orioledb-17' }}"
+
+    - name: Recursively create symbolic links and set permissions for the contrib/postgis-* dir
+      shell: >
+        sudo mkdir -p /usr/lib/postgresql/share/postgresql/contrib && \
+        sudo find /var/lib/postgresql/.nix-profile/share/postgresql/contrib/ -mindepth 1 -type d -exec sh -c 'for dir do sudo ln -s "$dir" "/usr/lib/postgresql/share/postgresql/contrib/$(basename "$dir")"; done' sh {} + \
+        && chown -R postgres:postgres "/usr/lib/postgresql/share/postgresql/contrib/"
+      become: yes
+      when: stage2_nix and not is_psql_oriole
 
 - name: Create symbolic links from /var/lib/postgresql/.nix-profile/share/postgresql/timezonesets to /usr/lib/postgresql/share/postgresql/timeszonesets
   file:
@@ -240,8 +291,16 @@
     line: pgsodium.getkey_script= '{{ pg_bindir }}/pgsodium_getkey.sh'
   when: stage2_nix
 
+- name: Create symbolic link for pgsodium_getkey script
+  file:
+    src: "/usr/lib/postgresql/bin/pgsodium_getkey.sh"
+    dest: "/usr/lib/postgresql/share/postgresql/extension/pgsodium_getkey"
+    state: link
+  become: yes
+  when: stage2_nix
+
 - name: Append GRN_PLUGINS_DIR to /etc/environment.d/postgresql.env
   ansible.builtin.lineinfile:
     path: /etc/environment.d/postgresql.env
     line: 'GRN_PLUGINS_DIR=/var/lib/postgresql/.nix-profile/lib/groonga/plugins'
-  become: yes
+  become: yes