Invocation of master_add_node() fails if citus wasn't created on a worker beforehand

[ivyazmitinov]

Description

Before 9.3.0 we used the following script to create a new database on an existing server:

CREATE DATABASE <db_name>;
SELECT run_command_on_workers($cmd$ CREATE DATABASE <db_name>; $cmd$);
\c <db_name>
CREATE EXTENSION citus;
SELECT master_add_node('worker_1_ip', 5432);
SELECT master_add_node('worker_2_ip', 5432);
SELECT run_command_on_workers($cmd$ CREATE EXTENSION citus; $cmd$);

Expected behavior

Database is created

Actual behavior

With 9.3.0 master_add_node fails with

WARNING:  function assign_distributed_transaction_id(integer, integer, unknown) does not exist
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

[heldopslippers]

We have the same problems.
Did you figure out how to fix the issue?

[ivyazmitinov]

@heldopslippers
Unfortunately, no. The only way now is to create the database and install the extension manually on all workers before the nodes configuration. So, now it goes like this:

CREATE DATABASE <db_name>;
-- ssh to workers and invoke
-- CREATE DATABASE <db_name>;
-- \c <db_name>
-- CREATE EXTENSION citus;
-- ALTER <user_name> PASSWORD '<old_pass>';
\c <db_name>
CREATE EXTENSION citus;
SELECT master_add_node('worker_1_ip', 5432);
SELECT master_add_node('worker_2_ip', 5432);

Also, for some reason a password for a user that invokes CREATE EXTENSION citus; on the new database on workers gets changed, so you have to alter it back.

[heldopslippers]

argh! Thanxs! You saved me a week probably ;)

[marcocitus]

Also, for some reason a password for a user that invokes CREATE EXTENSION citus; on the new database on workers gets changed, so you have to alter it back.

If you use different passwords between coordinator and worker, then it's best to disable that with citus.enable_alter_role_propagation = off.

[marcocitus]

The expectation is that the worker is up-and-running and has the Citus extension installed at master_add_node time. This is normally when reference tables are replicated, types are created, etc. We're gradually expanding the number of things that Citus replicates to workers, so even on an otherwise empty cluster some synchronization is done.

I think one thing that would be convenient for cluster set up is a function like run_command_on_workers that works on inactive nodes. You can then use master_add_inactive_node, run_command_on_inactive_workers, and master_activate_node.

[ivyazmitinov]

@marcocitus

The expectation is that the worker is up-and-running and has the Citus extension installed at master_add_node time.

If there is such the expectation now, it is better to adjust the docs accordingly, because the described way to create new databases is outdated.

[ivyazmitinov]

@marcocitus
Can confirm that citus.enable_alter_role_propagation = off worked, thanks for the hint :)

[marcocitus]

Makes sense, I opened an issue in our docs repo.

[rad1k4l]

@marcocitus Are citus, changes password for workers by enable_alter_role_propagation default value ?