Upgrading OpenSlides 4

[tanjix]

Hi Guys!

I recently upgraded my OS4 beta (I didn't note down which version) to OS4 RC version (4.0.0-rc-20221101-a23e23d).
Now, when I'm trying to login to OS with my account, it just keeps loading and after a while, I get a message at the bottom:

"Fehler: Missing 7 migrations to apply."

I assume, some migration work needs to be done (probably on pgsql?). However, I haven't found where and how to do it.

I tried playing around and found something on the openslides command, that says "migration". Trying to run a:

root@ga:~/openslides# ./openslides migrations migrate

ends with:

Error: connecting to gRPC server: creating gRPC client connection with grpc.DialContext(): context deadline exceeded

Could anyone assist in getting this resolved? Thanks!

[normanjaeckel]

I see that some documentation here is missing.

Please try the following steps:

• Make a backup so if can roll back if something went wrong.
• Start OpenSlides in a first terminal.
• Run openslides migrations --help in a second terminal. This shows you the resepctive commands.
• Run openslides migrations stats to see statistics.
• Then you have to decide whether just to run finalize or to use migrate and then, after finishing, run finalize. This depends on how large your database is and if there is any risk that the process tooks too long. Just try it out.

[tanjix]

Hello @normanjaeckel

thanks for your advise!

However, all commands that you suggest, end up with the error message I provided in my initial post:

Error: connecting to gRPC server: creating gRPC client connection with grpc.DialContext(): context deadline exceeded

Any suggestions?

Thanks!

[normanjaeckel]

The error message indicates that the command line manage tool openslides is not able to connect to a running OpenSlides instance. So the question is: Do you have a running instance (e. g. with docker-compose up)? Which host and port? With SSL or not? Maybe you have to use a custom --addressflag.

What can you see in Docker's logs?

If you started OpenSlides using the default way we described in the README than just try to check if everything is running with openslides check-server.

[tanjix]

Hello @normanjaeckel!

Do you have a running instance (e. g. with docker-compose up)

Yes, of course!

docker ps shows:

root@ga:/root/openslides# docker-compose up -d
Starting openslides_postgres_1 ... done
Starting openslides_redis_1           ... done
Starting openslides_media_1           ... done
Starting openslides_datastoreReader_1 ... done
Starting openslides_datastoreWriter_1 ... done
Starting openslides_autoupdate_1      ... done
Starting openslides_icc_1             ... done
Starting openslides_auth_1            ... done
Starting openslides_backendManage_1   ... done
Starting openslides_manage_1           ... done
Starting openslides_backendPresenter_1 ... done
Starting openslides_vote_1             ... done
Starting openslides_backendAction_1    ... done
Starting openslides_client_1           ... done
Starting openslides_proxy_1            ... done
root@ga:~/openslides# docker ps

CONTAINER ID   IMAGE                                                              COMMAND                  CREATED      STATUS                            PORTS
                   NAMES
1febba8c9f29   ghcr.io/openslides/openslides/openslides-proxy:latest              "/entrypoint caddy r…"   4 days ago   Up 3 seconds                      80/tcp, 443/tcp, 2019/tcp, a.b.c.d:443->8000/tcp   openslides_proxy_1
f6baa3042116   ghcr.io/openslides/openslides/openslides-client:latest             "/docker-entrypoint.…"   4 days ago   Up 4 seconds
                   openslides_client_1
1a8b58dd68c9   ghcr.io/openslides/openslides/openslides-backend:latest            "./entrypoint.sh pyt…"   4 days ago   Up 5 seconds                      9002-9003/tcp
                   openslides_backendAction_1
a295f8d23539   ghcr.io/openslides/openslides/openslides-backend:latest            "./entrypoint.sh pyt…"   4 days ago   Up 6 seconds
                   openslides_backendPresenter_1
5c37a2f1f00d   ghcr.io/openslides/openslides/openslides-vote:latest               "/vote"                  4 days ago   Up 6 seconds (health: starting)
                   openslides_vote_1
80434fd7e64d   ghcr.io/openslides/openslides/openslides-manage:latest             "/server"                4 days ago   Up 6 seconds (health: starting)
                   openslides_manage_1
22238bc69f3e   ghcr.io/openslides/openslides/openslides-backend:latest            "./entrypoint.sh pyt…"   4 days ago   Up 8 seconds
                   openslides_backendManage_1
5dec3d67c063   ghcr.io/openslides/openslides/openslides-auth:latest               "./entrypoint.sh nod…"   4 days ago   Up 7 seconds
                   openslides_auth_1
439313c671e3   ghcr.io/openslides/openslides/openslides-icc:latest                "/icc"                   4 days ago   Up 7 seconds (health: starting)
                   openslides_icc_1
5549549d0560   ghcr.io/openslides/openslides/openslides-datastore-reader:latest   "./entrypoint.sh /bi…"   4 days ago   Up 9 seconds
                   openslides_datastoreReader_1
6c5c9cde26ca   ghcr.io/openslides/openslides/openslides-media:latest              "./entrypoint.sh gun…"   4 days ago   Up 9 seconds
                   openslides_media_1
73eb3a1399e1   ghcr.io/openslides/openslides/openslides-datastore-writer:latest   "./entrypoint.sh /bi…"   4 days ago   Up 9 seconds
                   openslides_datastoreWriter_1
86f8162bf3fd   redis:latest                                                       "docker-entrypoint.s…"   4 days ago   Up 10 seconds
                   openslides_redis_1
53a1a9d0405e   postgres:11                                                        "docker-entrypoint.s…"   4 days ago   Up 10 seconds
                   openslides_postgres_1
root@ga:~/openslides#

I've replaced my public IP through a.b.c.d on the above output.

Which host and port?
With SSL or not?

The host is a public IP address, the port is 443 (SSL enabled)

If you started OpenSlides using the default way we described in the README than just try to check if everything is running with openslides check-server.

This fails with the same error message :(

root@ga:~/openslides# ./openslides check-server
Error: connecting to gRPC server: creating gRPC client connection with grpc.DialContext(): context deadline exceeded
root@ga:~/openslides#

Maybe you have to use a custom --address flag.

I've not seen a documentation on that nor anything on the help pages. How shall I use that and where?

Thanks!

[tanjix]

Ah! I found the --address thing

root@ga:~/openslides# ./openslides check-server -a a.b.c.d:443
Server is ready.
root@ga:~/openslides#

So, this works fine. I think I can continue from there, as the openslides --migrations migrate -a a.b.c.d:443 also does some stuff now!

Thanks!

[tanjix]

Hi again,

migration did now finish:

(...)
Position 1026 from MI 27 to MI 34 ...
Position 1027 from MI 27 to MI 34 ...
Position 1028 from MI 27 to MI 34 ...
Position 1029 from MI 27 to MI 34 ...
Done. Finalizing is still needed.
root@ga:~/openslides# ./openslides migrations finalize -a a.b.c.d:443
Progress:
Finalize migrations.
No migrations to apply, but finalizing is still needed. Current migration index: 34
Cleaning collectionfield helper tables...
Calculate helper tables...
Deleting all migration keyframes...
Swap events and migration_events tables...
Set the new migration index to 34...
Clean up migration data...
root@ga:~/openslides# 

However, after migration has been done, I still cannot login to OS. After entering my credentials and clicking 'login', there's just a circle moving but nothin happening any more, even after waiting several minutes.

[tanjix]

The console (running just docker-compose up) shows during my try to login:

datastoreReader_1   | [2022-11-20 14:48:18 +0000] [12] [INFO] FILTER-request: {'collection': 'user', 'filter': {'field': 'id', 'value': 2, 'operator': '='}, 'mapped_fields': ['username', 'password', 'id', 'is_active', 'meta_deleted']}
auth_1              | 2022-11-20T14:48:19.581Z http://my.hostname.com: POST -- /system/auth/login/
datastoreReader_1   | [2022-11-20 14:48:19 +0000] [12] [INFO] FILTER-request: {'collection': 'user', 'filter': {'field': 'username', 'value': 'tanjix', 'operator': '='}, 'mapped_fields': ['username', 'password', 'id', 'is_active', 'meta_deleted']}
auth_1              | 2022-11-20T14:48:19.628Z http://my.hostname.com: POST -- /system/auth/who-am-i/
datastoreReader_1   | [2022-11-20 14:48:19 +0000] [12] [INFO] FILTER-request: {'collection': 'user', 'filter': {'field': 'id', 'value': 2, 'operator': '='}, 'mapped_fields': ['username', 'password', 'id', 'is_active', 'meta_deleted']}

Does that help?

Apart from that, I also am getting tons of messages like:

proxy_1             | {"level":"error","ts":1668955704.653598,"logger":"http.log.error","msg":"dial tcp: lookup autoupdate on 127.0.0.11:53: no such host","request":{"remote_addr":"84.118.48.176:63480","proto":"HTTP/2.0","method":"GET","host":"my.hostname.com","uri":"/system/autoupdate/health","headers":{"Accept-Language":["de,en-US;q=0.7,en;q=0.3"],"Dnt":["1"],"Sec-Fetch-Dest":["empty"],"Te":["trailers"],"Ngsw-Bypass":["true"],"Sec-Fetch-Mode":["cors"],"Sec-Fetch-Site":["same-origin"],"Cookie":["refreshId=bearer%20eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzZXNzaW9uSWQiOiIxODcyOGFiNDdhMGRhMTQ2MjM3ZGVkMmM1OTdhOTBhYSIsInVzZXJJZCI6MiwiaWF0IjoxNjY4OTU1Njk5LCJleHAiOjE2NzE1NDc2OTl9.K-IGpOAaLgqCGQWuDuaz0l4ouc1Gs--P4GN0bWcUaCc"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:107.0) Gecko/20100101 Firefox/107.0"],"Accept":["*/*"],"Accept-Encoding":["gzip, deflate, br"],"Referer":["https://my.hostname.com/openslides-shared-worker.f761f22998231b77.js"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"my.hostname.com"}},"duration":0.011335852,"status":502,"err_id":"s619j86ru","err_trace":"reverseproxy.statusError (reverseproxy.go:783)"}`

I guess, that doesn't have anything to do with the login issue, but maybe you have any advise on that, as well?

[tanjix]

For testing purpose I did a completely new and clean install of OS4 and I encounter the same issue regarding not being able to login. I therefore suspect any other issue but nothing related to my migration / my data.

Any help would be appreciated!

Thanks!

[tanjix]

If I - on purpose - provide wrong credentials to log in, it says:

Username or password is incorrect.

But providing the correct credentials results in the above described problem.

[normanjaeckel]

Sorry for the late reply. I am not able to test this myself tonight nor am I tomorrow. Wednesday might be better ...

[normanjaeckel]

@ostcar Please have a look at the following error message. What can this be ("autoupdate ... no such host")?

Apart from that, I also am getting tons of messages like:
proxy_1 | {"level":"error","ts":1668955704.653598,"logger":"http.log.error","msg":"dial tcp: lookup autoupdate on 127.0.0.11:53: no such host","request":{"remote_addr":"84.118.48.176:63480","proto":"HTTP/2.0","method":"GET","host":"my.hostname.com","uri":"/system/autoupdate/health","headers":{"Accept-Language":["de,en-US;q=0.7,en;q=0.3"],"Dnt":["1"],"Sec-Fetch-Dest":["empty"],"Te":["trailers"],"Ngsw-Bypass":["true"],"Sec-Fetch-Mode":["cors"],"Sec-Fetch-Site":["same-origin"],"Cookie":["refreshId=bearer%20eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzZXNzaW9uSWQiOiIxODcyOGFiNDdhMGRhMTQ2MjM3ZGVkMmM1OTdhOTBhYSIsInVzZXJJZCI6MiwiaWF0IjoxNjY4OTU1Njk5LCJleHAiOjE2NzE1NDc2OTl9.K-IGpOAaLgqCGQWuDuaz0l4ouc1Gs--P4GN0bWcUaCc"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:107.0) Gecko/20100101 Firefox/107.0"],"Accept":["/"],"Accept-Encoding":["gzip, deflate, br"],"Referer":["https://my.hostname.com/openslides-shared-worker.f761f22998231b77.js"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"my.hostname.com"}},"duration":0.011335852,"status":502,"err_id":"s619j86ru","err_trace":"reverseproxy.statusError (reverseproxy.go:783)"}`

I guess, that doesn't have anything to do with the login issue, but maybe you have any advise on that, as well?

[normanjaeckel]

@tanjix Please provide your docker-compose.yml so I can check if we have an (undocumented) versioning issue.

[tanjix]

@normanjaeckel here it is.

Thanks!
docker-compose.txt

[ostcar]

The problem is, that the client can not connect to the autoupdate service. Instead of showing an error message, it tries again forever. I created an issue in the client repo, that a better error message is shown in the future.

This is the relevant part of the error message: dial tcp: lookup autoupdate on 127.0.0.11:53: no such host. It means, that the proxy tries to redirect the connection to the autoupdate serive, but can not find out the IP-Address of the service.

If you google that error message, it seems, that many docker users have that issue. For some, it helps to turn of VPN. I also had a simular error in the past and it helped (for some reason) to change the DNS-Server of my host-maschine to '8.8.8.8'.

[tanjix]

Hi @ostcar thanks for advising about the issue!

For some, it helps to turn of VPN

I am not using VPN on that server

to change the DNS-Server of my host-maschine to '8.8.8.8'.

My DNS servers already are on 8.8.8.8 and 8.8.4.4

I also tried adding a line to /etc/hosts like:

127.0.0.11 autoupdate

None of all these steps helped out.

Thanks!

[ostcar]

Since this is a docker issue, I have no idea how to help you.

[tanjix]

@normanjaeckel Any update on this? Thanks!

[normanjaeckel]

Sorry for the late answer. If this is still relevant: You have an old version of the docker-compose.yml. You should create a new one using the latest version of the openslides tool. (This way you can enable the postgres password as secret for the autoupdate service, which is required now and was not in the past.)

[tanjix]

So, what I've done now:

• downloaded the latest version of the openslides tool
• made a backup of my current docker-compose.yml
• let the openslides too recreate it (openslides config .)
• modified the "ports" section in that newly created file to get OpenSlides working on my public IP instead of 127.0.0.1
• tried to run the migration by doing: openslides migrations finalize -a a.b.c.d:443

This ends up with:

Running migrations.
Datastore is empty, nothing to do.

I assume, there's now an issue with the newly created docker-compose.yml file?

Can you please advise and guide me through the correct steps to get this sorted? Thank you!

[tanjix]

@normanjaeckel Any advise please? Thanks for your help!

[normanjaeckel]

Sorry.

We recently changed the location of the postgres volume from a path mounted volume to a docker volume (integrated).

I think you have to revert this small thing and use your old postgres volume again.

Compare the new compose file with your old postgres section.

  postgres:
    image: postgres:11
    environment:
      << : *default-environment
      POSTGRES_DB: openslides
      POSTGRES_USER: openslides
      POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
      PGDATA: /var/lib/postgresql/data/pgdata
    networks:
      - data
    user: 0:0
    secrets:
      - postgres_password
    volumes:
      - ./db-data:/var/lib/postgresql/data

You will (hopefully) see some differences. Try to use your old postgres section.

I am sorry for this undocumented breaking changes and I will issue this to our team to make it better next time.

[normanjaeckel]

@peb-adr Maybe let us discuss this topic when there is some time ...