Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support podman based development without permission issues #1421

Open
XSpielinbox opened this issue Aug 8, 2024 · 6 comments
Open

support podman based development without permission issues #1421

XSpielinbox opened this issue Aug 8, 2024 · 6 comments
Labels
feature New feature or request priority: eventually

Comments

@XSpielinbox
Copy link

XSpielinbox commented Aug 8, 2024

Describe the bug
Just running podman compose -f docker-compose.local.yml up --build when trying to run the the server locally on a laptop with plenty of system resources takes 25min until it's in a state of meilisearch running, but nothing is changing anymore, but rust consuming 100% CPU for the whole time from starting the build, if rust and cargo are not installed on the local system.
If rust and cargo are installed on the local system, this tremendous resource usage by rust is gone, but Node.js now uses significant amounts of ram (but by fare not everything that is available, so in case this is the limiting factor, I would for one question why it needs that much and if it does why it does not use it, if available to speed it up) and takes forever, before after about 25min again ending in a state where meilisearch and api in turns logging the same HTTP requests and errors indefinitely and nothing is changing anymore.

Always showing up, no matter what I do

[...]
 ERROR  (node:12) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 101 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
(Use node --trace-warnings ... to show where the warning was created)
 ERROR  (node:12) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 102 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
 ERROR  (node:12) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 103 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
 ERROR  (node:12) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 104 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
 ERROR  (node:12) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 105 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
 ERROR  (node:12) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 106 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
 ERROR  (node:12) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 107 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
 ERROR  (node:12) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 108 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
 ERROR  (node:12) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 109 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
 ERROR  (node:12) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 110 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
 ERROR  (node:12) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 111 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
 ERROR  (node:12) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 112 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
 ERROR  (node:12) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 113 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
 ERROR  (node:12) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 114 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
 ERROR  (node:12) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 115 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
 ERROR  (node:12) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 116 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
 ERROR  (node:12) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 117 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
[...]
 ERROR  [unhandledRejection] [GET] "http://api:3003/api/search?q=&lang=en&pre_highlight=%3Cb+class%3D%27text-blue%27%3E&post_highlight=%3C%2Fb%3E": <no response> fetch failed
[...]
 ERROR  [nitro] [unhandledRejection] [GET] "http://api:3003/api/search?q=&lang=en&pre_highlight=%3Cb+class%3D%27text-blue%27%3E&post_highlight=%3C%2Fb%3E": <no response> fetch failed
[...]
[martin]             | [2024-08-08T[...]Z ERROR martin] Unable to load config file /map/config.yaml: Permission denied (os error 13)
[...]
[osm2pgsql]          | Traceback (most recent call last):
[osm2pgsql]          |   File "/usr/local/bin/osm2pgsql-replication", line 724, in <module>
[osm2pgsql]          |     sys.exit(main())
[osm2pgsql]          |              ^^^^^^
[osm2pgsql]          |   File "/usr/local/bin/osm2pgsql-replication", line 704, in main
[osm2pgsql]          |     with DBConnection(args) as db:
[osm2pgsql]          |          ^^^^^^^^^^^^^^^^^^
[osm2pgsql]          |   File "/usr/local/bin/osm2pgsql-replication", line 121, in __init__
[osm2pgsql]          |     self.conn = psycopg.connect(dbname=args.database, user=args.username,
[osm2pgsql]          |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[osm2pgsql]          |   File "/venv/lib/python3.12/site-packages/psycopg2/__init__.py", line 122, in connect
[osm2pgsql]          |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
[osm2pgsql]          |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[osm2pgsql]          | psycopg2.OperationalError: connection to server on socket "/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
[osm2pgsql]          | 	Is the server running locally and accepting connections on that socket?
[osm2pgsql]          | 

If rust and cargo are installed, Node.js then produces this error, despite Python 3.12 being installed on the host system by default and Node.js beeing installed on the Host as well

.../node_modules/tree-sitter-json install: gyp ERR! find Python 
.../node_modules/tree-sitter-json install: gyp ERR! find Python Python is not set from command line or npm configuration
.../node_modules/tree-sitter-json install: gyp ERR! find Python Python is not set from environment variable PYTHON
.../node_modules/tree-sitter-json install: gyp ERR! find Python checking if "python3" can be used
.../node_modules/tree-sitter-json install: gyp ERR! find Python - executable path is ""
.../node_modules/tree-sitter-json install: gyp ERR! find Python - "" could not be run
.../node_modules/tree-sitter-json install: gyp ERR! find Python checking if "python" can be used
.../node_modules/tree-sitter-json install: gyp ERR! find Python - executable path is ""
.../node_modules/tree-sitter-json install: gyp ERR! find Python - "" could not be run
.../node_modules/tree-sitter-json install: gyp ERR! find Python 
.../node_modules/tree-sitter-json install: gyp ERR! find Python **********************************************************
.../node_modules/tree-sitter-json install: gyp ERR! find Python You need to install the latest version of Python.
.../node_modules/tree-sitter-json install: gyp ERR! find Python Node-gyp should be able to find and use Python. If not,
.../node_modules/tree-sitter-json install: gyp ERR! find Python you can try one of the following options:
.../node_modules/tree-sitter-json install: gyp ERR! find Python - Use the switch --python="/path/to/pythonexecutable"
.../node_modules/tree-sitter-json install: gyp ERR! find Python (accepted by both node-gyp and npm)
.../node_modules/tree-sitter-json install: gyp ERR! find Python - Set the environment variable PYTHON
.../node_modules/tree-sitter-json install: gyp ERR! find Python - Set the npm configuration variable python:
.../node_modules/tree-sitter-json install: gyp ERR! find Python npm config set python "/path/to/pythonexecutable"
.../node_modules/tree-sitter-json install: gyp ERR! find Python For more information consult the documentation at:
.../node_modules/tree-sitter-json install: gyp ERR! find Python https://github.com/nodejs/node-gyp#installation
.../node_modules/tree-sitter-json install: gyp ERR! find Python **********************************************************
.../node_modules/tree-sitter-json install: gyp ERR! find Python 
.../node_modules/tree-sitter-json install: gyp ERR! configure error 
.../node_modules/tree-sitter-json install: gyp ERR! stack Error: Could not find any Python installation to use
.../node_modules/tree-sitter-json install: gyp ERR! stack at PythonFinder.fail (/root/.cache/node/corepack/v1/pnpm/9.7.0/dist/node_modules/node-gyp/lib/find-python.js:306:11)
.../node_modules/tree-sitter-json install: gyp ERR! stack at PythonFinder.findPython (/root/.cache/node/corepack/v1/pnpm/9.7.0/dist/node_modules/node-gyp/lib/find-python.js:164:17)
.../node_modules/tree-sitter-json install: gyp ERR! stack at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
.../node_modules/tree-sitter-json install: gyp ERR! stack at async configure (/root/.cache/node/corepack/v1/pnpm/9.7.0/dist/node_modules/node-gyp/lib/configure.js:27:18)
.../node_modules/tree-sitter-json install: gyp ERR! stack at async run (/root/.cache/node/corepack/v1/pnpm/9.7.0/dist/node_modules/node-gyp/bin/node-gyp.js:81:18)
.../node_modules/tree-sitter-json install: gyp ERR! System Linux 6.9.12-200.fc40.x86_64
.../node_modules/tree-sitter-json install: gyp ERR! command "/usr/local/bin/node" "/root/.cache/node/corepack/v1/pnpm/9.7.0/dist/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
.../node_modules/tree-sitter-json install: gyp ERR! cwd /app/node_modules/.pnpm/[email protected]/node_modules/tree-sitter-json
.../node_modules/tree-sitter-json install: gyp ERR! node -v v21.7.3
.../node_modules/tree-sitter-json install: gyp ERR! node-gyp -v v10.1.0
.../node_modules/tree-sitter-json install: gyp ERR! not ok 
.../node_modules/tree-sitter-json install: Failed
[...]
.../node_modules/tree-sitter-yaml install: gyp ERR! find Python 
.../node_modules/tree-sitter-yaml install: gyp ERR! find Python Python is not set from command line or npm configuration
.../node_modules/tree-sitter-yaml install: gyp ERR! find Python Python is not set from environment variable PYTHON
.../node_modules/tree-sitter-yaml install: gyp ERR! find Python checking if "python3" can be used
.../node_modules/tree-sitter-yaml install: gyp ERR! find Python - executable path is ""
.../node_modules/tree-sitter-yaml install: gyp ERR! find Python - "" could not be run
.../node_modules/tree-sitter-yaml install: gyp ERR! find Python checking if "python" can be used
.../node_modules/tree-sitter-yaml install: gyp ERR! find Python - executable path is ""
.../node_modules/tree-sitter-yaml install: gyp ERR! find Python - "" could not be run
.../node_modules/tree-sitter-yaml install: gyp ERR! find Python 
.../node_modules/tree-sitter-yaml install: gyp ERR! find Python **********************************************************
.../node_modules/tree-sitter-yaml install: gyp ERR! find Python You need to install the latest version of Python.
.../node_modules/tree-sitter-yaml install: gyp ERR! find Python Node-gyp should be able to find and use Python. If not,
.../node_modules/tree-sitter-yaml install: gyp ERR! find Python you can try one of the following options:
.../node_modules/tree-sitter-yaml install: gyp ERR! find Python - Use the switch --python="/path/to/pythonexecutable"
.../node_modules/tree-sitter-yaml install: gyp ERR! find Python (accepted by both node-gyp and npm)
.../node_modules/tree-sitter-yaml install: gyp ERR! find Python - Set the environment variable PYTHON
.../node_modules/tree-sitter-yaml install: gyp ERR! find Python - Set the npm configuration variable python:
.../node_modules/tree-sitter-yaml install: gyp ERR! find Python npm config set python "/path/to/pythonexecutable"
.../node_modules/tree-sitter-yaml install: gyp ERR! find Python For more information consult the documentation at:
.../node_modules/tree-sitter-yaml install: gyp ERR! find Python https://github.com/nodejs/node-gyp#installation
.../node_modules/tree-sitter-yaml install: gyp ERR! find Python **********************************************************
.../node_modules/tree-sitter-yaml install: gyp ERR! find Python 
.../node_modules/tree-sitter-yaml install: gyp ERR! configure error 
.../node_modules/tree-sitter-yaml install: gyp ERR! stack Error: Could not find any Python installation to use
.../node_modules/tree-sitter-yaml install: gyp ERR! stack at PythonFinder.fail (/root/.cache/node/corepack/v1/pnpm/9.7.0/dist/node_modules/node-gyp/lib/find-python.js:306:11)
.../node_modules/tree-sitter-yaml install: gyp ERR! stack at PythonFinder.findPython (/root/.cache/node/corepack/v1/pnpm/9.7.0/dist/node_modules/node-gyp/lib/find-python.js:164:17)
.../node_modules/tree-sitter-yaml install: gyp ERR! stack at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
.../node_modules/tree-sitter-yaml install: gyp ERR! stack at async configure (/root/.cache/node/corepack/v1/pnpm/9.7.0/dist/node_modules/node-gyp/lib/configure.js:27:18)
.../node_modules/tree-sitter-yaml install: gyp ERR! stack at async run (/root/.cache/node/corepack/v1/pnpm/9.7.0/dist/node_modules/node-gyp/bin/node-gyp.js:81:18)
.../node_modules/tree-sitter-yaml install: gyp ERR! System Linux 6.9.12-200.fc40.x86_64
.../node_modules/tree-sitter-yaml install: gyp ERR! command "/usr/local/bin/node" "/root/.cache/node/corepack/v1/pnpm/9.7.0/dist/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
.../node_modules/tree-sitter-yaml install: gyp ERR! cwd /app/node_modules/.pnpm/[email protected]/node_modules/tree-sitter-yaml
.../node_modules/tree-sitter-yaml install: gyp ERR! node -v v21.7.3
.../node_modules/tree-sitter-yaml install: gyp ERR! node-gyp -v v10.1.0
.../node_modules/tree-sitter-yaml install: gyp ERR! not ok 
.../node_modules/tree-sitter-yaml install: Failed
[...]
.../node_modules/tree-sitter install: gyp ERR! find Python 
.../node_modules/tree-sitter install: gyp ERR! find Python Python is not set from command line or npm configuration
.../node_modules/tree-sitter install: gyp ERR! find Python Python is not set from environment variable PYTHON
.../node_modules/tree-sitter install: gyp ERR! find Python checking if "python3" can be used
.../node_modules/tree-sitter install: gyp ERR! find Python - executable path is ""
.../node_modules/tree-sitter install: gyp ERR! find Python - "" could not be run
.../node_modules/tree-sitter install: gyp ERR! find Python checking if "python" can be used
.../node_modules/tree-sitter install: gyp ERR! find Python - executable path is ""
.../node_modules/tree-sitter install: gyp ERR! find Python - "" could not be run
.../node_modules/tree-sitter install: gyp ERR! find Python 
.../node_modules/tree-sitter install: gyp ERR! find Python **********************************************************
.../node_modules/tree-sitter install: gyp ERR! find Python You need to install the latest version of Python.
.../node_modules/tree-sitter install: gyp ERR! find Python Node-gyp should be able to find and use Python. If not,
.../node_modules/tree-sitter install: gyp ERR! find Python you can try one of the following options:
.../node_modules/tree-sitter install: gyp ERR! find Python - Use the switch --python="/path/to/pythonexecutable"
.../node_modules/tree-sitter install: gyp ERR! find Python (accepted by both node-gyp and npm)
.../node_modules/tree-sitter install: gyp ERR! find Python - Set the environment variable PYTHON
.../node_modules/tree-sitter install: gyp ERR! find Python - Set the npm configuration variable python:
.../node_modules/tree-sitter install: gyp ERR! find Python npm config set python "/path/to/pythonexecutable"
.../node_modules/tree-sitter install: gyp ERR! find Python For more information consult the documentation at:
.../node_modules/tree-sitter install: gyp ERR! find Python https://github.com/nodejs/node-gyp#installation
.../node_modules/tree-sitter install: gyp ERR! find Python **********************************************************
.../node_modules/tree-sitter install: gyp ERR! find Python 
.../node_modules/tree-sitter install: gyp ERR! configure error 
.../node_modules/tree-sitter install: gyp ERR! stack Error: Could not find any Python installation to use
.../node_modules/tree-sitter install: gyp ERR! stack at PythonFinder.fail (/root/.cache/node/corepack/v1/pnpm/9.7.0/dist/node_modules/node-gyp/lib/find-python.js:306:11)
.../node_modules/tree-sitter install: gyp ERR! stack at PythonFinder.findPython (/root/.cache/node/corepack/v1/pnpm/9.7.0/dist/node_modules/node-gyp/lib/find-python.js:164:17)
.../node_modules/tree-sitter install: gyp ERR! stack at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
.../node_modules/tree-sitter install: gyp ERR! stack at async configure (/root/.cache/node/corepack/v1/pnpm/9.7.0/dist/node_modules/node-gyp/lib/configure.js:27:18)
.../node_modules/tree-sitter install: gyp ERR! stack at async run (/root/.cache/node/corepack/v1/pnpm/9.7.0/dist/node_modules/node-gyp/bin/node-gyp.js:81:18)
.../node_modules/tree-sitter install: gyp ERR! System Linux 6.9.12-200.fc40.x86_64
.../node_modules/tree-sitter install: gyp ERR! command "/usr/local/bin/node" "/root/.cache/node/corepack/v1/pnpm/9.7.0/dist/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
.../node_modules/tree-sitter install: gyp ERR! cwd /app/node_modules/.pnpm/[email protected]/node_modules/tree-sitter
.../node_modules/tree-sitter install: gyp ERR! node -v v21.7.3
.../node_modules/tree-sitter install: gyp ERR! node-gyp -v v10.1.0
.../node_modules/tree-sitter install: gyp ERR! not ok 
.../node_modules/tree-sitter install: Failed

And it end in an endless loop of

[meilisearch]        | 2024-08-08T[...]Z  INFO HTTP request{method=GET host="localhost:7700" route=/health query_parameters= user_agent=curl/8.5.0 status_code=200}: meilisearch: close time.busy=140µs time.idle=24.4µs
[api]                | {"timestamp":"2024-08-08T[...]Z","level":"DEBUG","fields":{"message":"starting new connection: https://campus.tum.de/","log.target":"reqwest::connect","log.module_path":"reqwest::connect","log.file":"/usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.27/src/connect.rs","log.line":449},"target":"reqwest::connect","span":{"name":"fetch_new_oauth_token"},"spans":[{"name":"all_entries"},{"ids":"[LocationKey { key: \"0509.EG.980\" }, LocationKey { key: \"0506.EG.606\" }, LocationKey { key: \"8120.01.101\" }, ...]","name":"request_events"},{"self":"APIRequestor { oauth_token: None }","name":"try_refresh_token"},{"name":"fetch_new_oauth_token"}]}
[api]                | {"timestamp":"2024-08-08T[...]Z","level":"ERROR","fields":{"message":"retrying to get oauth token because Server returned error response\n\nStack backtrace:\n   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from\n   1: navigatum_server::calendar::connectum::APIRequestor::fetch_new_oauth_token::{{closure}}::{{closure}}\n   2: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll\n   3: navigatum_server::calendar::connectum::APIRequestor::try_refresh_token::{{closure}}::{{closure}}\n   4: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll\n   5: navigatum_server::calendar::refresh::request_events::{{closure}}::{{closure}}\n   6: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll\n   7: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll\n   8: navigatum_server::run_maintenance_work::{{closure}}\n   9: tokio::runtime::task::core::Core<T,S>::poll\n  10: tokio::runtime::task::harness::Harness<T,S>::poll\n  11: tokio::runtime::runtime::Runtime::block_on\n  12: navigatum_server::main\n  13: std::sys_common::backtrace::__rust_begin_short_backtrace\n  14: std::rt::lang_start::{{closure}}\n  15: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once\n             at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/core/src/ops/function.rs:284:13\n  16: std::panicking::try::do_call\n             at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/panicking.rs:559:40\n  17: std::panicking::try\n             at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/panicking.rs:523:19\n  18: std::panic::catch_unwind\n             at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/panic.rs:149:14\n  19: std::rt::lang_start_internal::{{closure}}\n             at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/rt.rs:141:48\n  20: std::panicking::try::do_call\n             at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/panicking.rs:559:40\n  21: std::panicking::try\n             at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/panicking.rs:523:19\n  22: std::panic::catch_unwind\n             at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/panic.rs:149:14\n  23: std::rt::lang_start_internal\n             at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/rt.rs:141:20\n  24: main\n  25: <unknown>"},"target":"navigatum_server::calendar::refresh","span":{"ids":"[LocationKey { key: \"0509.EG.980\" }, LocationKey { key: \"0506.EG.606\" }, LocationKey { key: \"8120.01.101\" }, ...]","name":"request_events"},"spans":[{"name":"all_entries"},{"ids":"[LocationKey { key: \"0509.EG.980\" }, LocationKey { key: \"0506.EG.606\" }, LocationKey { key: \"8120.01.101\" }, ...]","name":"request_events"}]}

Also when trying to the osm-download

via these commands

wget -O data.pbf https://download.geofabrik.de/europe/germany/bayern/oberbayern-latest.osm.pbf
podman run -it -v $(pwd):/data -e PGPASSWORD=CHANGE_ME --network="host" iboates/osm2pgsql:latest osm2pgsql --create --slim --database postgres --user     postgres --host 127.0.0.1 --port 5432 /data/data.pbf --hstore --hstore-add-index --hstore-column raw
podman run -it -v $(pwd):/data -e PGPASSWORD=CHANGE_ME --network="host" iboates/osm2pgsql:latest replication init          --database postgres --username postgres --host localhost --port 5432

then in the hope this would solve the errors as one cannot run it before starting the build one still only gets an error

2024-08-08 [...]  ERROR: Open failed for '/data/data.pbf': Permission denied
2024-08-08 [...] [CRITICAL]: Cannot get timestamp from database. Use --start-at to set an explicit date.

To Reproduce
Steps to reproduce the behavior:

Just try to run podman compose -f docker-compose.local.yml up --build as per install instructions. Podman should be fully compatible to docker.

Expected behavior
It either produces a running working server directly or finishes in <5min and I can start the server in a different way.

Desktop (please complete the following information):

  • OS: Fedora Linux 40
  • Browser Firefox
  • Version 128.0.3
@XSpielinbox XSpielinbox added the bug Something isn't working label Aug 8, 2024
@CommanderStorm
Copy link
Member

CommanderStorm commented Aug 8, 2024

Thanks for the thurough report ^^

Okay, I invested a bit of time today to speed this up.
I am nowhere near fixing this, but let's start at the beginning

Tip

Some of the issues are likely permission issues between docker/podman.
Switching to docker should unblock you ^^

but nothing is changing anymore, but rust consuming 100% CPU for the whole time from starting the build

The first build using cargo is slow. Because Speed, Compile-times and safety are not great friends, Rust chose to sacrifice Compile-time.
Rust also builds the entire dependency tree from scratch, which is also slow.
The thing is that most operations can preserve the cache and only the last stage needs to run. While our Docker image is relatively crude in this department, we also try to provide incremental compilation.

It might be noted that local dependencies cannot should never impact the actual Docker build. That would make the Docker mantra of having a repeatable environment moot.

=> There is not much that can be done in this department other than waiting for language-features (polonius,cranelift,paralell-frontend,...) to make this suck less. I have done a few optimizations today, but those are likely not major

Always showing up, no matter what I do
Unable to load config file /map/config.yaml: Permission denied (os error 13)

Seems weird. This might be related to running podman vs running docker.
I am going to try to replicate/debug the difference between the podman and docker permission systems.

If rust and cargo are installed, Node.js then produces this error, despite Python 3.12 being installed on the host system by default and Node.js beeing installed on the Host as well

It might be that the webclient Dockerfile is not currently capable of being run with a rootless user. I will have to debug this more in detail.

And it end in an endless loop of

That endless loop means that the server has "successfully" initialized the database and is now trying to sync the calendar.
=> I have patched the environment variables to show a better error message and to not produce the same error in a loop. If you want to interact with the calendar, I can send you the API-Credentials ^^

About the Meilisearch /health-check, I cannot really do anything there. I have discussed this with the Meilisearch team here, and they think this is intentional/good.
=> I have reduced the frequency with which this endpoint gets called in dev

Also when trying to the osm-download
then in the hope this would solve the errors as one cannot run it before starting the build one still only gets an error

Seems to be another instance of the issue noted above with the permission difference between docker and podman.
Will look into this.

It either produces a running working server directly or finishes in <5min and I can start the server in a different way.

Honestly, for an initial build, that is likely not achievable with the current toolchain.
I know some avenues that can improve the startup time:

  • integrating the replication Dockerfile => no second command to execute
  • integrating the data backend into the server => less startup time
  • only building release builds in the CI/CD pipeline, possibly on a faster PC with caching
  • waiting for a faster Rust compiler
  • ???

@XSpielinbox
Copy link
Author

@CommanderStorm Thanks for your effort!

It would be awesome, if it would work with podman as well, but I would consider this a feature request then. Feel free to change this issue accordingly.

For the startup time: the 5min where just an arbitrary time as it was not clear to me when it is finished now, if it's <10min this would of course not be ideal, but having to wait 25min until this build finishes and in between having times where the log output does not change, but this Javascript Vue.js/Nitro stuff does whatever on earth that takes so long and does not give any progress, does not raise one's confidence in that I am actually on the right way to get a development setup running and makes it really hard to develop, as one is only waiting for these builds...

I don't want to interact with the calendar. I just wanted to get a local setup running to be able to iteratively figure out the coordinates that would be needed for the rooms. If you would have guidance on what would be needed for that, this would probably be most helpful.

@CommanderStorm
Copy link
Member

I just wanted to get a local setup running to be able to iteratively figure out the coordinates

you can append ?debug=true to the urls 😉
No need to setup the dev-environment for this..

@CommanderStorm CommanderStorm changed the title [Bug] ERROR during server build support podman based development without permission issues Aug 8, 2024
@CommanderStorm CommanderStorm added feature New feature or request and removed bug Something isn't working labels Aug 8, 2024
@XSpielinbox
Copy link
Author

I just wanted to get a local setup running to be able to iteratively figure out the coordinates

you can append ?debug=true to the urls 😉 No need to setup the dev-environment for this..

Ok, thank you, but how do I see where the location of the coordinates in the URL is and what is this second marker for that appears in this debug mode?

@CommanderStorm
Copy link
Member

Ah sorry, forgot to tell:
The result is saved to the copy paste-buffer, formatted as needed ^^

@CommanderStorm
Copy link
Member

FYI: I think I have improved the setup significantly, should be much more pleasant.
Some of the permission issues (such as the martin-intialisation) should be fixed. For others this needs deeper digging into (not a huge prio)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request priority: eventually
Projects
None yet
Development

No branches or pull requests

2 participants