Docker Build Failing with NPM Errors

[engineerdawood]

Bug description

Issue: Docker Installation Failing with NPM Error Despite System Upgrade

Environment:

• Version: 4.0
• Branch: 4.0 (confirmed up-to-date with origin/4.0)
• Original System Specs: 2 vCPU, 4GB RAM
• Upgraded System Specs: 4 vCPU, 8GB RAM

Problem Description:
I initially attempted to build and run my project using Docker on a system with 2 vCPU and 4GB of RAM. The Docker build failed due to insufficient memory, specifically when docker is running npm commands to build Node dependencies.

To address this, I upgraded my system to 4 vCPU and 8GB of RAM. However, after multiple attempts, I am still encountering the following NPM errors during the build process.

I’ve tried rebuilding the Docker container several times, but the error persists. Can anyone provide guidance on what might be causing this issue or suggest potential solutions?

Git Status:
On branch 4.0 Your branch is up to date with 'origin/4.0'.

Any help would be greatly appreciated!

superset_db | 2024-10-26 12:33:48.102 UTC [67] LOG: checkpoint starting: time superset_db | 2024-10-26 12:33:52.300 UTC [67] LOG: checkpoint complete: wrote 43 buffers (0.3%); 0 WAL file(s) added, 0 removed, 0 recycled; write=4.017 s, sync=0.013 s, total=4.199 s; sync files=11, longest=0.010 s, average=0.002 s; distance=252 kB, estimate=252 kB superset_node | npm ERR! code FETCH_ERROR superset_node | npm ERR! errno FETCH_ERROR superset_node | npm ERR! invalid json response body at https://registry.npmjs.org/@storybook%2faddon-outline reason: Invalid response body while trying to fetch https://registry.npmjs.org/@storybook%2faddon-outline: Socket timeout superset_node | superset_node | superset_node | npm ERR! A complete log of this run can be found in: superset_node | npm ERR! /root/.npm/_logs/2024-10-26T12_30_06_825Z-debug-0.log

Screenshots/recordings

No response

Superset version

4.0.2

Python version

3.11

Node version

16

Browser

Chrome

Additional context

No response

Checklist

• I have searched Superset docs and Slack and didn't find a solution to my problem.
• I have searched the GitHub issue tracker and didn't find a similar bug report.
• I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

[dosubot]

Hey @engineerdawood! I'm here to help you with any bugs, questions, or contributions you have for this project. Let's work together to solve this issue!

The issue you're encountering with the Docker build failing due to NPM errors might be related to the Node.js version you're using. For Superset 4.0.x, it's important to use Node.js version 18.x, as using an unsupported version like 16 can lead to build issues ^[1].

To resolve this, update your Dockerfile to use Node.js version 18.x. Here's a snippet you can use:

# Install nodejs for custom build
# https://nodejs.org/en/download/package-manager/
RUN set -eux; \
    curl -sL https://deb.nodesource.com/setup_18.x | bash -; \
    apt-get install -y nodejs; \
    node --version;

Make sure to rebuild your Docker container after making this change. This should help resolve the NPM errors you're experiencing.

To continue talking to Dosu, mention @dosu.

[samarsrivastav]

hey I am getting this error

Gracefully stopping... (press Ctrl+C again to force)
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/app/scripts/check-env.py": stat /app/scripts/check-env.py: no such file or directory: unknown

[heaven890922]

hey I am getting this error

Gracefully stopping... (press Ctrl+C again to force)
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/app/scripts/check-env.py": stat /app/scripts/check-env.py: no such file or directory: unknown

Have you resolved this issue yet? I am also experiencing the same problem.

[gokhandeveloper]

I was able to fix this by commenting out the health check in docker-compose.yml.

x-superset-user: &superset-user root
x-superset-depends-on: &superset-depends-on
  - db
  - redis
#  - superset-checks

It looks like something is up with this apache/superset-cache:3.10-slim-bookworm
I tried launching it and it comes up with Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/app/scripts/check-env.py": stat /app/scripts/check-env.py: no such file or directory: unknown

[rusackas]

@mistercrunch might have some ideas regarding check-env.py issues.

[mistercrunch]

We recommend 16GB

$ python scripts/check-env.py
==================
System Information
==================
OS: Darwin 23.6.0
CPU: 12 cores at 3504.00 MHz
Docker Platform: Docker Desktop (Darwin)


====================================================================================================
Status Software                  Version Found             Ideal Range               Supported Range
====================================================================================================
✅ python                    3.10.13                   3.10.0 - 3.10.999         3.9.0 - 3.11.999
✅ npm                       10.8.3                    10.0.0 - 999.999.999      10.0.0 - 999.999.999
✅ node                      v20.16.0                  20.0.0 - 20.999.999       20.0.0 - 20.999.999
✅ docker                    27.2.0                    20.10.0 - 999.999.999     19.0.0 - 999.999.999
✅ docker-compose            2.29.2                    2.28.0 - 999.999.999      1.29.0 - 999.999.999
✅ git                       2.44.0                    2.30.0 - 999.999.999      2.20.0 - 999.999.999
✅ Memory: 16.00 GB

[mistercrunch]

I think we wanted to add scripts/check-env.py in some contexts so that we can catch this earlier in the build cycle and provide clear error messages (instead of processes just vanishing as they do when they get killed by the OS), but at the same time I don't want to make the current CI flows more complicated where we do have 16GB.

I thought about adding it to the docker-compose workflow but it's difficult since docker-compose runs a build prior to doing anything else, which points to running this inside the Dockerfile which isn't super elegant. Other option is to wrap docker-compose into some shell script that runs the check first.

My favorite solution is probably to improve the docs to reference check-env.py as a step prior to building, maybe mentioning the system requirements / assumptions for Superset too.

[webbsledge]

Commenting that line out doesn't work for me. I still get the same error. Also I have 48GB of RAM allocated.

[gokhandeveloper]

Best to delete all containers and do docker compose up again.

Commenting that line out doesn't work for me. I still get the same error. Also I have 48GB of RAM allocated.

[webbsledge]

@gokhandeveloper Had the same thought. I tried that multiple times to no avail.

[samarsrivastav]

Best to delete all containers and do docker compose up again.

Commenting that line out doesn't work for me. I still get the same error. Also I have 48GB of RAM allocated.

tried but not getting the application to run

[gokhandeveloper]

Actually comment out the whole step like this and retry.
Also as @mistercrunch suggested, can you run python scripts/check-eny.py ?
In my case it came up with these errors.
Traceback (most recent call last):

  File "/username/pycharms/superset-master/scripts/check-env.py", line 222, in <module>
    main()
  File "/Users/anonymous001/miniconda3/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)x
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/username/miniconda3/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/username/miniconda3/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/username/miniconda3/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/username/pycharms/superset-master/scripts/check-env.py", line 182, in main
    print(get_cpu_info())
          ^^^^^^^^^^^^^^
  File "/username/pycharms/superset-master/scripts/check-env.py", line 92, in get_cpu_info
    cpu_freq = psutil.cpu_freq()
               ^^^^^^^^^^^^^^^^^
  File "/username/miniconda3/lib/python3.11/site-packages/psutil/__init__.py", line 1869, in cpu_freq
    ret = _psplatform.cpu_freq()
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/username/miniconda3/lib/python3.11/site-packages/psutil/_psosx.py", line 179, in cpu_freq
    curr, min_, max_ = cext.cpu_freq()
                       ^^^^^^^^^^^^^^^

I was able to run this in pycharm by installing the psutil package in Intelij python interpreter. If you are on mac, make sure you also have xcode command line tools installed before doing any of this.

https://stackoverflow.com/questions/72619143/unable-to-import-psutil-on-m1-mac-with-miniforge-mach-o-file-but-is-an-incomp

Also newer docker desktop broke the sym links. You can use docker desktop to auto repair/clean these broken links.

Or you can manually check all versions listed in the check-env to make sure you have the right stuff installed which is what I ended up doing. But then ended up fixing the check-env issue because it bugged me as my "fix" wasn't really a fix haha.

#  superset-checks:
#    build:
#      context: .
#      target: python-base
#      cache_from:
#        - apache/superset-cache:3.10-slim-bookworm
#    container_name: superset_checks
#    command: ["/app/scripts/check-env.py"]
#    env_file:
#      - path: docker/.env # default
#        required: true
#      - path: docker/.env-local # optional override
#        required: false
#    user: *superset-user

Best to delete all containers and do docker compose up again.

Commenting that line out doesn't work for me. I still get the same error. Also I have 48GB of RAM allocated.

tried but not getting the application to run

[samarsrivastav]

hey @gokhandeveloper !! commenting that out is breaking the website for me

[vivien-crayon]

Same as @samarsrivastav when commenting out the checks. Containers start but the app is not working
Also tried to increase the RAM for docker to 16GB, not working either, I still get the same error

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/app/scripts/check-env.py": stat /app/scripts/check-env.py: no such file or directory: unknown

Platform: MacOS with docker desktop latest version (updated today)

[VadzimBelski-ScienceSoft]

I fixed this issue this way in Docker compose

changing the command, so its started well on my mac

  superset-checks:
    build:
      context: .
      target: python-base
      cache_from:
        - apache/superset-cache:3.10-slim-bookworm
    container_name: superset_checks
    command: ["true"]
    env_file:
      - path: docker/.env # default
        required: true
      - path: docker/.env-local # optional override
        required: false
    user: *superset-user
    healthcheck:
      disable: true

[fisjac]

Also experienced this, @VadzimBelski-ScienceSoft's solution fixed it.

[adeelamin15]

Also faced same issue. VadzimBelski-ScienceSoft's guidance fixed it

[regivaldo]

Thanks @VadzimBelski-ScienceSoft ! Your solution fixed it!

[webbsledge]

Thank you @VadzimBelski-ScienceSoft that resolved the issue for me as well.

[mistercrunch]

Also as @mistercrunch suggested, can you run python scripts/check-eny.py ?

Currently check-env.py assumes you've pip install requirements/development.txt, it would be great to enhance check-env.py to try/check whether you have the required deps and fail and provide a helpful error message.