Fly Instance runs out of memory on deployment

[JavoByte]

Have you experienced this bug with the latest version of the template?

Yes

Steps to Reproduce

• Create an application
• Deploy it to Fly.io
• Do some changes to the DB Schema
• Re-deploy

Expected Behavior

Migrations are applied and app starts normally

And possibly, prisma shouldn't be installed again with npx. It should already be there from dependency installing phase.

Actual Behavior

Instance runs out of memory while running

npx prisma migrate deploy

So the app never starts and it's rolled back.

This is the log from Fly

2023-03-13T17:20:18.816 app[ff091a8a] gdl [info] Preparing to run: `sh start.sh` as root

2023-03-13T17:20:18.833 app[ff091a8a] gdl [info] 2023/03/13 17:20:18 listening on [fdaa:1:9e30:a7b:d8:5:ac23:2]:22 (DNS: [fdaa::3]:53)

2023-03-13T17:20:18.837 app[ff091a8a] gdl [info] + npx prisma migrate deploy

2023-03-13T17:20:22.736 app[ff091a8a] gdl [info] npm WARN exec The following package was not found and will be installed: [email protected]

2023-03-13T17:20:26.836 health[ff091a8a] gdl [error] Health check on port 8080 has failed. Your app is not responding properly. Services exposed on ports [80, 443] will have intermittent failures until the health check passes.

2023-03-13T17:20:27.400 app[ff091a8a] gdl [info] [ 8.678859] Out of memory: Killed process 604 (node) total-vm:11173416kB, anon-rss:124428kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:1444kB oom_score_adj:0

[JavoByte]

By the way, this seems to be a duplicate of #156 but that one is closed since the author opted to stopped using Fly, so probably that one is untracked.

[seashairo]

I'm also experiencing this. It's not ideal.

I was able to deploy reliably after the first revision by moving prisma from devDependencies into dependencies.

[silouanwright]

This just happened to me on a fresh newly created app (today). I can confirm that the above comment from @seashairo helped me get it working.

[Apsysikal]

There is a related issue on the fly.io community: Prisma + SQLite causes an out of memory error on deploy which references the following fly.io docs page Optimizing your deployment.

As far as I understand if you turn on swap mode before running migrations, it will use disk space instead of running out of memory. Removing the swap directory and disabling swap after running migrations also seems reasonable to me.

The modified start.sh file would then look somewhat like this.

#!/bin/sh

# This file is how Fly starts the server (configured in fly.toml). Before starting
# the server though, we need to run any prisma migrations that haven't yet been
# run, which is why this file exists in the first place.
# Learn more: https://community.fly.io/t/sqlite-not-getting-setup-properly/4386

set -ex

# Setup 512MB of space for swap and set permissions and turn on swapmode
fallocate -l 512M /swapfile
chmod 0600 /swapfile
mkswap /swapfile
echo 10 > /proc/sys/vm/swappiness
swapon /swapfile

# Run migrations
npx prisma migrate deploy

# Turn off swap mode and remove swap directory
swapoff /swapfile
rm /swapfile

# Finally start the app
npm run start

Can anyone try this out?

[mcansh]

nice find @Apsysikal, if this does fix the problem, more than happy to accept a PR 🙂

[hilja]

@Apsysikal I tested the swap and it worked great. It was on a postgress instance tho.

[Apsysikal]

@hilja Nice 👌 I won’t have time to test it myself for the next few weeks with Sqlite but once I have, I’ll create a pull request.

[RAGrundy]

I was having this issue with a completely fresh npx create-remix@latest using Sqlite and it fixed it for me :)

2023-03-31T16:35:54.755 app[9185900a4e3978] lhr [info] + npx prisma migrate deploy

2023-03-31T16:35:56.832 app[9185900a4e3978] lhr [info] npm WARN exec The following package was not found and will be installed: [email protected]

2023-03-31T16:36:00.514 app[9185900a4e3978] lhr [info] [ 5.916011] Out of memory: Killed process 597 (node) total-vm:11112664kB, anon-rss:108160kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:1160kB oom_score_adj:0

2023-03-31T16:36:00.543 app[9185900a4e3978] lhr [info] Killed

That was the error before and this was the result after:

2023-03-31T16:43:09.903 app[9185900a4e3978] lhr [info] Preparing to run: `./start.sh` as root

2023-03-31T16:43:09.939 app[9185900a4e3978] lhr [info] 2023/03/31 16:43:09 listening on [fdaa:1:cc1d:a7b:be65:789f:2041:2]:22 (DNS: [fdaa::3]:53)

2023-03-31T16:43:09.939 app[9185900a4e3978] lhr [info] + fallocate -l 512M /swapfile

2023-03-31T16:43:09.947 app[9185900a4e3978] lhr [info] + chmod 0600 /swapfile

2023-03-31T16:43:09.949 app[9185900a4e3978] lhr [info] + mkswap /swapfile

2023-03-31T16:43:09.977 app[9185900a4e3978] lhr [info] Setting up swapspace version 1, size = 512 MiB (536866816 bytes)

2023-03-31T16:43:09.977 app[9185900a4e3978] lhr [info] no label, UUID=d6e6b09e-ef48-4580-b278-cd989098c8e2

2023-03-31T16:43:09.980 app[9185900a4e3978] lhr [info] + echo 10

2023-03-31T16:43:09.980 app[9185900a4e3978] lhr [info] + swapon /swapfile

2023-03-31T16:43:09.992 app[9185900a4e3978] lhr [info] + npx prisma migrate deploy

2023-03-31T16:43:12.437 app[9185900a4e3978] lhr [info] npm WARN exec The following package was not found and will be installed: [email protected]

2023-03-31T16:43:18.141 app[9185900a4e3978] lhr [info] Prisma schema loaded from prisma/schema.prisma

2023-03-31T16:43:18.144 app[9185900a4e3978] lhr [info] Datasource "db": SQLite database "sqlite.db" at "file:/data/sqlite.db"

2023-03-31T16:43:18.281 app[9185900a4e3978] lhr [info] 1 migration found in prisma/migrations

2023-03-31T16:43:18.290 app[9185900a4e3978] lhr [info] No pending migrations to apply.

2023-03-31T16:43:18.399 app[9185900a4e3978] lhr [info] npm notice

2023-03-31T16:43:18.399 app[9185900a4e3978] lhr [info] npm notice New major version of npm available! 8.19.4 -> 9.6.3

2023-03-31T16:43:18.400 app[9185900a4e3978] lhr [info] npm notice Changelog: <https://github.com/npm/cli/releases/tag/v9.6.3>

2023-03-31T16:43:18.400 app[9185900a4e3978] lhr [info] npm notice Run `npm install -g [email protected]` to update!

2023-03-31T16:43:18.400 app[9185900a4e3978] lhr [info] npm notice

2023-03-31T16:43:18.440 app[9185900a4e3978] lhr [info] + swapoff /swapfile

2023-03-31T16:43:18.473 app[9185900a4e3978] lhr [info] + rm /swapfile

2023-03-31T16:43:18.480 app[9185900a4e3978] lhr [info] + npm run start

2023-03-31T16:43:19.008 app[9185900a4e3978] lhr [info] > start

2023-03-31T16:43:19.008 app[9185900a4e3978] lhr [info] > remix-serve build

2023-03-31T16:43:19.514 app[9185900a4e3978] lhr [info] Remix App Server started at http://localhost:8080 (http://172.19.133.250:8080)

2023-03-31T16:43:21.687 health[9185900a4e3978] lhr [info] Health check on port 8080 is now passing.

2023-03-31T16:43:23.103 app[9185900a4e3978] lhr [info] HEAD / 200 - - 29.662 ms

2023-03-31T16:43:23.125 app[9185900a4e3978] lhr [info] GET /healthcheck 200 - - 72.490 ms

[korzewski]

@Apsysikal works like a charm! Thanks a lot 🚀

I'm using Indie stack with LiteSQL

[MichaelDeBoey]

This will be fixed by #232

[tdraper-dev]

@Apsysikal -- you're my hero, thank you 🙏

[pdandradeb]

I was still getting this error using 512 for swap. Changing to 2048 resolved it.

fly.toml:

swap_size_mb = 2048