How to release Mongoose model from memory? (memory leak)

[fedu]

How to release the model from memory after accessing it? I'm using global.gc() with --expose-gc option to see that this is eating my memory.

The following code creates 10000 collections:

// mongoose connection
var db = mongoose.createConnection(...);

// amount of collections
var amount = 10000;

// create collections (100/per second)
var sync = async.queue(function(n, cb) {
    var schema = new mongoose.Schema({
        data: mongoose.Schema.Types.Mixed
    });
    var collection = 'model_'+n;
    var model = db.model(collection, schema);
    setTimeout(function() { cb(); }, 10);
}, 1);

// push to queue
for(var i=0; i<amount; i++) {
    sync.push(i);
}

// done
sync.drain = function(err) {
    console.log('all '+amount+' models done');
};

// garbage collector (every second)
setInterval(function() {
    try { global.gc(); } catch(gcerr) { }
}, 1000);

The memory usage is increasing as collections are created and the memory is never released:

1 - Memory used: 30 MB
2 - Memory used: 36 MB
3 - Memory used: 42 MB
4 - Memory used: 48 MB
5 - Memory used: 54 MB
6 - Memory used: 61 MB
7 - Memory used: 65 MB
8 - Memory used: 71 MB
9 - Memory used: 77 MB
10 - Memory used: 82 MB
all 10000 models done
11 - Memory used: 86 MB
12 - Memory used: 86 MB

Any ideas how to purge the model from the memory manually without closing the connection?

[vkarpov15]

Connections keep track of their associated models in the models map, try doing delete db.models[collection] and the gc should in theory be able to clean up the model instance.

[fedu]

Deleting models seems to free about half of the eaten memory, with all amounts tried.

100k without

24 - Memory used: 651 MB
all 100000 models done
25 - Memory used: 658 MB

100k with delete db.models[collection]:

24 - Memory used: 324 MB
all 100000 models done
25 - Memory used: 328 MB

Any ideas how to free it completely?

[fedu]

I think some model.purge() or model.close() would be a very good feature for handling thousands of user data collections with Mongoose. Now it's impossible?

[vkarpov15]

There's no good way to do that AFAIK but as far as freeing up memory goes I think delete db.models[collection] should be sufficient. I'll dig in to the memory usage and see why there's still memory left over. In the meantime can you please confirm which version of mongoose?

[fedu]

Mongoose v. 4.0.1

[fedu]

These lines did the trick and got all my memory back:

delete db.models[collection];
delete db.collections[collection];
delete db.base.modelSchemas[collection];

[vkarpov15]

I see, so that's all the places where it's tracking the collection data. Thanks for investigating, very useful information :)

[StuartHickey]

I have a very similar issue with trying to retrieve large data sets and the memory growing rapidly on each retrieval from the client.

Code in my controller is as follows:

'use strict';
var mongoose = require('mongoose');
var User = mongoose.model('User');

exports.getUser = function *() {
var sort = {'x': 1};
var user1 = yield User.find({ userid: this.request.body.userid1 }, {'_id': 0,'x':1, 'y':1, 'date':1, 'station': 1}).sort(sort).exec();
if (!user1) {
user1 = new User();
}
var user2 = yield User.find({ userid: this.request.body.userid2 }, {'_id': 0,'x':1, 'y':1, 'date':1, 'station': 1}).sort(sort).exec();
if (!user2) {
user2 = new User();
}
var user3 = yield User.find({ userid: this.request.body.userid3 }, {'_id': 0,'x':1, 'y':1, 'date':1, 'station': 1}).sort(sort).exec();
if (!user3) {
user3 = new User();
}
var user4 = yield User.find({ userid: this.request.body.userid4 }, {'_id': 0,'x':1, 'y':1, 'date':1, 'station': 1}).sort(sort).exec();
if (!user4) {
user4 = new User();
}
this.body = { user1: user1, user2: user2, user3: user3, user4: user4 };
};

How can I implement something like above:
delete db.models[collection];
delete db.collections[collection];
delete db.base.modelSchemas[collection];

when I try and do I get the following error
delete mongoose.collections[User];
^
TypeError: Cannot convert undefined or null to object

and where, to free up memory once the data have been retrieved. I only care about retrieving the data, not saving it again afterwards.

Thanks for any assistance.

[vkarpov15]

Try:

delete mongoose.models['User'];
delete mongoose.connection.collections['users'];
delete mongoose.modelSchemas['User'];

[illarion]

The other solution that works perfectly for me is to not use global moongoose, instantiated with
var moongoose = require('moongoose');

use the following instead:

var Mongoose = require('mongoose/lib').Mongoose;
....
new Moongoose().createConnection(...)

so that you can manage references to Moongoose() instances just like any other objects and it will be garbage collected when needed.

[vkarpov15]

Good suggestion @illarion 👍

[manu354]

@illarion @vkarpov15 That seems like a really good solution to the same problem I have as OP.

Two questions:

1. If I have two instances which both connect to the same mongodb database do I still have to create two separate connections using this method?

2. How would I go about "managing references to Mongoose() instances"? Any tutorial out there or small write up?

Thanks heaps!

[vkarpov15]

1. Depends on whether you need to garbage collect your models. Do you need this?

2. Not that I know of but there isn't much to manage, const m = new mongoose.Mongoose(); then m will get garbage collected when it goes out of scope.

[QuotableWater7]

Leaving this snippet here in case anyone else finds it useful. It's mostly just a variation of what's already been discussed.

We create our models directly on connection objects, and our Jest tests were quickly chewing through available memory. We were able to programmatically remove references to all models/connections with this code in an afterAll hook:

mongoose.connections.forEach(connection => {
  const modelNames = Object.keys(connection.models)

  modelNames.forEach(modelName => {
    delete connection.models[modelName]
  })

  const collectionNames = Object.keys(connection.collections)
  collectionNames.forEach(collectionName => {
    delete connection.collections[collectionName]
  })
})

const modelSchemaNames = Object.keys(mongoose.modelSchemas)
modelSchemaNames.forEach(modelSchemaName => {
  delete mongoose.modelSchemas[modelSchemaName]
})

[sibelius]

@QuotableWater7 do you have a repo example of this with jest?

@vkarpov15 should mongoose provide a helper to cleanup models from memory?

any other place where memory can leak from mongoose?

related to jestjs/jest#6787

[vkarpov15]

Opened up a separate issue to track ☝️

[Lwdthe1]

Can someone please elaborate on @illarion suggestion. My current code that is causing leaks:

// -- userDbSchema.js

const mongoose = require('mongoose')
var Schema = mongoose.Schema;

var Schema_User = new Schema({
    // schema definition ...
})

Schema_User.statics.getById = function(id) {
 // ... 
}

module.exports = mongoose.model('User', Schema_User);




// -- userOperation.js

let userDbSchema = require('userDbSchema.js')

class UserOperation {
  constructor() {
     // ...
  }

  getUserById(id) {
    return userDbSchema.getById(id)
       .then((dbUser) => {
            // do some processing
            return dbUser
       })
  }
}

module.exports = UserOperation

I have 64 models in my system defined on the mongoose instance. I only want one connection to the database. In that suggestion, wouldn't I be creating a new connection for every API request that comes into my server? That doesn't sound right. What I've been doing is using a single call to mongoose.model to get each model in my system. Then I hold on each model in the house namespace and never get rid of until my server restarts. I'm guessing this is the cause of my massive memory leak. I need a way to get a new instance of a model for every API request and fetch or write the data I need from the database, then get rid of the model. I would appreciate some guidance because this leak has become unsustainably expensive monetarily.

Would switching

module.exports = mongoose.model('User', Schema_User);
to
module.exports = () => mongoose.model('User', Schema_User);

help by creating a new instance of the model every time another file does require('userDbSchema.js') which would be garbage collectible?

[Lwdthe1]

I tried doing this after every request, and it slowed down my server requests considerably:

delete mongoose.models['User'];
delete mongoose.connection.collections['users'];
delete mongoose.modelSchemas['User'];

I guessing it's because the model has to be created before every request.

It'd be great if there was a definitive tutorial on how to build an web app using mongoose without causing the memory leaks that have been shared here.

[sibelius]

This is only needed for tests

You should create a single mongoose connection per server and you would be good to go

[Lwdthe1]

Hmm, if it's a problem for tests, then it must also be a problem for the thing you're testing, the server, no?

[Lwdthe1]

I found the leak. I'm happy to report that it is not mongoose. Thanks for the help everyone who contributed to this thread.

[vkarpov15]

@Lwdthe1 because it isn't a leak, you typically don't delete a model once you've created it. Creating a new model using mongoose.model() on every request is wrong. It also isn't a problem if you don't create a new model in every test, which is generally a bad idea.

[chuan-qin]

#2874 (comment)
@Lwdthe1, So, what is the non-mongoose leak? (I may be facing a similar issue...)

[weichtg3]

I know this is a Closed issue, but I would like to add what we discovered recently.

BLUF: if you are repetitively building a model using model.discriminator(), the schema is getting exponentially bigger due to schema._originalSchema = schema.clone().

Against recommendations, we are doing open/close with each request instead of caching connections. This is because our multi-tenancy adds an additional level of security in that each tenant gets their own database with their own set of user credentials. Mongoose useDb() did not seem to support that, nor any other mod/hack I could find on Google. (Note: I have since modified a library called mobgoose to cache down to that level and may use it going forward).

We initially tried deleting everything under the sun as suggested above to no avail. If you use connection.model(), we could get things to clean themselves up. However, we were using model.discriminator(). Looking inside the mongoose library code and seeing how the Model.discriminator() and Schema prototypes work, we see a lot of clone() calls. The exponential memory leak seems to come from schema._originalSchema = schema.clone().

Our current solution, if we are to continue with open/close per request, is to re-generate the Schema on each call too. Mucking around inside the Schema to delete individual properties seemed too risky, and this gets us past the current issue while we explore a better solution going forward.

[vkarpov15]

@weichtg3 that's an interesting use case, thanks for elaborating. We'll look into whether we can get rid of _originalSchema.

[snowshoes]

There is no mongoose.modelSchemas neither in 2019 nor in 2020

[vkarpov15]

@snowshoes just use mongoose.deleteModel(). That handles the task of removing a model from memory for you.