ByteTrack Deployment on Deepstream 6.0+ Major Memory Leak

[EmpireofKings]

Hello,

I am currently testing the deployment guide of bytetrack in deepstream on a jetson AGX xavier with 8 camera feeds, and it works great apart from the major memory leak. The program uses the entire device memory within 35 minutes of running. If i switch back to nvidia's default trackers no leak occurs.

I have gone through the numerous issues in this repo that attempt to fix the memory leak, such as:

#253

#252

#249

https://github.com/ifzhang/ByteTrack/pull/158/files

Unfortunately none of the above 'solutions' fix the memory leak. I have ran the deepstream pipeline with valgrind and can verify a leak is occurring, but valgrind does not show exactly where the leak is taking place, it just shows that a pointer is continuously being allocated somewhere in the heap.

I am actively trying to fix this issue, but since i have relatively no experience with the mechanics of bytetrack, i can only do so much besides poking and proding around the codebase until a solution presents itself to me.

Im hoping @ifzhang, @chirag4798 or @callmesora might be able to reply to this issue with some assistance and or insight, otherwise if you are reading this you should just treat this issue as a warning to not deploy or use the deepstream version of bytetrack until this issue has been marked resolved by me or one of the above authors.

[Ritesh1991]

have you found the solution of this issue ?

[callmesora]

Sorry I have been busy lately and could not take a look at this.
@chirag4798 is your friend in this situation :D !

Best of luck

[Ritesh1991]

Thanx for Reply,In this repo it hasn't been committed yet, So I need to use chirag's repo to resolve this issue.

[kongjibai]

@Ritesh1991 @EmpireofKings, Refer this for the fix - #286

@chirag4798
Thanks for your efforts for this bug, but after I modified my code as yours, it runs error as follow, while the not modified version code runs normally. Any advice?
ERRROR from element tracker: Failed to submit input to tracker. Error details: gstnvtracker.cpp(610): gst_nv_tracker_submit_input_buffer(): /GstPipeline:test-pipeline/GstNvTracker:tracker

[indraksha]

@Ritesh1991 @EmpireofKings, Refer this for the fix - #286

Is anyone able to run it in deepstream after this fix , i am also getting the issue:

ERROR from element tracker: Failed to submit input to tracker. Error details: gstnvtracker.cpp(610): gst_nv_tracker_submit_input_buffer(): /GstPipeline:test-pipeline/GstNvTracker:tracker

[huihui308]

I have solved byteTracker memory leak on deepstream-6.0

[Ritesh1991]

@huihui308 can you please provide link for it.

[callmesora]

Hey guys, I don't have a DeepStream compatible PC/device to test this anymore but here's some hints that might help.

It looks like the memory leak may be caused by the creation of new objects in the processFrame() function. Specifically, in the innermost loop, we are creating new NvMOTTrackedObj objects with the new keyword and storing them in an array, but we are not deleting them anywhere. This means that the memory allocated for these objects will not be freed, resulting in a leak. Additionally, you are creating a new array of NvMOTTrackedObj objects each time the function is called, but you are not freeing any previous instances of the array.

You can try to fix this by using std::vector instead of a raw array and new keyword, so that it will be freed automatically.

Also, you are using std::make_shared to create a new shared_ptr object of BYTETracker, but you are not deleting it anywhere, which can also cause a memory leak.

You can try to fix this by using std::unique_ptr instead of std::shared_ptr.

It's also worth to note that the map byteTrackerMap is not being cleared when the stream is closed, so you might want to clear the map when the stream is closed.

[callmesora]

In code (got a little help from ChatGPT but I have no means of testing it so please do )

#include "Tracker.h"
#include "BYTETracker.h"
#include <fstream>
#include <unordered_map>

class NvMOTContext {
public:
    NvMOTContext(const NvMOTConfig &configIn, NvMOTConfigResponse &configResponse);
    NvMOTStatus processFrame(const NvMOTProcessParams *params, NvMOTTrackedObjBatch *pTrackedObjectsBatch);
    NvMOTStatus processFramePast(const NvMOTProcessParams *params, NvDsPastFrameObjBatch *pPastFrameObjectsBatch);
    NvMOTStatus removeStream(const NvMOTStreamId streamIdMask);

private:
    std::unordered_map<uint64_t,std::unique_ptr<BYTETracker>> byteTrackerMap;
};

NvMOTContext::NvMOTContext(const NvMOTConfig &configIn, NvMOTConfigResponse &configResponse) {
    configResponse.summaryStatus = NvMOTConfigStatus_OK;
}

NvMOTStatus NvMOTContext::processFrame(const NvMOTProcessParams *params, NvMOTTrackedObjBatch *pTrackedObjectsBatch) {
    for (uint streamIdx = 0; streamIdx < pTrackedObjectsBatch->numFilled; streamIdx++){
        NvMOTTrackedObjList   *trackedObjList = &pTrackedObjectsBatch->list[streamIdx];
        NvMOTFrame            *frame          = &params->frameList[streamIdx];
        std::vector<NvObject> nvObjects(frame->objectsIn.numFilled);
        for (uint32_t numObjects = 0; numObjects < frame->objectsIn.numFilled; numObjects++) {
            NvMOTObjToTrack *objectToTrack = &frame->objectsIn.list[numObjects];
            NvObject nvObject;
            nvObject.prob    = objectToTrack->confidence;
            nvObject.label   = objectToTrack->classId;
            nvObject.rect[0] = objectToTrack->bbox.x
            nvObject.rect[1] = objectToTrack->bbox.y;
            nvObject.rect[2] = objectToTrack->bbox.width;
            nvObject.rect[3] = objectToTrack->bbox.height;
            nvObject.associatedObjectIn = objectToTrack;
            nvObjects.push_back(nvObject);
        }

        if (byteTrackerMap.find(frame->streamID) == byteTrackerMap.end())
            byteTrackerMap.insert(std::pair<uint64_t, std::unique_ptr<BYTETracker>>(frame->streamID, std::make_unique<BYTETracker>(15, 30)));

        std::vector<STrack> outputTracks = byteTrackerMap.at(frame->streamID)->update(nvObjects);

        std::vector<NvMOTTrackedObj> trackedObjs;
        trackedObjs.reserve(512);

        for (STrack &sTrack: outputTracks) {
            std::vector<float> tlwh        = sTrack.original_tlwh;
            NvMOTRect          motRect{tlwh[0], tlwh[1], tlwh[2], tlwh[3]};
            NvMOTTrackedObj    trackedObj;
            trackedObj.classId                        = 0;
            trackedObj.trackingId                     = (uint64_t) sTrack.track_id;
            trackedObj.bbox                           = motRect;
            trackedObj.confidence                     = 1;
            trackedObj.age                            = (uint32_t) sTrack.tracklet_len;
            trackedObj.associatedObjectIn             = sTrack.associatedObjectIn;
            trackedObj.associatedObjectIn->doTracking = true;
            trackedObjs.push_back(trackedObj);
        }

        trackedObjList->streamID     = frame->streamID;
        trackedObjList->frameNum     = frame->frameNum;
        trackedObjList->valid        = true;
        trackedObjList->list         = trackedObjs.data();
        trackedObjList->numFilled    = trackedObjs.size();
        trackedObjList->numAllocated = trackedObjs.size();
    }
    return NvMOTStatus_OK;
}

NvMOTStatus NvMOTContext::processFramePast(const NvMOTProcessParams *params,
                                           NvDsPastFrameObjBatch *pPastFrameObjectsBatch) {
    return NvMOTStatus_OK;
}

NvMOTStatus NvMOTContext::removeStream(const NvMOTStreamId streamIdMask) {
    byteTrackerMap.erase(streamIdMask);
    return NvMOTStatus_OK;
}

In this implementation, I've replaced the use of raw arrays and the new keyword with std::vector so that the memory is automatically managed. Also, I've replaced the use of std::shared_ptr with std::unique_ptr, which will automatically manage the memory of the BYTETracker objects created.

Also, I've added a removeStream function that erases the streamId of the closed stream from the byteTrackerMap so that it won't cause memory leak in the future.

[sunxirui310]

std::vector<NvMOTTrackedObj> trackedObjs;
seems calls release error

[Leon-Cheung-CQ]

Anyone solve this efficiently?