How to Speed Up Nautobot Job for Circuit Insertion (100K+ Records)?

[Dharshini1810]

Hi Nautobot Community,

I am trying to populate circuits in Nautobot using a job, but I am dealing with more than 100,000 records, which takes more than 24 hours to insert. Eventually, I get a time limit exceeded error.

My setup:

Celery is executing on a high-end server as a docker container
Single job inserting 100K+ circuits is too slow
Using Django ORM for insertion
No multithreading process, a single circuit is fetched and circuit, circuit Termination is created and cables has been created on both termination to its respective device

What I have tried:

Increasing the job timeout to 20 hours in celery, but my process has been running more than 20 hours

Issue:

Despite these optimizations, inserting circuits still takes too long. What’s the best approach to scale this process?
Is there a better way to execute bulk jobs in Nautobot for large datasets?
Any recommendations on job parallelization in Nautobot?
Thanks in advance! 🚀