Data Import - very very slow import - strange problmes

[mysugarape]

Hi,

Sorry, not sure where to post this, so hopefully, it’s okay here. It’s not exactly an issue—more like a scream for help. :)

After spending several weeks working with BerliCRM, things started making sense. Any issue that came up was resolved either independently or with the help of old forum threads (sometimes from 2015!). Overall, it fits all requirements perfectly.

However, a really strange problem occurred during data import. At this point, the situation is completely confusing, and honestly, there’s no clue where to even start.

All import files are correct, tested on a backup system. The setup includes one productive/live server and one test server. The live server runs on a VPS, while the test server is on a shared hosting plan. The test server holds a backup of the live server that’s less than five days old. No code changes have been made in the last few days, meaning both systems should be identical. MySQL & PHP requirements are also met on both.

The import files were carefully prepared—one for contacts (over 700 entries) and another for service contracts (over 5000 entries). A test with a single row worked perfectly.

Based on recommendations from old Vtiger posts, larger files were split into chunks of fewer than 500 entries. Importing to the live server then began. After pressing the import button, nothing seemed to happen at first, but checking phpMyAdmin revealed that the tables were being filled. So far, so good.

Here’s the issue: importing 400 contracts took 12 hours! That’s about 2 minutes per contract, which seems ridiculously slow. Still, after 12 hours, the process finished successfully, and all entries were correctly written.

Later, while testing something else, the import was accidentally started on the test server instead of the live one. The exact same file completed in just one minute. That’s when things started getting really confusing.

At first, the assumption was that the server was the culprit. A detailed comparison followed, adjusting some non-essential settings to ensure both environments were truly identical. The import was then repeated—same file, same setup—but the results remained unchanged: 1 minute on the test server, 12 hours on the live server.

A new plan was formed: create a fresh backup of the live system, restore it on the test server, import the necessary data there, then export and re-import it back into the live server—saving over 100 hours of import time. However, on this newly created backup, the same slowdown occurred. Why?

No major code changes happened recently. The systems are identical, yet something is clearly off when it comes to importing.

Now, with no clear direction, the next step is trying file comparison AI tools to spot potential differences. But has anyone encountered this issue before? Any ideas on where to start troubleshooting?

Cheers!

[urban-thinking]

Hiya @mysugarape ,

to get it right ... you could get the slow import on the test server as well, when you made a full clone of the live installation? How did you clone in the first place (when test import was not slow) and how did you clone the last time, when test was slow as well?

Are the DB settings 100% identical? The CRM is extremely sensitive to wrong DB settings :-/

Just some thoughts ... to be sure, both systems would have to be examined more closely. I will asks some colleagues as well, maybe they have more ideas.

Regards!

[mysugarape]

Hi,

Short answer: Yes. :)

Did some testing between the servers and backups—the error remains the same, regardless of which server is used. It must be something related to the files or the database. But honestly, that shouldn’t be the case. A file-by-file comparison of both backups (AI-driven) showed no differences at all. The only variations were in server settings. However, since the comparison tool was a 14-day trial version, there's no certainty about its accuracy.

After several hours of wasted time, a fresh installation was the next step. All necessary changes and customizations (around 10–15 files) were applied, and then the import was tested again.

And guess what? It worked perfectly! 🚀

All >5000 contracts were imported in just 5 minutes, along with customer data. The process was identical to before—some files were even copied over. No clue what went wrong earlier. 🤷‍♂️

For most coding-related issues, i like to deep-dive into errors/issues - is essential to learn from them. But in this case, it’s just too weird to figure out. If you’re interested, both versions are still running—no DSGVO-sensitive data included - you could test it by yourself -

New issue incoming... 😅

Now that everything is working, a new problem appeared: workflow emails aren't being sent. Previously (on the old live version), they worked just fine. Head-scratching moment.

Maybe you have an idea?

✅ Cron job runs every 15 minutes
✅ Emails are present in the database table [com_vtiger_workflowtask_queue]

Thanks for your help!

[mysugarape]

Hi,

EDIT 2: Workflow emails are now working, but some changes were necessary. This might be interesting or relevant for other users facing the same issue.

Cron job is set up and working correctly
Ubuntu 22 Server
Plesk

Issue:
Workflow emails were stuck in the queue (com_vtiger_workflowtask_queue).

Fix
File: vtigercron.php
To resolve the "access denied" error when calling the cron job, the following modification was made:

Issue:
Workflow emails get stuck in queue. table: com_vtiger_workflowtask_queue

File: vtigercron.php

to get rid of the access denied error when calling the cron. modified part:

if(PHP_SAPI === "fpm-fcgi" || empty($_SERVER['REMOTE_ADDR']) || (isset($_SESSION["authenticated_user_id"]) && isset($_SESSION["app_unique_key"]) && $_SESSION["app_unique_key"] == $application_unique_key)){

File: vtlib/Vtiger/Cron.php

` /**
* Check if task is right state for running.
*/
function isRunnable() {
return true;
$runnable = false;

    if (!$this->isDisabled()) {
        // Take care of last time (end - on success, start - if timedout)
        // Take care to start the cron im
        $lastTime = ($this->getLastStart() > 0) ? $this->getLastStart() : $this->getLastEnd();
        $elapsedTime = time() - $lastTime;
        $runnable = ($elapsedTime >= ($this->getFrequency()-60));
    }
    return $runnable;
}

/**`

File: cron/vtigercron.php

change to

export USE_PHP=/opt/plesk/php/7.4

after all this modifications, the workflow mails working

[mysugarape]

issue resolved, with modification