Workbook creation performance for large courses

[jmerrill86]

Greetings!

I've been chasing down a performance issue related to the report processing of an assignment with over 12,000 students/submissions. The query that pulls the data related to the course module is performs very well and returns data needed for the report in less than five seconds which is outstanding. Once the process has the data, the Excel document building takes just over 18 minutes to build the document for download.

Since the function report_componentgrades_add_data is mostly calling the MoodleExcelWorksheet class to build the document, I'm not sure there is any changes that could be made to the plugin itself to help processing the document quicker.

I'm curious about a level of effort for adding a CSV export for the report and being able to toggle the format of the document in the administration settings. I also had the thought of adding a scheduled task that built the report ahead of time once a deadline for the module was closed.

I'm attaching an XHProf screen grab of the top function calls while running that report.

Please let me know if you have any questions or I can provide more examples or clarify anything.

Thanks,

Justin

[marcusgreen]

"The query that pulls the data related to the course module is performs very well and returns data needed for the report in less than five seconds which is outstanding. "

At the moment it breaks if users can have multiple attempts so if that ever gets fixed it would be good to run that benchmark to check it is still acceptable.

"Excel document building takes just over 18 minutes to build the document for download"..
That doesn't surprise me, however I don't have access that sort of data to check against. If I do some more development it would be very helpful if you would test early code to see that performance does not get degraded further.

"I'm curious about a level of effort for adding a CSV export for the report and being able to toggle the format of the document in the administration settings."

With the current architecture I think that would be difficult.

I have already started experimenting with re-building it with a better architecture. My criteria are that it would be possible to add unit tests for the various methods, be much easier to maintain and at least allow for the possibility of exporting in different formats. But as the time required is probably about 10 days or more and I have other stuff on at the moment, I don't have any roadmap.

Thanks for the feedback, it has at minimum nudged it up my priority list.

[marcusgreen]

Would you contact me at marcusavgreen at gmail.com to discuss something that might help with performance