Django 4.2 Upgrade

[UsamaSadiq]

Abstract

Django 4.2, the latest LTS version for Django has been released for the community. To keep our codebase updated, we need to upgrade Django version in all IDAs and packages in the edx and openedx organisations.

Context & Background

According to our Support Windows Spreadsheet, we should upgrade Django to latest LTS release in the next planned release so community can also upgrade their projects to latest tech-stack.

Scope & Approach

The initial approach for this issue is as following (It'll be updated later on):

• arbi-bom team will be the one responsible to manage this upgrade among all services and provide any help to the teams if needed or requested by the other teams.
• All the private and public repos (containing the usage of Django) under edx and openedx organisations will be updated.
• All the owning teams will be approached to have their preferred method to upgrade the repositories under their ownership and collaborate to complete the task before the supposed deadline.

Value & Impact

Updating the Django version to the latest LTS version Django 4.2 will provide long term security upgrades assurance to the complete codebase and the community which is using the openedX codebase as underlying base for their projects.

Milestones and/or Epics

Following will be the major milestones for this upgrade process (can be improved later on):

Milestones

Preview Give feedback

1. Create a roadmap issue in the platform-roadmap repository
2. Add this checklist to the roadmap issue's description
3. Add the "maintenance" label to the roadmap issue
4. Add the roadmap issue to the Backlog column of the Open edX Roadmap project
5. Set appropriate values for the roadmap project's custom fields for the issue (especially "Proposed by", "Platform map - Super Level", "Strategy", and "Type")
6. Set an appropriate release milestone for the roadmap issue
7. Add an explicit target completion date to the roadmap issue description, and explain there why it was chosen
8. Select an orchestration team
9. Name the orchestration team in the roadmap issue description
10. Create a repo health check to identify most/all of the repositories impacted by the upgrade (and ideally, whether or not the upgrade is believed to be complete)
11. Create a new value for the "Project" field in the Maintenance project board for this upgrade project
12. Create a new view in the Maintenance project board that filters down to only the issues in this upgrade project
13. Create an issue in each impacted repository for the upgrade, and add it to the Todo column of the Maintenance project board; specify at least the "Project" and "Source" field for each issue (and "Owner" also if you're a 2U or Arbisoft employee)
14. Create a task list in the roadmap issue listing all of the impacted repository issues
15. Create a page under Upgrades in Confluence for documentation related to the upgrade
16. Add a link to the Confluence page from the roadmap issue
17. Add a link to the roadmap issue from the Confluence page
18. Document in a Confluence child page the changes believed to be most problematic and/or interesting about the upgrade
19. Create a ticket to determine the appropriate amount of automation (codemods, repo health checks, etc.) to create for the upgrade
20. Perform the automation discovery work and write upgrade instructions for all project participants in a Confluence child page
21. Link to the upgrade instructions from the roadmap issue
22. Send an announcement of the upgrade, asking code maintainers to read the upgrade instructions and select an upgrade service level for each impacted repository in its corresponding issue
23. Create issues in public-engineering for each external dependency which still needs code changes and/or a release to support the upgrade
24. Add the "help wanted" and "maintenance" labels to each public-engineering issue created above
25. Add each public-engineering issue created above to the Maintenance project board
26. Set appropriate values for the Maintenance project board custom fields for each added public-engineering issue
27. Ask the Open edX developer community (especially Core Committers) for assistance with the added public-engineering issues
28. Create Build-Test-Release Working Group and/or 2U SRE tickets if they will need to do work for the upgrade
29. Complete and update all of the created implementation tickets
30. Deploy all of the updated services
31. Update the support windows spreadsheet to correctly indicate the version now used
32. Announce the successful completion of the upgrade
33. Make sure any deployment instructions make it into the release notes of the next Open edX named release (collaborate with the BTR WG)
34. Remove any CI matrix entries for no-longer-supported versions of the dependency
35. Remove the Maintenance project board view/tab created for the upgrade
36. Stop running repo health checks related to the upgrade which are no longer relevant
37. Schedule and run a retrospective meeting about the upgrade
38. Update the Upgrade Project Runbook based on retrospective findings, if appropriate
39. Mark the roadmap issue for the upgrade as complete
40. Move the root Confluence page for this upgrade’s docs under Past Upgrades
41. Ticket work to replace or take over dependencies which were found during the upgrade to be inadequately maintained

Named Release

Quince

Timeline

Due Date: Sep 30th, 2023 - The due date is before the deadline to cut branches for the next target release.

Target Release: Quince

Proposed By

2U

Orchestration Team

Arbi-BOM

Additional Info

No response

[github-actions]

Thanks for your submission, @openedx/open-edx-project-managers will review shortly.

[ormsbee]

@UsamaSadiq: If I'm working on a fairly new repo and I want to be testing on both Django versions from the start, what's a good repo to look at for tox.ini / workflow file examples?

[UsamaSadiq]

@ormsbee: We've already added support to run checks with multiple Django versions in some repos. For reference, you can have a look at the https://github.com/openedx/XBlock repo.

[ormsbee]

@UsamaSadiq: That's perfect, thank you.

[UsamaSadiq]

Created new project field Django 4.2 Upgrade in the Maintenance Project and a new view Django 4.2 to track the issues related to Django 4.2 upgrade.

[UsamaSadiq]

Added all upgrades issues on the maintenance board. The new Django 42 view now tracks all the issues.

[ormsbee]

@UsamaSadiq, @jmbowman: Is there an updated estimate for when 4.2 support will land for edx-platform? I'm developing a feature that would be configured differently between 3.2 and 4.2. I'm going under the assumption that I'll have to support both, but I won't bother if 4.2 is landing in edx-platform in the next couple of weeks.

[UsamaSadiq]

@UsamaSadiq, @jmbowman: Is there an updated estimate for when 4.2 support will land for edx-platform? I'm developing a feature that would be configured differently between 3.2 and 4.2. I'm going under the assumption that I'll have to support both, but I won't bother if 4.2 is landing in edx-platform in the next couple of weeks.

So far, we are still stuck on successfully doing the MySQL 8.0 upgrade which is a major blocker for the Django 4.2 upgrade. Currently, we can not give a potential deadline for this blocker to be resolved since we (including SRE team) have already made two attempts (blue/green deployment, data migration to replica db) in MySQL 8.0 upgrade but those failed due to the large table size we need to migrate.
Hence it'd be better if you can provide support for both Django versions for now.

[feanil]

@UsamaSadiq sholud this ticket still be open? It seems like it could be closed.

[feanil]

A quick grep through the django versions in the openedx org does not show any major services that are still running older versions of PyMongo. Please re-open if you you find something I missed.