feat: option to include deleted drafts while fetching unpublished entities [FC-0062]

[navinkarkera]

Related PRs and issues:

• fix: delete discarded components from meilisearch index [FC-0062] edx-platform#35243
• fix: discard changes [FC-0062] open-craft/frontend-app-authoring#58 fix: discard button [FC-0062] frontend-app-authoring#1214
• Discard changes on Library Info Sidebar breaks the library frontend-app-authoring#1200

[openedx-webhooks]

Thanks for the pull request, @navinkarkera!

What's next?

Please work through the following steps to get your changes ready for engineering review:

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

• If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
  • This process (including the steps you'll need to take) is documented here.
• If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

• Dependencies

  This PR must be merged before / after / at the same time as ...

• Blockers

  This PR is waiting for OEP-1234 to be accepted.

• Timeline information

  This PR must be merged by XX date because ...

• Partner information

  This is for a course on edx.org.

• Supporting documentation
• Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

🔘 Let us know that your PR is ready for review:

Who will review my changes?

This repository is currently maintained by @axim-engineering. Tag them in a comment and let them know that your changes are ready for review.

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

• Overview of Review Process for Community Contributions
• Pull Request Status Guide
• Making changes to your pull request

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

• The size and impact of the changes that it introduces
• The need for product review
• Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

[ormsbee]

The mechanism that Learning Core uses to "delete" published things is to:

1. Set the Draft version to be null (removes it from Studio, but not the published LMS version).
2. The act of publishing then sets the Published version of that PublisahbleEntity to also be null, which makes it disappear from the LMS.

Wouldn't excluding soft-deleted drafts from the publish process break that?

[navinkarkera]

@ormsbee Ahh yes, that change is not really required. Thank you for pointing it out, I missed to remove it and test once I got around fixing the meilisearch caching. Updated!

[ChrisChV]

@navinkarkera Looks good 👍

• I tested this: I followed the testing instructions on fix: discard changes [FC-0062] open-craft/frontend-app-authoring#58 (review)
• I read through the code and considered the security, stability and performance implications of the changes.

[bradenmacdonald]

@ormsbee Are you OK with this PR now?

[ormsbee]

Please add a docstring explaining this behavior (that it omits the entities that were soft-deleted), or make it a keyword argument. Otherwise, I'm fine with this.

Question that this brought to mind (doesn't have to be part of this PR): Does publish_all_drafts have a bug where it's re-publishing soft-deletes over and over again because NULL != NULL in SQL?

[navinkarkera]

Please add a docstring explaining this behavior (that it omits the entities that were soft-deleted), or make it a keyword argument. Otherwise, I'm fine with this.

Done.

Question that this brought to mind (doesn't have to be part of this PR): Does publish_all_drafts have a bug where it's re-publishing soft-deletes over and over again because NULL != NULL in SQL?

Yes, it does re-publish it over and over again. We probably need to use a specific value in version field like -1 for soft-deletes.

[ormsbee]

I think we can stop the duplicate publishing by excluding Draft versions that are null where the matching Published version is also null, without resorting to special version field values. I'll make a PR for it.

[ormsbee]

This will erroneously return soft-deleted entries that have already been published, because of the NULL comparisons. I think that if we're making this flag, it needs to be something like:

entities_qs = (
    PublishableEntity.objects
    .filter(learning_package_id=learning_package_id)
    .exclude(draft__version=F('published__version'))
)

if include_deleted_drafts:
    # This means that we should also return PublishableEntities where the draft
    # has been soft-deleted, but that deletion has not been published yet. Just
    # excluding records where the Draft and Published versions don't match won't
    # be enough here, because that will return soft-deletes that have already
    # been published (since NULL != NULL in SQL).
    # 
    # So we explicitly exclude already-published soft-deletes:
    return entities_qs.exclude(
        Q(draft__version__isnull=True) & Q(published__version__isnull=True)
    )

# Simple case: exclude all entities that have been soft-deleted.
return entities_qs.exclude(draft__version__isnull=True)

(I had to deal with this in #211, which I'd love to get a review of if you have a moment.)

[navinkarkera]

@ormsbee Makes sense. Updated and included a test inspired by your PR: #211

[ormsbee]

@navinkarkera: Are you okay with me squashing and merging this now?

[openedx-webhooks]

@navinkarkera 🎉 Your pull request was merged! Please take a moment to answer a two question survey so we can improve your experience in the future.