Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Quince testing] <add course mode from django admin>: <Getting 403 error while adding course mode from django admin> #325

Closed
fayyazahmed66 opened this issue Nov 6, 2023 · 23 comments
Labels
release blocker Blocks the upcoming release (fix needed)
Milestone

Comments

@fayyazahmed66
Copy link

Release

quince

Expected behavior

The user should be able to add course mode without any issues.

Actual behavior

I get 403 when I try to save course mode.

Steps to reproduce

  1. Go to https://quince.demo.edly.io/admin/course_modes/coursemode/add/
  2. Select any course and add its mode
  3. click on the save button

Additional information

Screenshot 2023-11-03 at 4 43 01 PM
@regisb
Copy link
Contributor

regisb commented Nov 6, 2023

It should be noted that the "csrftoken" appears to be correctly set (value = QJvNDZ9GQQFw3aydr6L4MLWpQPXJKzbD):

Screenshot from 2023-11-06 09-35-14

The "csrfmiddlewaretoken" request parameter is also included in the POST request, but it has a different value from the cookie (sdJux230jFcULjJES9oPtrYtDZSNwFPw8M470R2wZlHgEj7H95ZJ52KIjEFm64QZ):

image

Maybe this discrepancy is causing the 403 response?

In any case, the solution should NOT be to ignore CSRF parameters for the LMS host (via CSRF_TRUSTED_ORIGINS or CORS_ORIGIN_WHITELIST). CSRF checking is an important security feature and it should not be bypassed.

@iamsobanjaved
Copy link

Not familiar with the way the tutor is set up but this may help.
In Django 4.0, the Origin header is now checked and Django recommends adding the origin to the CSRF_TRUSTED_ORIGINS if the request is coming from a subdomain, more details here: https://docs.djangoproject.com/en/4.2/releases/4.0/#csrf-trusted-origins-changes

@regisb
Copy link
Contributor

regisb commented Nov 6, 2023

@iamsobanjaved the request is not coming from a subdomain. The request is coming from the same host as the LMS.

@iamsobanjaved
Copy link

Can we set DEBUG = True in this test instance temporarily to get the reason behind the error?

@regisb
Copy link
Contributor

regisb commented Nov 6, 2023

Sure. There you go:

image

@mariajgrimaldi
Copy link
Member

mariajgrimaldi commented Nov 6, 2023

I believe this is failing with any operation made from the Django Admin, here is an example of when I tried to edit a user:
image

@mariajgrimaldi mariajgrimaldi added this to the Quince.1 milestone Nov 6, 2023
@mariajgrimaldi
Copy link
Member

label: quince testing

@mariajgrimaldi mariajgrimaldi added release blocker Blocks the upcoming release (fix needed) and removed needs triage labels Nov 6, 2023
@regisb
Copy link
Contributor

regisb commented Nov 6, 2023

For the record, I am unable to reproduce the issue in development mode. (tutor dev start)

EDIT: I am also unable to reproduce the issue locally, with local.overhang.io as the LMS_HOST. I suspect the error may be due to HTTPS.

EDIT2: I can confirm that the issue is fixed in HTTP mode (ENABLE_HTTPS=false).

EDIT3: here is the error log from the lms container:

tutor_local-lms-1            | 2023-11-06 15:34:19,809 WARNING 7 [django.security.csrf] [user 6] [ip 77.195.23.18] log.py:241 - Forbidden (Origin checking failed - https://quince.demo.edly.io does not match any trusted origins.): /admin/auth/user/6/change/

@mariajgrimaldi
Copy link
Member

mariajgrimaldi commented Nov 6, 2023

I tried a different POST from the LMS and it's also failing:
image

But POST requests from Studio are OK:
image

@regisb
Copy link
Contributor

regisb commented Nov 6, 2023

I'll write here my investigations.

In django.middleware.csrf._origin_verified the test if request_origin == good_origin is failing because request.secure() is false, which is surprising. What is even more surprising is that request.scheme == "https". Yet, the implementation of is_secure is unambiguous: https://github.com/django/django/blob/main/django/http/request.py#L268

Turns out that the request.is_secure method is MANUALLY OVERRIDDEN by edx-platform (all caps to emphasise my growing surprise). Right here: https://github.com/openedx/edx-platform/blob/e51c01bf4ec69d254e4fd4fb79a341f99309fd38/openedx/core/djangoapps/cors_csrf/helpers.py#L75 (skip_cross_domain_referer_check context manager)

The call to skip_cross_domain_referer_check that concerns us is the one from the CorsCSRFMiddleware middleware: https://github.com/openedx/edx-platform/blob/89c8f21b94d420a3b58e33e3890bf4410a787487/openedx/core/djangoapps/cors_csrf/middleware.py#L76

What is strange to me is that this middleware has not changed for many years... And it's been enabled in the LMS for a long time. So maybe this is a false lead?

I need to log out. If someone wants to pick up from there, be my guest 🤗

@timmc-edx
Copy link
Contributor

Yeah, that cors_csrf middleware is kind of scary. I filed a DEPR for it a couple of days ago (https://discuss.openedx.org/t/deprecation-removal-cors-csrf-middleware-and-utilities-edx-platform-33627/11577) after I saw that code during review of openedx/edx-platform#33226.

@regisb
Copy link
Contributor

regisb commented Nov 13, 2023

There are two candidate solutions. It's important to realize that both are temporary, so let's not close this issue.

  1. Add LMS_HOST to CSRF_TRUSTED_ORIGINS (or CSRF_TRUSTED_ORIGINS_WITH_SCHEME): I think this might resolve the issue but it might open a new attack vector.
  2. Remove the 'openedx.core.djangoapps.cors_csrf.middleware.CorsCSRFMiddleware' entry from the MIDDLEWARE setting in the lms.envs.production module. I have no idea if there are unforeseen consequences with this change. This might break the marketing sites, and probably other sites as well.

I implemented solution 2 so that testing can resume. But we must realise that the issue is caused by the upgrade to Django 4.2 in the Quince branch. This backport DOES NOT EXIST in master. So we are in a strange situation where Quince runs Django 4.2, but master runs Django 3. This is highly unusual.

@mariajgrimaldi
Copy link
Member

mariajgrimaldi commented Nov 13, 2023

In Django 4. x, a new check for the CSRF middleware was introduced:
django/django#13829
https://github.com/django/django/blob/main/docs/releases/4.0.txt#L233-L238
That might explain why it's not failing in Django 3.x. But as far as I understand it doesn't explain why it fails with the LMS and not CMS that uses the same middleware.

@mariajgrimaldi
Copy link
Member

mariajgrimaldi commented Nov 13, 2023

It looks like in edx.org already added the host to CSRF_TRUSTED_ORIGINS so they only added the missing scheme, so maaaybe it wouldn't fail even if they were using django 4.0
#315 (comment)
https://openedx.atlassian.net/wiki/spaces/COMM/pages/3726802953/Pending+Release+Quince#Operational-%E2%9A%99%EF%B8%8F

@regisb
Copy link
Contributor

regisb commented Nov 14, 2023

maaaybe it wouldn't fail even if they were using django 4.0

I think you're right. But I'm not sure that disabling CSRF for the LMS is the right choice.

@zawan-ila
Copy link

why it fails with the LMS and not CMS that uses the same middleware

I believe it's the order of the middleware that's causing this. Note that in the CMS django's csrf middleware runs before 'openedx.core.djangoapps.cors_csrf.middleware.CorsCSRFMiddleware' whereas it's the other way around in the LMS.

In particular, in the CMS, once openedx.core.djangoapps.cors_csrf.middleware.CorsCSRFMiddleware calls process_view with its own patch for request.is_secure, process_view has already been called by django's CSRF view middleware(because of the order) and the second call short circuits here.

@regisb
Copy link
Contributor

regisb commented Nov 15, 2023

👏 👏 👏 @zawan-ila 👏 👏 👏

Nailed it! Following your suggestion, I added to the LMS production settings:

MIDDLEWARE.remove('django.middleware.csrf.CsrfViewMiddleware')
MIDDLEWARE.insert(MIDDLEWARE.index('corsheaders.middleware.CorsMiddleware'), 'django.middleware.csrf.CsrfViewMiddleware')

Lo and behold, this resolves the issue.

Thus, the right fix would be to move the Django middleware up in the LMS common settings. @zawan-ila would you like to open a PR or should I do it? Once the fix is in master we should also backport it to Quince.

Thank you so much for researching this @zawan-ila, we were truly out of our depth here.

@DeanJayMathew
Copy link

DeanJayMathew commented Nov 15, 2023 via email

@zawan-ila
Copy link

zawan-ila commented Nov 15, 2023

@regisb Feel free to open any pull requests as you see appropriate. I will not be making one :-)

@regisb
Copy link
Contributor

regisb commented Nov 16, 2023

Who wants to review this? openedx/edx-platform#33727

@mariajgrimaldi
Copy link
Member

@regisb: I'll do it.

mariajgrimaldi pushed a commit to eduNEXT/edunext-platform that referenced this issue Nov 21, 2023
POST requests to the LMS are failing systematically when HTTPS is
enabled. This issue is observed in the Quince release branch. Here is
the root cause analysis:

- CorsCSRFMiddleware overrides the `is_secure` attribute by setting it
  to "false".
- CorsCSRFMiddleware calls the parent `process_view` method, from the
  CsrfViewMiddleware.
- CsrfViewMiddleware checks the Origin header, including the scheme. It
  is equal to "https://LMSHOST". But because the request is not
  considered secure, the expected origin is "http://LMSHOST".
- The check fails with "Origin checking failed"

We resolve this issue by running the CsrfViewMiddleware *before* the
custom CorsCSRFMiddleware. After a successful check of the
CsrfViewMiddleware, the request has the "csrf_processing_done = True"
attribute, and CorsCSRFMiddleware is short-circuited.

This issue did not happen in the following environments:

- in Palm because the CsrfViewMiddleware did not check the "Origin" header in Django 3.
- in the Studio, because the Studio already runs the CsrfViewMiddleware before
  the CorsCSRFMiddleware.
- in the master branch because the master branch does not yet run on
  Django 4. But the issue will happen in the master branch without this
  proposed change.

To bypass this issue in the master branch, it was proposed that we add
"https://LMSHOST" to CSRF_TRUSTED_ORIGINS. This would effectily bypass
CSRF checking entirely for all requests that originate from the LMS.
Such a solution would not be acceptable, as we would lose the security
guarantees offered by CSRF.

See discussion: openedx/wg-build-test-release#325
mariajgrimaldi pushed a commit to openedx/edx-platform that referenced this issue Nov 23, 2023
POST requests to the LMS are failing systematically when HTTPS is
enabled. This issue is observed in the Quince release branch. Here is
the root cause analysis:

- CorsCSRFMiddleware overrides the `is_secure` attribute by setting it
  to "false".
- CorsCSRFMiddleware calls the parent `process_view` method, from the
  CsrfViewMiddleware.
- CsrfViewMiddleware checks the Origin header, including the scheme. It
  is equal to "https://LMSHOST". But because the request is not
  considered secure, the expected origin is "http://LMSHOST".
- The check fails with "Origin checking failed"

We resolve this issue by running the CsrfViewMiddleware *before* the
custom CorsCSRFMiddleware. After a successful check of the
CsrfViewMiddleware, the request has the "csrf_processing_done = True"
attribute, and CorsCSRFMiddleware is short-circuited.

This issue did not happen in the following environments:

- in Palm because the CsrfViewMiddleware did not check the "Origin" header in Django 3.
- in the Studio, because the Studio already runs the CsrfViewMiddleware before
  the CorsCSRFMiddleware.
- in the master branch because the master branch does not yet run on
  Django 4. But the issue will happen in the master branch without this
  proposed change.

To bypass this issue in the master branch, it was proposed that we add
"https://LMSHOST" to CSRF_TRUSTED_ORIGINS. This would effectily bypass
CSRF checking entirely for all requests that originate from the LMS.
Such a solution would not be acceptable, as we would lose the security
guarantees offered by CSRF.

See discussion: openedx/wg-build-test-release#325
regisb added a commit to overhangio/edx-platform that referenced this issue Nov 24, 2023
POST requests to the LMS are failing systematically when HTTPS is
enabled. This issue is observed in the Quince release branch. Here is
the root cause analysis:

- CorsCSRFMiddleware overrides the `is_secure` attribute by setting it
  to "false".
- CorsCSRFMiddleware calls the parent `process_view` method, from the
  CsrfViewMiddleware.
- CsrfViewMiddleware checks the Origin header, including the scheme. It
  is equal to "https://LMSHOST". But because the request is not
  considered secure, the expected origin is "http://LMSHOST".
- The check fails with "Origin checking failed"

We resolve this issue by running the CsrfViewMiddleware *before* the
custom CorsCSRFMiddleware. After a successful check of the
CsrfViewMiddleware, the request has the "csrf_processing_done = True"
attribute, and CorsCSRFMiddleware is short-circuited.

This issue did not happen in the following environments:

- in Palm because the CsrfViewMiddleware did not check the "Origin" header in Django 3.
- in the Studio, because the Studio already runs the CsrfViewMiddleware before
  the CorsCSRFMiddleware.
- in the master branch because the master branch does not yet run on
  Django 4. But the issue will happen in the master branch without this
  proposed change.

To bypass this issue in the master branch, it was proposed that we add
"https://LMSHOST" to CSRF_TRUSTED_ORIGINS. This would effectily bypass
CSRF checking entirely for all requests that originate from the LMS.
Such a solution would not be acceptable, as we would lose the security
guarantees offered by CSRF.

See discussion: openedx/wg-build-test-release#325
mariajgrimaldi pushed a commit to openedx/edx-platform that referenced this issue Nov 24, 2023
POST requests to the LMS are failing systematically when HTTPS is
enabled. This issue is observed in the Quince release branch. Here is
the root cause analysis:

- CorsCSRFMiddleware overrides the `is_secure` attribute by setting it
  to "false".
- CorsCSRFMiddleware calls the parent `process_view` method, from the
  CsrfViewMiddleware.
- CsrfViewMiddleware checks the Origin header, including the scheme. It
  is equal to "https://LMSHOST". But because the request is not
  considered secure, the expected origin is "http://LMSHOST".
- The check fails with "Origin checking failed"

We resolve this issue by running the CsrfViewMiddleware *before* the
custom CorsCSRFMiddleware. After a successful check of the
CsrfViewMiddleware, the request has the "csrf_processing_done = True"
attribute, and CorsCSRFMiddleware is short-circuited.

This issue did not happen in the following environments:

- in Palm because the CsrfViewMiddleware did not check the "Origin" header in Django 3.
- in the Studio, because the Studio already runs the CsrfViewMiddleware before
  the CorsCSRFMiddleware.
- in the master branch because the master branch does not yet run on
  Django 4. But the issue will happen in the master branch without this
  proposed change.

To bypass this issue in the master branch, it was proposed that we add
"https://LMSHOST" to CSRF_TRUSTED_ORIGINS. This would effectily bypass
CSRF checking entirely for all requests that originate from the LMS.
Such a solution would not be acceptable, as we would lose the security
guarantees offered by CSRF.

See discussion: openedx/wg-build-test-release#325
@mariajgrimaldi
Copy link
Member

This was fixed by the PR linked above.

@fayyazahmed66
Copy link
Author

@mariajgrimaldi Verified. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release blocker Blocks the upcoming release (fix needed)
Projects
Development

No branches or pull requests

7 participants