[Incident] 403 Errors on UofT and Temple Hubs

[jmunroe]

Summary

Community reports of errors with the UofT hubs at https://jupyter.utoronto.ca/ and https://r.datatools.utoronto.ca/ like:

403 : Forbidden
Sorry, you are not currently authorized to use this hub. Please contact the hub administrator.
First report was just after midnight (Eastern) today. We are able to replicate the issue. Please give it a look, thanks!

Impact on users

Users cannot log into the JuptyerHub.

Important information

• Hub URL: https://jupyter.utoronto.ca/
• Support ticket ref: https://2i2c.freshdesk.com/a/tickets/974 , https://2i2c.freshdesk.com/a/tickets/975

Tasks and updates

• Discuss and address incident, leaving comments below with updates
• Incident has been dealt with or is over
• Copy/paste the after-action report below and fill in relevant sections
• Incident title is discoverable and accurate
• All actionable items in report have linked GitHub Issues

After-action report template

# After-action report

These sections should be filled out once we've resolved the incident and know what happened.
They should focus on the knowledge we've gained and any improvements we should take.

## Timeline

_A short list of dates / times and major updates, with links to relevant comments in the issue for more context._

All times in {{ most convenient timezone}}.

- 2023-09-14 - [Summary of first update](link to comment)
- {{ yyyy-mm-dd }} - [Summary of another update](link to comment)
- {{ yyyy-mm-dd }} - [Summary of final update](link to comment)


## What went wrong

_Things that could have gone better. Ideally these should result in concrete
action items that have GitHub issues created for them and linked to under
Action items._

- Thing one
- Thing two

## Where we got lucky

_These are good things that happened to us but not because we had planned for them._

- Thing one
- Thing two

## Follow-up actions

_Every action item should have a GitHub issue (even a small skeleton of one) attached to it, so these do not get forgotten. These issues don't have to be in `infrastructure/`, they can be in other repositories._

### Process improvements

1. {{ summary }} [link to github issue]
2. {{ summary }} [link to github issue]

### Documentation improvements

1. {{ summary }} [link to github issue]
2. {{ summary }} [link to github issue]

### Technical improvements

1. {{ summary }} [link to github issue]
2. {{ summary }} [link to github issue]

[jmunroe]

There was an upgrade last night in the oauthenticator (15.1 -> 16) package that appears to have caused issues for logging across many hubs.

• Upgrade to z2jh 3.0.2 from 3.0.0-beta.1 - oauthenticator 15.1 bumped to 16.0 #3118

That change has now been rolled back

• Revert "Upgrade to z2jh 3.0.2 from 3.0.0-beta.1 - oauthenticator 15.1 bumped to 16.0" #3139

This incident should now be resolved and students should be able to log in as before. Both UofT and Temple have been communicated with and the corresponding Freshdesk tickets are now closed.

[jmunroe]

This was @jmunroe first time going through the Incident Response process in 'incident commander' role. I can confirm that the documentation given at https://team-compass.2i2c.org/projects/managed-hubs/incidents/ was sufficient for me to follow.

My one small hiccup was not seeing immediately how to create the Slack channel. The instructions referred to "checking the box for Create a dedicated Public Slack channel for this incident and I either missed it or it wasn't there. I spent some time trying to find how to create the incident Slack channel from the PagerDuty web UI before recognizing that there was a button in the #pagerduty-notifications slack widget tool.

While it was clear to me that this constituted an "incident" so I felt I needed to shift focus to it as soon as I learned about it (via Freshdesk email notification), I was a bit hesitant to assume I should take on the 'Incident Commander' role. In life guarding/first aid contexts, the first person on the scene is automatically in that role until they are officially relieved of that role so I assumed it would be the same here.

[jmunroe]

I appreciated @consideRatio being available and willing to assist with this outage even though he was scheduled for leave today.

The PR that caused the outage #3118 looked like it affected many parts of our infrastructure. We should revisit our testing procedures to improve our likelihood of catching authentication related errors.

Was the nature of the upgrade such at all hubs needs to be changed over together? I don't understand enough about details of the upgrade z2jh and oauthenticator to know if they are intimately linked or could have been split into two separate changes.

For testing, I think we need to have at least one 'regular user' (non-privileged, non-admin, non-2i2c) account able to log in for the hubs. This is especially important for our education hubs where we are often not using GitHub or Google but and education oauth provider.

For UofT, I have a UTORid that I think would have caught the issue if I had tested it with those credentials. I don't know how we would have verified the Temple hub. My understanding is authentication was still working against Google but the problem was related to these other oauth providers and the changes in oauthenticator 16.0.

[jmunroe]

@consideRatio (and a few others) is on leave today and tomorrow, so we should revisit this incident early next week to debrief and create an incident report.