chore: Upgrade Superset to version 4.0.1

[saraburns1]

closes #835

Tested as Superuser and Instructor (non-superuser). All local dashboards work correctly, no new errors in lms/superset logs.

[pomegranited]

Hi @saraburns1 CC @bmtcril :)

I'm trying to rebuild my Tutor Aspects dev stack from scratch (which I haven't done for a couple months), but I'm getting this error, which prevents the init-superset.sh from completing.

Table 'superset.ab_permission_view_role' doesn't exist

I've tried the following steps using superset versions 4.0.0, 4.0.1, and 4.0.2:

• rollback all alembic migrations: tutor dev do alembic -c "downgrade base"
• rebuild aspects-superset image: tutor images build --no-cache aspects-superset
• re-run init: tutor dev do init -l aspects

..but the error persists. It might be a regression for superset -- people were reporting this same issue for 2.0.0 -> 2.0.1 -- but I haven't found anyone reporting this issue for 4.x.

Have you seen this before? Is there anything else I could try?

[bmtcril]

I've seen this a couple of times recently, mostly in CI. I've always been able to re-run and had it succeed the 2nd time. It does seem entirely like a superset regression.

[saraburns1]

I'd also try rebuilding the aspects image and/or doing a 'tutor dev launch'.

…

On Wed, Jul 31, 2024 at 5:04 PM Brian Mesick ***@***.***> wrote: I've seen this a couple of times recently, mostly in CI. I've always been able to re-run and had it succeed the 2nd time. It does seem entirely like a superset regression. — Reply to this email directly, view it on GitHub <#838 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BIR4DO3M5SNNFZAFJ7MT723ZPFGMTAVCNFSM6AAAAABIO5MBTOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENRRGQ2TQOBYHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[cmltaWt0]

I've faced the same issue with a Production deployment.
Info:

1. tutor-contrib-aspects==v1.3.2
2. superset 4.0.2

Exception during the init jobs

Displaying logs from Namespace: credix-prod for Job: superset-job-20250110103143. Logs from 10/01/2025, 12:31:49
Loading...
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2043, in _handle_dbapi_exception
util.raise_(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
File "/usr/local/lib/python3.10/site-packages/MySQLdb/cursors.py", line 206, in execute
res = self._query(query)
File "/usr/local/lib/python3.10/site-packages/MySQLdb/cursors.py", line 319, in _query
db.query(q)
File "/usr/local/lib/python3.10/site-packages/MySQLdb/connections.py", line 254, in query
_mysql.connection.query(self, query)
sqlalchemy.exc.ProgrammingError: (MySQLdb._exceptions.ProgrammingError) (1146, "Table 'superset.ab_permission_view_role' doesn't exist")
[SQL: SELECT ab_role.id AS ab_role_id, ab_role.name AS ab_role_name, ab_permission_view_1.id AS ab_permission_view_1_id, ab_permission_view_1.permission_id AS ab_permission_view_1_permission_id, ab_permission_view_1.view_menu_id AS ab_permission_view_1_view_menu_id
FROM ab_role LEFT OUTER JOIN (ab_permission_view_role AS ab_permission_view_role_1 INNER JOIN ab_permission_view AS ab_permission_view_1 ON ab_permission_view_1.id = ab_permission_view_role_1.permission_view_id) ON ab_role.id = ab_permission_view_role_1.role_id]
(Background on this error at: https://sqlalche.me/e/14/f405)

######################################################################


Init Step 1/5 [Starting] -- Applying DB migrations


######################################################################

2025-01-10 10:32:10,543:INFO:superset.utils.logging_configurator:logging was configured successfully
2025-01-10 10:32:10,550:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
/usr/local/lib/python3.10/site-packages/flask_limiter/extension.py:293: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
warnings.warn(
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.runtime.migration] Running upgrade a8173232b786 -> e38177dbf641, security converge saved queries
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
Loaded your Docker configuration at [/app/pythonpath/superset_config_docker.py]
Loaded your LOCAL configuration at [/app/pythonpath/superset_config.py]
self.dialect.do_execute(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
File "/usr/local/lib/python3.10/site-packages/MySQLdb/cursors.py", line 206, in execute
res = self._query(query)
File "/usr/local/lib/python3.10/site-packages/MySQLdb/cursors.py", line 319, in _query
db.query(q)
File "/usr/local/lib/python3.10/site-packages/MySQLdb/connections.py", line 254, in query
_mysql.connection.query(self, query)
MySQLdb._exceptions.ProgrammingError: (1146, "Table 'superset.ab_permission_view_role' doesn't exist")

Interesting that for the Dev environment everything works fine and the table is presented:

mysql> show tables;
+----------------------------+
| Tables_in_superset         |
+----------------------------+
| ab_permission              |
| ab_permission_view         |
| ab_permission_view_role    |
| ab_register_user           |
| ab_role                    |
| ab_user                    |
| ab_user_role               |
| ab_view_menu               |
| alembic_version            |
| annotation                 |
| annotation_layer           |
| cache_keys                 |
| css_templates              |
| dashboard_roles            |
| dashboard_slices           |
| dashboard_user             |
| dashboards                 |
| dbs                        |
| dynamic_plugin             |
| embedded_dashboards        |
| favstar                    |
| key_value                  |
| keyvalue                   |
| logs                       |
| query                      |
| report_execution_log       |
| report_recipient           |
| report_schedule            |
| report_schedule_user       |
| rls_filter_roles           |
| rls_filter_tables          |
| row_level_security_filters |
| saved_query                |
| sl_columns                 |
| sl_dataset_columns         |
| sl_dataset_tables          |
| sl_dataset_users           |
| sl_datasets                |
| sl_table_columns           |
| sl_tables                  |
| slice_user                 |
| slices                     |
| sql_metrics                |
| sqlatable_user             |
| ssh_tunnels                |
| tab_state                  |
| table_columns              |
| table_schema               |
| tables                     |
| tag                        |
| tagged_object              |
| user_attribute             |
| user_favorite_tag          |
+----------------------------+
53 rows in set (0.00 sec)

I've tried to re-run the deploy from scratch on the new DB w/o any success for now. But at least I have a reproducible issue :)

[cmltaWt0]

UDP:

While writing this message, I decided to try one last option: manually dropping the database with DROP DATABASE superset;. Then, I re-ran the deployment on production, and magically, all the required tables were created!

I have no idea why this worked because, previously, I had tried completely destroying the MySQL instance along with the PVC (it's a Kubernetes installation) without any success.

@bmtcril @pomegranited FYI

[bmtcril]

Yep, there are a number of issues on the Superset repository about this. It seems very random, probably some kind of race condition. On our side the workaround is just to re-run Tutor init limited to Aspects. It's otherwise harmless, just annoying.