v7 - The grown up edition
Alright, put down that coffee, this is a though one. You'll see, come here and sit with me.
I had to break a lot of things.
- Postgres (PG) Database
- Node Red flows
- New way to fetch RSS feeds
- New admin option
- Metabase module
Database
Gregory was using a SQLite database, which is ok for most uses and was enough for the first year. The data was moved to a new Postgres database. This solves issues we were having with incoherent dates and allows Metabase to run a better analysis on everything involving a date and time. Same goes for True and False values, which SQLite considered to be either 0 and 1 and made some queries feel weird.
One change was adding some new fields to the table of sources, these now include the method used to fetch new articles. More on that up ahead.
Node Red flows
New database meant reviewing the flows that search and retrieve new articles, making them use PG. During this, I cleaned up some nodes and flows that didn't make much sense. Sorry for not documenting this, but either way, Gregory is meant to mostly use your flows and not what I believe is right.
New way to fetch rss feeds
On the python-ml directory there is now a feedreader.py script that will query the database for a list of sources that use RSS, fetch new articles and add them to the database. This may feel like a nuisance, but it's necessary to move towards a system where you add sources through an admin panel instead of coding a Node Red flow for each one.
New admin option
Sometimes you need to edit information on the articles, sources or trials. To make this easier I have added a Django install to the docker-compose.yaml file.
When running this for the first time, *remember to execute python manage.py migrate and then python manage.py createsuperuser so that it will add the necessary tables and a user account.
This is available at http://localhost:8000 and you will have to bind it with your preferred reverse proxy.
Metabase module
I broke this harder than my last girlfriend broke my heart.
It didn't make sense to have another PG container running just for Metabase. Instead, it was added to the new one that is holding the articles and clinical trials. We keep things separate by using two different databases. Should have done the same for the Django install but didn't know how to and had to pick priorities.
This means that the charts in the observatory section of the website have all gone to hell and will need to brought back. User accounts will also need to be recreated.
That's all, and that's a lot
This was a lot of work and it will break all other installs of Gregory, I'm sorry about that. Hopefully you will agree it was a necessary step towards a system that is easier to customize and adapt to other conditions other than Multiple Sclerosis.
I'd like to thank @malduarte and @chbm who gave me a hand in migrating to PG. It would have been a lot harder without their help.
Any questions, comments or suggestions are more than welcome. Meanwhile I am just going to take it easy for rest of the week and calmly make sure everything is working as intended.
falls down in Portuguese
What's Changed
I messed up the auto-generated release notes, so you'll want to check the v7 milestone.
