Skip to content

Commit

Permalink
update readme
Browse files Browse the repository at this point in the history
  • Loading branch information
@brunoamaral
brunoamaral committed Apr 15, 2022
1 parent 254d6b1 commit 69ef14f
Showing 1 changed file with 48 additions and 45 deletions.
93 changes: 48 additions & 45 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# Gregory

Gregory aggregates searches in JSON and outputs to a Hugo static site

# Sources for searches
Expand All @@ -16,88 +17,90 @@ Gregory aggregates searches in JSON and outputs to a Hugo static site

# Live Version

https://gregory-ms.com
<https://gregory-ms.com>

# Install

## Server Requirements

- [ ] [Docker](https://www.docker.com/) and [docker-compose](https://docs.docker.com/compose/) with 2GB of swap memory to be able to build the MachineLearning Models. [Adding swap for Ubuntu](https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-20-04)
- [ ] [Hugo](https://gohugo.io/)
- [ ] [Docker](https://www.docker.com/) and [docker-compose](https://docs.docker.com/compose/) with 2GB of swap memory to be able to build the MachineLearning Models. [Adding swap for Ubuntu](https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-20-04)
- [ ] [Hugo](https://gohugo.io/)
- [ ] [Mailgun](https://www.mailgun.com/)

## Setup the environment

Run `pip3 install -r python-ml/requirements.txt`

Execute `python3 setup.py`. The script will check if you have all the requirements and run the Node-RED container.
1. Edit the .env file to reflect your settins and credentials.
2. Execute `python3 setup.py`. The script will check if you have all the requirements and run the Node-RED container.
3. Run `sudo docker-compose up -d` to fetch the required images and run the

Edit the .env file's variables to reflect your environment.
The final website is built by running `python3 ./build.py`.

Finally, build the site with `python3 ./build.py`.
## Running django

# Node-RED
Once the db container is running, start the admin container and run the following:

This is where all of the processing of information happens. Node-RED provides a set of API endpoints to access the information it collects.
```bash
sudo docker exec -it admin /bin/bash
python manage.py makemigrations
python manage.py migrate
python manage.py createsuperuser
```

The following tabs have been configured to divide the flows:
Now you can login at <https://YOUR-SUB.DOMAIN/admin> or wherever your reverse proxy is listening on.

1. Research (Collect data from different sources)
2. Email Digest (Sent to the admin and to subscribers)
3. API
4. Tests
# How everything fits together

`data/articles.json` and `data/trials.json` are generated from a Node-Red flow available in the `flows.json` file.
## Node-RED

Node-RED is installed with a custom dockerfile from their official repository. https://github.com/node-red/node-red-docker/tree/master/docker-custom
We are using [Node-RED](https://nodered.org/) to collect articles and clinical trials.

# Mailgun
Node-RED is installed with a custom dockerfile that includes some Machine Learning and Artificial Intelligence python modules, in case we need them. It is derived from the [Node-Red repository](https://github.com/node-red/node-red-docker/tree/master/docker-custom).

Currently, we are using Mailgun to send emails to the admin and subscribers. These nodes can be found on the Email Digest tab of Node-RED and have been disabled.
## Django and Postgres

To enable them, you will need a mailgun account, or you can replace them with another way to send emails.
The database and the API are managed using the `admin` container that comes with [Django](https://www.djangoproject.com/) and the [Django Rest Framework](https://www.django-rest-framework.org/) installed.

# Database
The database is running on a postgres container called `db`.

The path https://api.gregory-ms.com/articles/all and https://api.gregory-ms.com/trials/all includes the full database export.
## Metabase

The same information is available in excel and json format: https://gregory-ms.com/downloads/
A database is only as good as it's ability to answer questions. We have a separate `metabase` container in the docker-compose.yaml file that connects directly to Gregory's database.

# Update the Machine Learning Algorithms
It's available at <http://localhost:3000/>

1. `cd docker-python; source .venv/bin/activate`
2. `python3 1_data_processor.py`
3. `python3 2_train_models.py`
4. Login to sqlite3: `sqlite3 gregory/docker-data/gregory.db`
5. Reset the Machine Learning records with `UPDATE articles SET ml_prediction_gnb = NULL, ml_prediction_lr = NULL WHERE article_id > 0;`
6. The Node-Red flow to review the articles runs every 10 minutes.
The current website is also using some embeded dashboards whose keys are produced each time you run `build.py`. An example can be found in the [MS Observatory Page](https://gregory-ms.com/observatory/)

## Mailgun

Email are sent from the `admin` container using Mailgun.

To enable them, you will need a mailgun account, or you can replace them with another way to send emails.

## Running django

Once the db container is running, start the admin container and run the following:
You need to configure the relevant variables for this to work:

```bash
sudo docker exec -it admin /bin/bash
python manage.py makemigrations
python manage.py migrate
python manage.py createsuperuser
EMAIL_USE_TLS=true
EMAIL_MAILGUN_API='YOUR API KEY'
EMAIL_DOMAIN='YOURDOMAIN'
EMAIL_MAILGUN_API_URL="https://api.eu.mailgun.net/v3/YOURDOMAIN/messages"
```

Now you can login at https://YOUR-SUB.DOMAIN/admin or wherever your reverse proxy is listening on.
# Update the Machine Learning Algorithms

This is not working right now and there is a [pull request to setup an automatic process to keep the machine learning models up to date](https://github.com/brunoamaral/gregory/pull/110).

1. `cd docker-python; source .venv/bin/activate`
2. `python3 1_data_processor.py`
3. `python3 2_train_models.py`

# Thank you to

@[Antoniolopes](https://github.com/antoniolopes) for helping with the Machine Learning script.
@[Chbm](https://github.com/chbm) for help in keeping the code secure.
@[Malduarte](https://github.com/malduarte) for help with the migration from sqlite to postgres.
@[Jneves](https://github.com/jneves) for help with the build script
@[Melo](https://github.com/melo) for showing me [Hugo](https://github.com/gohugoio/hugo)
@[Nurv](https://github.com/nurv) for the suggestion in using Spacy.io
@[Rcarmo](https://github.com/rcarmo) for showing me [Node-RED](https://github.com/node-red/node-red)
@[Chbm](https://github.com/chbm) for help in keeping the code secure.
@[Malduarte](https://github.com/malduarte) for help with the migration from sqlite to postgres.
@[Jneves](https://github.com/jneves) for help with the build script
@[Melo](https://github.com/melo) for showing me [Hugo](https://github.com/gohugoio/hugo)
@[Nurv](https://github.com/nurv) for the suggestion in using Spacy.io
@[Rcarmo](https://github.com/rcarmo) for showing me [Node-RED](https://github.com/node-red/node-red)

And the **Lobsters** at [One Over Zero](https://github.com/oneoverzero)

0 comments on commit 69ef14f

Please sign in to comment.