Elasticsearch 7.x.0 setup and test locally

1. Install Elasticsearch 7.x.0 locally (both 7.4.0 and 7.6.0 work well)

1)Check Java version run: java -version jdk: openjdk version "11.0.2" 2019-01-15 jre: OpenJDK Runtime Environment 18.9(build 11.0.2+9)
2)you can create a folder: es7
3)run these two commands to install
curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.4.0-darwin-x86_64.tar.gz
tar -xvf elasticsearch-7.4.0-darwin-x86_64.tar.gz

or if you want to install ES7.6.0:
curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.6.0-darwin-x86_64.tar.gz
tar -xvf elasticsearch-7.6.0-darwin-x86_64.tar.gz

4) To fix "controller.app can't be opened error", open elasticsearch-7.x.0/config/elasticsearch.yml
below line 37: #path.logs: /path/to/logs
add: xpack.ml.enabled: false

5) Start Elasticsearch
cd elasticsearch-7.x.0/bin
./elasticsearch
6) Test your local elasticsearch server on browser: http://127.0.0.1:9200/

2. Setup openFEC repo environment

1)Check out branch: develop branch
2) Create a new virtual environment and run: pip install -r requirements.txt
3) Set SQLA_CONN to dev db

3. Create es-index, alias and mapping

1)Start local elasticsearch server 7.x.0
2)Create index 'xxxx_index' and two aliases ('xxxx_alias' and 'search_alias')
ex1: python cli.py create_index case_index
ex2: python cli.py create_index ao_index
ex3: python cli.py create_index arch_mur_index

4. Display local Elasticsearch resources

1)Display index and alias
From terminal: python cli.py display_index_alias
From browser: http://localhost:9200/_cat/indices?v
http://localhost:9200/_aliases?pretty

2)Display mappings
From terminal:
ex1: python cli.py display_mapping case_index
ex2: python cli.py display_mapping ao_index
ex3: python cli.py display_mapping arch_mur_index
From browser: http://localhost:9200/_mappings

5. Delete index and alias

python cli.py delete_index <index_name>

ex1: python cli.py delete_index case_index
ex2: python cli.py delete_index ao_index
ex3: python cli.py delete_index arch_mur_index

6. Upload legal documents to local elasticsearch server

Notes:
a)The legal pdf files won't upload to S3 bucket when you upload from local. so there is no S3 connection locally, please ignore some warning messages.
b)You can un-comment this line (logger.setLevel(logging.DEBUG)) on the top of each upload python file to enable DEBUG mode.
c)All document types (ES mapping type) Under index 'case_index':
'murs', 'adrs', 'admin_fines',
d)All document types (ES mapping type) Under index 'ao_index':
'advisory_opinions', 'statutes'
e)The document type (ES mapping type) Under index 'arch_mur_index': 'murs'

1) Upload AO (advisory_opinions.py)
python cli.py load_advisory_opinions YYYY-MM
(this command will load AO docs from YYYY-MM to current time)
ex: python cli.py load_advisory_opinions 2020-03
Test on browser: http://localhost:9200/ao_index/_doc/2020-05

2) Upload current murs (current_case.py)
python cli.py load_current_murs (upload all current murs docs)
python cli.py load_current_murs <mur_no> (upload single current mur doc)
ex: python cli.py load_current_murs 7212
Test on browser: http://localhost:9200/case_index/_doc/mur_7212

3) Upload adrs (current_case.py)
python cli.py load_adrs (upload all adr docs)
python cli.py load_adrs <adr_no> (upload single adr doc)
ex: python cli.py load_adrs 008
Test on browser: http://localhost:9200/case_index/_doc/adr_008

4) Upload admin fines (current_case.py)
python cli.py load_admin_fines (upload all admin fines docs)
python cli.py load_admin_fines <admin_fines_no> (upload single admin fine doc)
ex: python cli.py load_admin_fines 3314
Test on browser: http://localhost:9200/case_index/_doc/af_3314

5) Upload statutes (statutes.py)
python cli.py load_statutes (upload all statutes docs)
There is no way to test statutes on browser.
You can use curl command to run in terminal:
curl -X GET "localhost:9200/ao_index/_search?pretty" -H 'Content-Type: application/json' -d'{"query": {"terms": {"_id": [ "/us/usc/t26/s9009", "/us/usc/t52/s30110"] }}}'

6) Upload archived MURs (archived_murs.py)
python cli.py load_archived_murs (upload all archived mur docs)
python cli.py load_archived_murs <mur_no> (upload single archived murs doc)
ex: python cli.py load_archived_murs 400
Test on browser: http://localhost:9200/archived_murs/_doc/mur_400

7)Show the type of analyzer set on an index: http://localhost:9200/<index_name>/_settings
ex: http://localhost:9200/case_index/_settings
ex: http://localhost:9200/ao_index/_settings
ex: http://localhost:9200/arch_mur_index/_settings

7. Test endpoints

Provide feedback

Saved searches

Use saved searches to filter your results more quickly