[Issue #10] Populate the search index from the opportunity tables

[chouinar]

Summary

Fixes #10

Time to review: 10 mins

Changes proposed

Setup a script to populate the search index by loading opportunities from the DB, jsonify'ing them, loading them into a new index, and then aliasing that index.

Several utilities were created for simplifying working with the OpenSearch client (a wrapper for setting up configuration / patterns)

Context for reviewers

Iterating over the opportunities and doing something with them is a common pattern in several of our scripts, so nothing is really different there.

The meaningful implementation is how we handle creating and aliasing the index. In OpenSearch you can give any index an alias (including putting multiple indexes behind the same alias). The approach is pretty simple:

• Create an index
• Load opportunities into the index
• Atomically swap the index backing the opportunity-index-alias
• Delete the old index if they exist

This approach means that our search endpoint just needs to query the alias, and we can keep making new indexes and swapping them out behind the scenes. Because we could remake the index every few minutes, if we ever need to re-configure things like the number of shards, or any other index-creation configuration, we just update that in this script and wait for it to run again.

Additional information

I ran this locally after loading 83250 records, and it took about 61s.

You can run this locally yourself by doing:

make init
make db-seed-local
poetry run flask load-search-data load-opportunity-data

If you'd like to see the data, you can test it out on http://localhost:5601/app/dev_tools#/console - here is an example query that filters by the word research across a few fields and filters to just forecasted/posted.

GET opportunity-index-alias/_search
{
  "size": 25,
  "from": 0,
  "query": {
    "bool": {
      "must": [
        {
          "simple_query_string": {
            "query": "research",
            "default_operator": "AND", 
            "fields": ["agency.keyword^16", "opportunity_title^2", "opportunity_number^12", "summary.summary_description", "opportunity_assistance_listings.assistance_listing_number^10", "opportunity_assistance_listings.program_title^4"]
          }
        }
      ],
      "filter": [
        {
          "terms": {
            "opportunity_status": [
              "forecasted",
              "posted"
            ]
          }
        }
      ]
    }
  }
}

[acouch]

(thought) This could be a pain point in the future depending on how large the records get, but looks great for now.

[chouinar]

Its much faster than you'd think. Locally I ran ~87000 records through this script in about 61 seconds. And that includes querying the DB, joining across all the tables, doing this iteration to jsonify, and then doing the bulk inserts.

The batching (which is done by the DB queries at 5000 records per batch) makes it scale pretty uneventfully.

I imagine on an actual cluster it'll be a bit slower as we'll probably have more nodes, but I don't see why this would ever go beyond ~5 minutes in a run, and there are a few quick optimizations (removing the index refresh from every bulk insert)

[acouch]

Also exciting. We have relevancy!

The base branch was changed.