Skip to content
This repository has been archived by the owner on Aug 1, 2024. It is now read-only.

feat: Added Opensearch 1.2.0 #925

Merged
merged 1 commit into from
Apr 27, 2022
Merged

feat: Added Opensearch 1.2.0 #925

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,20 +227,22 @@ impl-dev.provision.%: dev.check-memory ## Provision specified services.
dev.provision.%: ## Provision specified services.
@scripts/send_metrics.py wrap "dev.provision.$*"

dev.backup: dev.up.mysql57+mongo+elasticsearch+elasticsearch7+elasticsearch710+coursegraph ## Write all data volumes to the host.
dev.backup: dev.up.mysql57+mongo+elasticsearch+elasticsearch7+elasticsearch710+opensearch12+coursegraph ## Write all data volumes to the host.
docker run --rm --volumes-from $$(make --silent --no-print-directory dev.print-container.mysql57) -v $$(pwd)/.dev/backups:/backup debian:jessie tar zcvf /backup/mysql57.tar.gz /var/lib/mysql
docker run --rm --volumes-from $$(make --silent --no-print-directory dev.print-container.mongo) -v $$(pwd)/.dev/backups:/backup debian:jessie tar zcvf /backup/mongo.tar.gz /data/db
docker run --rm --volumes-from $$(make --silent --no-print-directory dev.print-container.elasticsearch) -v $$(pwd)/.dev/backups:/backup debian:jessie tar zcvf /backup/elasticsearch.tar.gz /usr/share/elasticsearch/data
docker run --rm --volumes-from $$(make --silent --no-print-directory dev.print-container.elasticsearch7) -v $$(pwd)/.dev/backups:/backup debian:jessie tar zcvf /backup/elasticsearch7.tar.gz /usr/share/elasticsearch/data
docker run --rm --volumes-from $$(make --silent --no-print-directory dev.print-container.elasticsearch710) -v $$(pwd)/.dev/backups:/backup debian:jessie tar zcvf /backup/elasticsearch710.tar.gz /usr/share/elasticsearch/data
docker run --rm --volumes-from $$(make --silent --no-print-directory dev.print-container.opensearch12) -v $$(pwd)/.dev/backups:/backup debian:jessie tar zcvf /backup/opensearch12.tar.gz /usr/share/opensearch/data
docker run --rm --volumes-from $$(make --silent --no-print-directory dev.print-container.coursegraph) -v $$(pwd)/.dev/backups:/backup debian:jessie tar zcvf /backup/coursegraph.tar.gz /data

dev.restore: dev.up.mysql57+mongo+elasticsearch+elasticsearch7+elasticsearch710+coursegraph ## Restore all data volumes from the host. WILL OVERWRITE ALL EXISTING DATA!
dev.restore: dev.up.mysql57+mongo+elasticsearch+elasticsearch7+elasticsearch710+opensearch12+coursegraph ## Restore all data volumes from the host. WILL OVERWRITE ALL EXISTING DATA!
docker run --rm --volumes-from $$(make --silent --no-print-directory dev.print-container.mysql57) -v $$(pwd)/.dev/backups:/backup debian:jessie tar zxvf /backup/mysql57.tar.gz
docker run --rm --volumes-from $$(make --silent --no-print-directory dev.print-container.mongo) -v $$(pwd)/.dev/backups:/backup debian:jessie tar zxvf /backup/mongo.tar.gz
docker run --rm --volumes-from $$(make --silent --no-print-directory dev.print-container.elasticsearch) -v $$(pwd)/.dev/backups:/backup debian:jessie tar zxvf /backup/elasticsearch.tar.gz
docker run --rm --volumes-from $$(make --silent --no-print-directory dev.print-container.elasticsearch7) -v $$(pwd)/.dev/backups:/backup debian:jessie tar zxvf /backup/elasticsearch7.tar.gz
docker run --rm --volumes-from $$(make --silent --no-print-directory dev.print-container.elasticsearch710) -v $$(pwd)/.dev/backups:/backup debian:jessie tar zxvf /backup/elasticsearch710.tar.gz
docker run --rm --volumes-from $$(make --silent --no-print-directory dev.print-container.opensearch12) -v $$(pwd)/.dev/backups:/backup debian:jessie tar zxvf /backup/opensearch12.tar.gz
docker run --rm --volumes-from $$(make --silent --no-print-directory dev.print-container.coursegraph) -v $$(pwd)/.dev/backups:/backup debian:jessie tar zxvf /backup/coursegraph.tar.gz

# List of Makefile targets to run database migrations, in the form dev.migrate.$(service)
Expand Down
23 changes: 23 additions & 0 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,25 @@ services:
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"

# This is meant to be used to test OS upgrades.
opensearch12:
container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.opensearch12"
hostname: opensearch12.devstack.edx
image: opensearchproject/opensearch:1.2.0
networks:
default:
aliases:
- edx.devstack.opensearch12
ports:
- "9202:9200"
- "9600:9600"
volumes:
- opensearch12_data:/usr/share/opensearch/data
environment:
- discovery.type=single-node
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"

firefox:
container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.firefox"
hostname: firefox.devstack.edx
Expand Down Expand Up @@ -396,6 +415,7 @@ services:
- memcached
- mongo
- elasticsearch710
- opensearch12
image: edxops/forum:${OPENEDX_RELEASE:-latest}
stdin_open: true
tty: true
Expand All @@ -418,6 +438,7 @@ services:
- memcached
- mongo
- mysql57
- opensearch12
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Can you update the PR description with a little more background about how and when each of the services will rely on this?
  2. Will this just be needed for a rare use case? I’m not clear on what “test OS upgrades” entails, but it seems like a more rare activity and that maybe this isn’t a required dependency for most uses of LMS or other services? This is a question. If not, maybe there should simply be doc changes for when and how to bring this container up?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the best reference for this is the Elasticsearch -> OpenSearch ticket we have on the DEPR board. We are migrating all our services off Elasticsearch and for those that can't use MySQL full text search, we're going to use OpenSearch. openedx/public-engineering#16

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we are migrating LMS, studio, cs_comments(forum), and blockstore from Elasticsearch -> OpenSearch.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. Seems fine if you think we need both dependencies now.

# Allows attachment to the LMS service using 'docker attach <containerID>'.
stdin_open: true
tty: true
Expand Down Expand Up @@ -577,6 +598,7 @@ services:
- memcached
- mongo
- mysql57
- opensearch12
# Allows attachment to the Studio service using 'docker attach <containerID>'.
stdin_open: true
tty: true
Expand Down Expand Up @@ -814,4 +836,5 @@ volumes:
elasticsearch7_data:
elasticsearch710_data:
mongo_data:
opensearch12_data:
mysql57_data:
2 changes: 1 addition & 1 deletion options.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,4 +90,4 @@ credentials+discovery+ecommerce+insights+lms+registrar+studio
# All third-party services.
# Separated by plus signs. Listed in alphabetical order for clarity.
THIRD_PARTY_SERVICES ?= \
chrome+coursegraph+devpi+elasticsearch+elasticsearch7+elasticsearch710+firefox+memcached+mongo+mysql57+redis+namenode+datanode+resourcemanager+nodemanager+sparkmaster+sparkworker+vertica
chrome+coursegraph+devpi+elasticsearch+elasticsearch7+elasticsearch710+firefox+memcached+mongo+mysql57+opensearch12+redis+namenode+datanode+resourcemanager+nodemanager+sparkmaster+sparkworker+vertica