RCORE-1928 Use baasaas for baas integration tests in CI #7423

jbreams · 2024-03-06T17:19:09Z

What, How & Why?

This updates the baas integration tests in CI to use baas managed via baasaas rather than spinning up an ubuntu machine via host.create.

☑️ ToDos

📝 Changelog update
🚦 Tests (or not relevant)
C-API, if public C++ API changed
bindgen/spec.yml, if public C++ API changed

coveralls-official · 2024-03-06T20:44:59Z

Pull Request Test Coverage Report for Build jonathan.reams_3083

Details

201 of 291 (69.07%) changed or added relevant lines in 4 files are covered.
No unchanged relevant lines lost coverage.
Overall first build on jbr/baasaas at 91.027%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
test/object-store/util/sync/sync_test_utils.cpp	6	23	26.09%
test/object-store/util/sync/baas_admin_api.cpp	187	260	71.92%

Totals
Change from base Build 2121:	91.0%
Covered Lines:	238243
Relevant Lines:	261727

💛 - Coveralls

ironage

Looking really good!

ironage · 2024-03-07T00:20:49Z

test/object-store/util/sync/baas_admin_api.cpp

+    return {};
+}
+
+static std::string unquote_string(std::string_view possibly_quoted_string)


Can we make this a public shared utility function from sync_test_utils.hpp instead of copying it?

ironage · 2024-03-07T00:27:13Z

test/object-store/util/sync/baas_admin_api.cpp

+
+    ~Baasaas()
+    {
+        stop();


Does baasaas have a TTL or something just in case our tests on PRs crash and leave a bunch of containers running?

Yes, they'll get reaped automatically after about 60 minutes.
Ideally, it would be great to guarantee these get cleaned up even if the test process itself dies. Maybe we could have the setup write the container ID to a file that the evergreen task could check for in the "post" task, or something like that. Seems like this should be relatively uncommon though so I think it would be fine to just revisit this later.

I've added a lock file to this type so it will create a file called baasaas_instance.lock in the working directory of the tests and remove it after the instance is stopped. I also added the ability to specify a container ID so the tests can still interact with baasaas to get their base urls and stuff, but won't tear down the container if it was started manually to make this easier to work with the baasaas cli script.

ironage · 2024-03-07T00:32:18Z

test/object-store/util/sync/baas_admin_api.cpp

+    {
+        app::Request request;
+        request.url = util::format(
+            "https://us-east-1.aws.data.mongodb-api.com/app/baas-container-service-autzb/endpoint/%1", api_path);


can you leave a comment about who owns this app and if there is any documentation about the endpoints?

done. I've also made it so we can override this via an env variable.

ironage · 2024-03-07T00:57:43Z

evergreen/config.yml

@@ -977,18 +989,17 @@ tasks:
      test_label: objstore-local
      test_executable_name: "realm-object-store-tests"
      verbose_test_output: true
+      disable_tests_against_baas: true
  - func: "check branch state"

 # These are baas object store tests that run against baas running on a remote host
 - name: baas-integration-tests
  tags: [ "test_suite", "for_pull_requests", "requires_baas" ]


It is a bit confusing to have both the disable_tests_against_baas var and requires_baas. Can we have only one of these for clarity?

Unfortunately no, requires_baas is a tag rather than an expansion which tells evergreen which tasks to run and disable_tests_against_baas is an expansion used in commands to tell them how to compile the object-store tests.

ironage · 2024-03-07T00:59:41Z

evergreen/config.yml

  - func: "check branch state"

 # These are baas object store tests that run against baas running on a remote host
 - name: baas-integration-tests
  tags: [ "test_suite", "for_pull_requests", "requires_baas" ]
  exec_timeout_secs: 3600
  commands:
-  - func: "launch remote baas"


It would be nice to eventually get rid of some of the launch/wait shell scripts, but I guess we'd have to wait to port over the network tests to this as well. Is there a viable path to doing this with baasaas?

Yeah, I was gonna do that in a separate PR.

ironage · 2024-03-12T18:47:22Z

Can you double check this run, it looks like a new kind of failure to me?

[2024/03/12 09:00:36.610] 4: /System/Volumes/Data/data/mci/afc16f9a9d450c759de0ef5d7a93bcbb/realm-core/test/object-store/util/sync/baas_admin_api.cpp:724: [realm-core-14.1.0] Assertion failed: resp.http_status_code >= 200 && resp.http_status_code < 300 with (util::format("url: %1 request: %2, reply: %3", m_url, body.dump(), resp.body)) =  ["url: http://18.212.184.129:32875/api/admin/v3.0/groups/65f07c16b1f6f93c1205954f/apps/65f07c22b1f6f93c120597bd/services/65f07c24b1f6f93c120597ee/config request: {\"sync\":{\"database_name\":\"test_data_c_api_client_reset_tests_65f07c22dbdf61a7bc02b837\",\"is_recovery_mode_disabled\":false,\"partition\":{\"key\":\"realm_id\",\"permissions\":{\"read\":true,\"write\":true},\"required\":false,\"type\":\"string\"},\"state\":\"enabled\"}}, reply: {\"error\":\"failed sync post-commit hook: error initializing stores: error ensuring indexes on \\\"__realm_sync.session_tracker\\\": (DatabaseDropPending) Cannot create collection __realm_sync.session_tracker - database is in the process of being dropped.\"}"]

ironage

LGTM once CI succeeds, or the failures are investigated and appear unrelated.

jbreams · 2024-03-12T18:50:42Z

@mpobrien, I think the failure @ironage is calling out is known? i remember talking to you about failed sync post-commit hook: error initializing stores: error ensuring indexes on \\\"__realm_sync.session_tracker\\\": (DatabaseDropPending) Cannot create collection __realm_sync.session_tracker - database is in the process of being dropped.

mpobrien · 2024-03-12T18:52:01Z

https://jira.mongodb.org/browse/BAAS-28824 should resolve that issue - it's due to the interaction between an optimization to drop the DB when terminating sync (when there's only one app in the DB) and mongodb's restriction that you can't modify collections or indexes when DB drops are in progress.
As a workaround until https://jira.mongodb.org/browse/BAAS-28824 is in, you could just create a second app in the backing DB and leave it there for the duration of the entire test suite. This would prevent the optimization from kicking in, which should eliminate that failure mode.
https://jira.mongodb.org/browse/BAAS-28824 is unfortunately not the quick win I thought it would be as it requires threading a flag down into some hard-to-reach spots, but I will try to prioritize it soon if there's no other viable way around it for now

jbreams · 2024-03-12T20:15:20Z

@ironage and @mpobrien , could I get a quick review on the last commit? I updated the how-to-build.md file to reflect using baasaas and I made it so we create the "default" app on process-launch rather than first use so we'll always have a pbs app around to try to prevent the database is in the process of being dropped problem. I also updated the changelog.

mpobrien · 2024-03-12T20:48:24Z

how-to-build.md


+Due to MongoDB security policies, running baas requires company issued credentials.


I think it might be worth also pointing readers to https://wiki.corp.mongodb.com/display/10GEN/%28Device+Sync%29+Using+Docker+to+run+a+BAAS+server+instance and encouraging that workflow to be used for local development (the container is identical to what you get from launching one on baasaas). Baasaas should be reserved mostly for CI tasks.

ironage · 2024-03-12T20:53:03Z

how-to-build.md

+You can tell the object-store tests to use a specific version of baas with the
+`BAASAAS_START_MODE` environment variable, which can either be `githash`, `patchid`,
+or `branch`. If you specify a start mode, you need to tell it which githash or
+branch name to start with via the `BAASAAS_REF_SPEC` environment variable. Ommitting


Typo: Ommitting -> Omitting

ironage

LGTM

jbreams added 22 commits January 10, 2024 14:02

Start a baasaas instance to run baas integration tests against

42b5cfa

Make redirect tests work against random baas urls

64737cb

Merge remote-tracking branch 'origin/master' into jbr/baasaas

5267fb2

fix merge

74cec93

log baas coid for admin api requests

e41a531

fix schema migration tests

c28d816

allow starting with a patchid

a5381d7

fix merge

b8b2c24

Merge remote-tracking branch 'origin/master' into jbr/baasaas

56c7e43

Merge remote-tracking branch 'origin/master' into jbr/baasaas

845d7b2

make dependencies.list YAML

af8424b

Merge remote-tracking branch 'origin/master' into jbr/baasaas

a6af90f

Merge remote-tracking branch 'origin/master' into jbr/baasaas

65fa191

remove some debugging lines

4a65835

debugging rhel7 crash

2c048cd

temporarily download ca certs package

abd1c15

Merge remote-tracking branch 'origin/master' into jbr/baasaas

2c97874

fix some test failures that were my fault

c8b3b1f

Merge remote-tracking branch 'origin/master' into jbr/baasaas

970e236

Merge remote-tracking branch 'origin/master' into jbr/baasaas

b63cbde

Merge remote-tracking branch 'origin/master' into jbr/baasaas

7378d98

Merge remote-tracking branch 'origin/master' into jbr/baasaas

a54815e

cla-bot bot added the cla: yes label Mar 6, 2024

github-actions bot assigned jbreams Mar 6, 2024

jbreams added 5 commits March 6, 2024 12:20

update baas

95de793

Merge remote-tracking branch 'origin/master' into jbr/baasaas

4f4f406

fix bad merge

208f565

lint

2e96fc3

use correct path for ca bundle

faae632

jbreams changed the title ~~Jbr/baasaas~~ RCORE-1928 Use baasaas for baas integration tests in CI Mar 6, 2024

jbreams marked this pull request as ready for review March 6, 2024 22:02

jbreams requested review from mpobrien and ironage March 6, 2024 22:02

ironage reviewed Mar 7, 2024

View reviewed changes

mpobrien approved these changes Mar 12, 2024

View reviewed changes

fixes from PR

2dfe1db

jbreams requested a review from ironage March 12, 2024 15:56

ironage approved these changes Mar 12, 2024

View reviewed changes

jbreams added 3 commits March 12, 2024 15:36

Merge remote-tracking branch 'origin/master' into jbr/baasaas

1ece73b

changelog and bump baas version

0785694

update docs and force create an app that lives forever

7f55bcf

jbreams requested review from ironage and mpobrien March 12, 2024 20:15

mpobrien reviewed Mar 12, 2024

View reviewed changes

update docs

a0d375b

ironage reviewed Mar 12, 2024

View reviewed changes

ironage approved these changes Mar 12, 2024

View reviewed changes

typo

f7aeff0

jbreams merged commit 666bb25 into master Mar 12, 2024
35 of 37 checks passed

jbreams deleted the jbr/baasaas branch March 12, 2024 23:18

github-actions bot locked as resolved and limited conversation to collaborators Apr 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RCORE-1928 Use baasaas for baas integration tests in CI #7423

RCORE-1928 Use baasaas for baas integration tests in CI #7423

jbreams commented Mar 6, 2024 •

edited

Loading

coveralls-official bot commented Mar 6, 2024 •

edited

Loading

ironage left a comment

ironage Mar 7, 2024

jbreams Mar 12, 2024

ironage Mar 7, 2024

mpobrien Mar 7, 2024

jbreams Mar 12, 2024

ironage Mar 7, 2024

jbreams Mar 12, 2024

ironage Mar 7, 2024

jbreams Mar 12, 2024

ironage Mar 7, 2024

jbreams Mar 12, 2024

ironage commented Mar 12, 2024

ironage left a comment

jbreams commented Mar 12, 2024

mpobrien commented Mar 12, 2024 •

edited

Loading

jbreams commented Mar 12, 2024

mpobrien Mar 12, 2024 •

edited

Loading

ironage Mar 12, 2024

ironage left a comment


		Due to MongoDB security policies, running baas requires company issued credentials.

RCORE-1928 Use baasaas for baas integration tests in CI #7423

RCORE-1928 Use baasaas for baas integration tests in CI #7423

Conversation

jbreams commented Mar 6, 2024 • edited Loading

What, How & Why?

☑️ ToDos

coveralls-official bot commented Mar 6, 2024 • edited Loading

Pull Request Test Coverage Report for Build jonathan.reams_3083

Details

💛 - Coveralls

ironage left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ironage commented Mar 12, 2024

ironage left a comment

Choose a reason for hiding this comment

jbreams commented Mar 12, 2024

mpobrien commented Mar 12, 2024 • edited Loading

jbreams commented Mar 12, 2024

mpobrien Mar 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ironage left a comment

Choose a reason for hiding this comment

jbreams commented Mar 6, 2024 •

edited

Loading

coveralls-official bot commented Mar 6, 2024 •

edited

Loading

mpobrien commented Mar 12, 2024 •

edited

Loading

mpobrien Mar 12, 2024 •

edited

Loading