Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Database Trigger Options #611

Merged
merged 36 commits into from
Nov 1, 2023
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
c3fdc3b
testing table formatting
MongoCaleb Oct 16, 2023
584d75f
sanity check
MongoCaleb Oct 16, 2023
fe5d96b
add recursion warning and fix formatting of table
MongoCaleb Oct 16, 2023
e3c565a
more formatting fun
MongoCaleb Oct 16, 2023
985f758
tabs, spaces, indents, oh my
MongoCaleb Oct 16, 2023
7654e99
more formatting...
MongoCaleb Oct 16, 2023
f194b81
change messaging and hopefully fix the table formatting
MongoCaleb Oct 16, 2023
c1de63f
review 1
MongoCaleb Oct 16, 2023
ac536fc
add new operations and mark db and collection names as optional
MongoCaleb Oct 16, 2023
b4a1d27
add to config reference
MongoCaleb Oct 16, 2023
80dc42a
change database->collection
MongoCaleb Oct 16, 2023
29969a3
review change & fix column width
MongoCaleb Oct 16, 2023
90edce6
make more clear which operations for which trigger types
MongoCaleb Oct 16, 2023
54b8333
clean up note and fix indentation
MongoCaleb Oct 16, 2023
4e1d7b5
Merge branch 'master' into DOCSP-33588
MongoCaleb Oct 16, 2023
21dea41
Merge branch 'master' into DOCSP-33588
MongoCaleb Oct 16, 2023
02f7a1b
testing removal of warnings that act like errors
MongoCaleb Oct 16, 2023
124e590
Merge branch 'DOCSP-33588' of github.com:MongoCaleb/docs-app-services…
MongoCaleb Oct 16, 2023
545c72a
final formatting fix
MongoCaleb Oct 16, 2023
4961239
changes after discussion
MongoCaleb Oct 17, 2023
254fb2b
differentiate deployment types for change events
MongoCaleb Oct 17, 2023
aa52585
remove note and make inline
MongoCaleb Oct 17, 2023
14f6370
adjust table width
MongoCaleb Oct 17, 2023
785e508
merge docsp-33274 / PR #612 to sinmplify
MongoCaleb Oct 17, 2023
b76041b
add note about dedicated tiers; fix lists
MongoCaleb Oct 18, 2023
449a077
Merge branch 'master' into DOCSP-33588
MongoCaleb Oct 18, 2023
740c26d
add missing opertation types
MongoCaleb Oct 18, 2023
6d519eb
Merge branch 'DOCSP-33588' of github.com:MongoCaleb/docs-app-services…
MongoCaleb Oct 18, 2023
e6653e0
merge
MongoCaleb Nov 1, 2023
eecdaf4
Merge branch 'master' into DOCSP-33588
MongoCaleb Nov 1, 2023
239d746
merge
MongoCaleb Nov 1, 2023
a0b53cc
re-incorping pre-image ticket and fixing table headings
MongoCaleb Nov 1, 2023
00b7d91
clarifying note about disabled
MongoCaleb Nov 1, 2023
e7fcc05
merge
MongoCaleb Nov 1, 2023
a04195e
remove extra header
MongoCaleb Nov 1, 2023
4366c18
remove _
MongoCaleb Nov 1, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 3 additions & 6 deletions source/reference/config/triggers.txt
Original file line number Diff line number Diff line change
Expand Up @@ -172,12 +172,9 @@ configuration files:

.. note::

Regardless of this setting:

- ``INSERT`` and ``REPLACE`` events always include the
``fullDocument`` field.

- ``DELETE`` events never include the ``fullDocument`` field.
Regardless of this setting, ``INSERT`` and ``REPLACE`` events always
include the``fullDocument`` field. ``DELETE`` events never include the
``fullDocument`` field.

* - | ``config.full_document_before_change``
| Boolean
Expand Down
218 changes: 147 additions & 71 deletions source/triggers/database-triggers.txt
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,10 @@ Database Triggers
:depth: 2
:class: singlecol

Database Triggers allow you to execute server-side logic whenever a
document is added, updated, or removed in a linked MongoDB Atlas cluster.
Database Triggers allow you to execute server-side logic whenever a database
change occurs on a linked MongoDB Atlas cluster. You can configure triggers on
individual collections, entire databases, and on an entire cluster.

Unlike SQL data triggers, which run on the database server, triggers run
on a serverless compute layer that scales independently of the database
server. Triggers automatically call :ref:`Atlas Functions <functions>`
Expand All @@ -30,18 +32,6 @@ single change stream for each collection with at least one enabled
trigger. If multiple triggers are enabled for a collection they all
share the same change stream.

You control which operations cause a trigger to fire as well as what
happens when it does. For example, you can run a function whenever a
specific field of a document is updated. The function can access the
entire change event, so you always know what changed. You can also pass
the change event to :ref:`AWS EventBridge <aws-eventbridge>` to handle
the event outside of Atlas.

Triggers support :manual:`$match
</reference/operator/aggregation/match>` expressions to filter change
events and :manual:`$project </reference/operator/aggregation/project>`
expressions to limit the data included in each event.

.. important:: Change Stream Limitations

There are limits on the total number of change streams you can open
Expand All @@ -53,6 +43,27 @@ expressions to limit the data included in each event.
<serverless-caveats>` or :ref:`{+adf-instance+}
<data-federation-caveats>` because they do not support change streams.

You control which operations cause a trigger to fire as well as what
happens when it does. For example, you can run a function whenever a
specific field of a document is updated. The function can access the
entire change event, so you always know what changed. You can also pass
the change event to :ref:`AWS EventBridge <aws-eventbridge>` to handle
the event outside of Atlas.

Triggers support :manual:`$match </reference/operator/aggregation/match>`
expressions to filter change events and :manual:`$project </reference/operator/aggregation/project>`
expressions to limit the data included in each event.

.. _trigger-recursion:

.. warning::

It is possible to configure triggers that cause other triggers to fire on the
same collection, database, or cluster, resulting in recursion. Examples
include a database trigger writing to a collection within the same
database, or a cluster-level logger writing database logs to another database
in the same cluster.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we may want to explicitly call out log forwarders as well here. Since, they are a first class resource that our users may also be using that could cause this issue.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't that just a specific case of a process writing to a monitored database and/or collection?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are correct, but just wanted to make customer's slightly more conscious if they didn't make that connection.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Talked offline in the project channel with team and Laura, we would like to go with.

Suggested change
.. warning::
It is possible to configure triggers that cause other triggers to fire on the
same collection, database, or cluster, resulting in recursion. Examples
include a database trigger writing to a collection within the same
database, or a cluster-level logger writing database logs to another database
in the same cluster.
.. warning::
In deployment and database level triggers, it is possible to configure triggers
in a way that causes other triggers to fire, thereby resulting in recursion.
Examples include a database-level trigger writing to a collection within the
same database, or a cluster-level logger or log forwarder writing logs to
another database in the same cluster.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MongoCaleb Could we get an update on this since it should be going in tomorrow.

.. _create-a-database-trigger:

Create a Database Trigger
Expand Down Expand Up @@ -104,6 +115,9 @@ Configuration

Database Triggers have the following configuration options:

Trigger Details
~~~~~~~~~~~~~~~

.. list-table::
:header-rows: 1
:widths: 15 30
Expand Down Expand Up @@ -137,81 +151,144 @@ Database Triggers have the following configuration options:

.. include:: /includes/trigger-event-ordering.rst

Within the :guilabel:`Trigger Source Details` section, the following configuration
options are available:

.. list-table::
:header-rows: 1
:widths: 15 30

* - Field
- Description

* - :guilabel:`Cluster Name`

- The name of the MongoDB Service that the Trigger is
associated with.

* - :guilabel:`Database Name`
Trigger Source Details
~~~~~~~~~~~~~~~~~~~~~~

- The MongoDB database that contains the watched
collection.
Within the :guilabel:`Trigger Source Details` section, you first select the
:guilabel:`Source Type`, based on the level of granularity you want. Your
options are:

* - :guilabel:`Collection Name`
- :guilabel:`Collection`, when a change occurs on a specified collection
- :guilabel:`Database`, when a change occurs on any collection in a
specified database
- :guilabel:`Deployment`, when deployment changes occur on a specified
cluster. If you select the Deployment source type, the following
databases are **not watched** for changes:

- The name of the collection that the Trigger watches for
change events.

* - :guilabel:`Operation Type`
- The admin databases ``admin``, ``local``, and ``config``
- The sync databases ``__realm_sync`` and ``__realm_sync_<app_id>``

Depending on which source type you are using, the additional options differ. The
following table describes these options.

- Checkboxes of :ref:`database operation types <database-event-operation-types>`
that cause the Trigger to fire. Select the operation types you want the
trigger to respond to.
.. list-table::
:header-rows: 1
:widths: 15 30

.. warning::
* - Source Type
- Options

Update operations executed from MongoDB Compass or the MongoDB Atlas
Data Explorer fully replace the previous document. As a result,
update operations from these clients will generate **Replace**
change events rather than **Update** events.
* - :guilabel:`Collection`
- |

* - :guilabel:`Full Document`
- :guilabel:`Cluster Name`. The name of the MongoDB cluster that the
Trigger is associated with.

- :guilabel:`Database Name`. The MongoDB database that contains the watched
collection.

- If enabled, **Update** change events include the latest
:manual:`majority-committed </reference/read-concern-majority/>`
version of the modified document *after* the change was applied in
the ``fullDocument`` field.
- :guilabel:`Collection Name`. The MongoDB collection to watch.

.. note::
- :guilabel:`Operation Type`. The collection operation types
that cause the Trigger to fire. Select the operation types you want the
trigger to respond to. Options include:

- Create Collection
- Modify Collection
- Rename Collection
- Shard Collection
- Drop Collection

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned in our chat, these are actually Database+ change event types.


.. note::

Update operations executed from MongoDB Compass or the MongoDB Atlas
Data Explorer fully replace the previous document. As a result,
update operations from these clients will generate **Replace**
change events rather than **Update** events.

- :guilabel:`Full Document`. If enabled, **Update** change events include
the latest :manual:`majority-committed </reference/read-concern-majority/>`
version of the modified document *after* the change was applied in
the ``fullDocument`` field.

.. note::

Regardless of this setting, **Insert** and **Replace** events always
include the ``fullDocument`` field. **Delete** events never include
the ``fullDocument`` field.

- :guilabel:`Document Preimage`. Disabled for Collection and Database
sources to limit unnecessary watches on the cluster for a new
collection being created.

Regardless of this setting:

- **Insert** and **Replace** events always include the
``fullDocument`` field.

- **Delete** events never include the ``fullDocument`` field.
* - :guilabel:`Database`
- |

* - :guilabel:`Document Preimage`
- :guilabel:`Cluster Name`. The name of the MongoDB cluster that the
Trigger is associated with.

- :guilabel:`Database Name`. The MongoDB database to watch.

- :guilabel:`Operation Type`. The :ref:`database operation types
<database-event-operation-types>` that cause the Trigger to fire.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Q] Only this Database row has a link, do we want to have a link for Collection and Deployment as well?

Select the operation types you want the trigger to respond to.
Options include:

- Create Collection
- Modify Collection
- Rename Collection
- Shard Collection
- Drop Collection

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing the 2 new sharding types


.. note::

Update operations executed from MongoDB Compass or the MongoDB Atlas
Data Explorer fully replace the previous document. As a result,
update operations from these clients will generate **Replace**
change events rather than **Update** events.

- :guilabel:`Full Document`. If enabled, **Update** change events include
the latest :manual:`majority-committed </reference/read-concern-majority/>`
version of the modified document *after* the change was applied in
the ``fullDocument`` field.

.. note::

Regardless of this setting, **Insert** and **Replace** events always
include the ``fullDocument`` field. **Delete** events never include
the ``fullDocument`` field.

- :guilabel:`Document Preimage`. Disabled for Collection and Database
sources to limit unnecessary watches on the cluster for a new
collection being created.

- If enabled, change events include a copy of the modified document
from immediately *before* the change was applied in the
``fullDocumentBeforeChange`` field. All change events except for
**Insert** events include the document preimage. Document preimages are
supported on non-sharded Atlas clusters running MongoDB 4.4+, and
on sharded Atlas clusters running MongoDB 5.3 and later.
* - :guilabel:`Deployment`
- |

.. tip:: Performance Optimization
- :guilabel:`Cluster Name`. The name of the MongoDB cluster that the
Trigger is associated with.

- :guilabel:`Operation Type`. The operation types
that occur in the cluster that cause the Trigger to fire. Select the
operation types you want the trigger to respond to. Options include:

- Drop Database
- Create Collection
- Modify Collection
- Rename Collection
- Shard Collection
- Drop Collection

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only Drop Database is a deployment only operation type. All the rest of these are database operation types.

[Q] Also do we want to add this list to each of them (COLLECTION, DATABASE, DEPLOYMENT)?


Preimages require additional storage overhead that may affect
performance. If you're not using preimages on a collection,
you should disable preimages. To learn more, see :ref:`Disable
Collection-Level Preimages
<database-triggers-disable-collection-level-preimages>`.
Function
~~~~~~~~

Within the :guilabel:`Function` section, you choose what action is taken when
the trigger fires. You can choose to run a :ref:`function <functions>` or use
:ref:`AWS EventBridge <aws-eventbridge>`.

Advanced
~~~~~~~~

Within the :guilabel:`Advanced` section, the following **optional** configuration
options are available:

Expand Down Expand Up @@ -287,7 +364,6 @@ following operation types:
* - **Replace**
- Represents a new document that replaced a document in the collection.


Database change event objects have the following general form:

.. code-block:: json
Expand Down
Loading