Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix too-long lines; update architecture diagrams #271

Merged
merged 2 commits into from
Oct 9, 2018
Merged

Fix too-long lines; update architecture diagrams #271

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5047a03
Fixes for indentation and too-long lines
Oct 9, 2018
7ed9ab1
Update architecture diagrams; add alt text
Oct 9, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
125 changes: 75 additions & 50 deletions developing/architecture.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,16 +19,20 @@ additional worker hosts and by starting a greater number of pods on each host.

This is how the world interacts with a Deconst cluster:

.. image:: /_images/deconst-external.png
.. image:: /_images/deconst-architecture.png
:alt: A diagram of the overall Deconst architecture

None of the service containers store any internal, persistent state: the sources
of truth for all Deconst state are Cloud Files containers, MongoDB collections,
or GitHub repositories. This means that you can adaptively destroy or launch
Deconst worker hosts without fear of losing information.
None of the service containers store any internal, persistent state:
the sources of truth for all Deconst state are Cloud Files containers,
MongoDB collections, or GitHub repositories. This means that you can
adaptively destroy or launch Deconst worker hosts without fear of
losing information.

Each pod includes the following arrangement of interlinked service containers:
Each pod includes the following arrangement of interlinked service
containers:

.. image:: /_images/deconst-internal.png
:alt: A diagram of interlinked services

On the build host, a dedicated `Strider CD
<https://github.com/Strider-CD/strider>`_ continuous integration server manages
Expand All @@ -37,6 +41,7 @@ service containers that act as a :ref:`staging environment <staging>` that can
be used to preview content in context before it's merged and shipped.

.. image:: /_images/deconst-build.png
:alt: A diagram of the Deconst build host

Access to Strider is managed by membership in a GitHub organization or in teams
within an organization, as configured in the instance's credentials file.
Expand Down Expand Up @@ -82,6 +87,10 @@ Components
envelope into an appropriate :term:`template` and send the final HTML back
in an HTTP response.

.. image:: ../_images/deconst-presenter.png
:scale: 90 %
:alt: A diagram of the Deconst presenter service

nginx
Reverse proxy that accepts requests from off of the host, terminates TLS,
and delegates to the local :term:`presenter` and :term:`content service`.
Expand All @@ -105,17 +114,19 @@ When a content consumer initiates an HTTPS request:
to discover the :term:`content ID` of the content that should be rendered at
that path.

#. Next, the presenter queries the :term:`content service` to acquire the
content for that ID. The content service locates the appropriate :term:`metadata
envelope`, all site-wide assets, and performs any necessary post-processing.
#. Next, the presenter queries the :term:`content service` to acquire
the content for that ID. The content service locates the
appropriate :term:`metadata envelope`, all site-wide assets, and
performs any necessary post-processing.

#. If any :term:`addenda` are requested by the current envelope, each addenda
envelope is fetched from the content service.

#. The presenter locates the Nunjucks :term:`template` that should be used to
decorate the raw content based on a regular expression match on the presented
URL. If no template is routed, this request is skipped and a null layout (that
renders the envelope's body directly) is used.
#. The presenter locates the Nunjucks :term:`template` that should be
used to decorate the raw content based on a regular expression
match on the presented URL. If no template is routed, this request
is skipped and a null layout (that renders the envelope's body
directly) is used.

#. The presenter renders the metadata envelope using the layout. The resulting
HTML document is returned to the user.
Expand All @@ -126,54 +137,68 @@ Lifecycle of a Control Repository Update

When a change is merged into the live branch of the :term:`control repository`:

#. A Strider build executes the asset :term:`preparer` on the latest commit of
the repository. Stylesheets, javascript, images, and fonts found within the
``assets`` directory are compiled, concatenated, minified, and submitted to the
:term:`content service` to be fingerprinted, stored on the CDN-enabled asset
container, and made available as global assets to all metadata envelopes.
#. A Strider build executes the asset :term:`preparer` on the latest
commit of the repository. Stylesheets, javascript, images, and
fonts found within the ``assets`` directory are compiled,
concatenated, minified, and submitted to the :term:`content service`
to be fingerprinted, stored on the CDN-enabled asset container,
and made available as global assets to all metadata envelopes.

#. Once all assets have been published, the preparer sends the latest
git commit SHA of the control repository to the
:term:`content service`, where it's stored in MongoDB.

#. Once all assets have been published, the preparer sends the latest git commit
SHA of the control repository to the :term:`content service`, where it's stored
in MongoDB.
#. Each entry within the ``content-repositories.json`` file is checked
against the list of :term:`strider` builds. If any new entries have
been added, a content build is created and configured with a newly
issued API key.

#. Each entry within the ``content-repositories.json`` file is checked against
the list of :term:`strider` builds. If any new entries have been added, a
content build is created and configured with a newly issued API key.
#. During each request, each :term:`presenter` queries its linked
:term:`content service` for the active control repository SHA. If
it doesn't match last-loaded control repository SHA, the presenter
triggers an asynchronous update.

#. During each request, each :term:`presenter` queries its linked :term:`content
service` for the active control repository SHA. If it doesn't match last-loaded
control repository SHA, the presenter triggers an asynchronous update.
#. If successful, the new content and template mappings, redirects,
and templates are atomically installed. Otherwise, the presenter
logs an error with the details and waits for further changes before
attempting to reload.

#. If successful, the new content and template mappings, redirects, and
templates are atomically installed. Otherwise, the presenter logs an error with
the details and waits for further changes before attempting to reload.

Lifecycle of a Content Repository Update
----------------------------------------

When a change is merged into the live branch of a :term:`content repository`:
.. image:: /_images/content-repo-update-lifecycle.png
:alt: A diagram of the content repository update lifecycle

#. A Strider build scans the latest commit of the repository for directories
containing ``_deconst.json`` files and executes the appropriate :term:`preparer`
within a Docker container that's given each context.

#. The preparer copies each referenced asset to an asset output directory within
the shared workspace container. The offset of the asset reference is saved in an
"asset_offsets" map.
When a change is merged into the live branch of a :term:`content
repository`:

#. The preparer generates a :term:`metadata envelope` for each page that would
be rendered, assigns it a :term:`content ID` using a configured base ID, and
writes it to the envelope output directory.
#. A Strider build scans the latest commit of the repository for
directories containing ``_deconst.json`` files and executes the
appropriate :term:`preparer` within a Docker container that's given
each context.

#. The submitter queries the :term:`content service` with the SHA-256
fingerprints of each asset in the asset directory. If any assets are missing or
have changed, the submitter bulk-uploads them to the :term:`content service`
API. If more than 30MB of assets need to be uploaded, assets are uploaded in
batches of just over 30MB to avoid overwhelming the upload process.
#. The preparer copies each referenced asset to an asset output
directory within the shared workspace container. The offset of the
asset reference is saved in an "asset_offsets" map.

#. The submitter inserts the public CDN URLs of each asset into the body of each
metadata envelope at the recorded offsets and removes the "asset_offsets" key.
#. The preparer generates a :term:`metadata envelope` for each page
that would be rendered, assigns it a :term:`content ID` using a
configured base ID, and writes it to the envelope output directory.

#. The submitter queries the content service with the SHA-256 fingerprint of a
stable (key-sorted) representation of each envelope. Any envelopes that have
been changed are bulk-uploaded to the content service.
#. The submitter queries the :term:`content service` with the SHA-256
fingerprints of each asset in the asset directory. If any assets
are missing or have changed, the submitter bulk-uploads them to the
:term:`content service` API. If more than 30MB of assets need to be
uploaded, assets are uploaded in batches of just over 30MB to avoid
overwhelming the upload process.

#. The submitter inserts the public CDN URLs of each asset into the
body of each metadata envelope at the recorded offsets and removes
the "asset_offsets" key.

#. The submitter queries the content service with the SHA-256
fingerprint of a stable (key-sorted) representation of each
envelope. Any envelopes that have been changed are bulk-uploaded to
the content service.
19 changes: 10 additions & 9 deletions developing/envelope.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,13 @@
Metadata Envelope Schema
========================

Much of the deconst system involves the manipulation of :term:`metadata
envelopes`, the JSON documents produced by each :term:`preparer` that contain
the actual content to render. To be presented properly, envelopes must adhere to
a common schema.
Much of the deconst system involves the manipulation of
:term:`metadata envelopes`, the JSON documents produced by each
:term:`preparer` that contain the actual content to render. To be
presented properly, envelopes must adhere to a common schema.

This is an example envelope that demonstrates the full document structure,
including all optional fields:
This is an example envelope that demonstrates the full document
structure, including all optional fields:

.. code-block:: json

Expand Down Expand Up @@ -137,9 +137,10 @@ including all optional fields:
character offsets into ``body`` that should be replaced by the full, public
URL of the asset.

The documents retrieved from the content store consist of the requested envelope
and a number of additional attributes that are derived and injected at retrieval
time. The full content document looks like this:
The documents retrieved from the content store consist of the
requested envelope and a number of additional attributes that are
derived and injected at retrieval time. The full content document
looks like this:

.. code-block:: json

Expand Down
12 changes: 6 additions & 6 deletions developing/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ like "content IDs" and "presented URLs".

.. toctree::

setup
terminology
architecture
preparer
staging
envelope
setup
terminology
architecture
preparer
staging
envelope
39 changes: 20 additions & 19 deletions developing/preparer.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,23 +6,24 @@ Writing a Preparer
If you want to include content from a new :term:`content repository` format,
you'll need to create a new :term:`preparer`. Generally, a preparer needs to:

#. Parse the markup language, configuration files, and other metadata for some
content format. When possible, you should use the format's native libraries and
tooling to do so.
#. Parse the markup language, configuration files, and other metadata
for some content format. When possible, you should use the format's
native libraries and tooling to do so.

#. Parse the ``_deconst.json`` file. Consult the :ref:`new content repository
section <adding-new-content-repository>` for its schema.
#. Parse the ``_deconst.json`` file. Consult the :ref:`new content
repository section <adding-new-content-repository>` for its schema.

#. Copy assets (usually images) to the directory specified by the environment
variable ``ASSET_DIR``. It's best to preserve as much of the local directory
structure as possible from the source repository, unless two assets in different
subdirectories have the same filename.
#. Copy assets (usually images) to the directory specified by the
environment variable ``ASSET_DIR``. It's best to preserve as much
of the local directory structure as possible from the source
repository, unless two assets in different subdirectories have the
same filename.

#. Use the markup to produce rendered HTML. The preparer should use a
single-character placeholder for each asset URL. As it does so, it should
generate a map that associates the path of each asset relative to ``ASSET_DIR``
to a collection of character offsets within the body text at which that asset is
referenced.
single-character placeholder for each asset URL. As it does so, it
should generate a map that associates the path of each asset
relative to ``ASSET_DIR`` to a collection of character offsets
within the body text at which that asset is referenced.

As a rule, the rendered HTML *should omit any layouts* from the content
repository itself and only render the page content, unadorned. In Deconst,
Expand Down Expand Up @@ -56,14 +57,14 @@ Deconst preparer containers should respect the following configuration values:
* ``ENVELOPE_DIR``: The preparer must write completed envelopes to this
directory.

* ``CONTENT_ID_BASE``: *(optional)* If set, this should *override* the content
ID base specified in ``_deconst.json`` for this preparation run, preferably with
some kind of message if they differ.
* ``CONTENT_ID_BASE``: *(optional)* If set, this should *override* the
content ID base specified in ``_deconst.json`` for this preparation
run, preferably with some kind of message if they differ.

* ``CONTENT_ROOT``: *(optional)* If specified, the preparer should prepare
content mounted to a volume at this path within the container. Otherwise, it
should default to preparing ``/usr/content-repo``.

When run with no arguments, the preparer container should prepare the content as
described above, then exit with an exit status of 0 if preparation was
successful, or nonzero if it was not.
When run with no arguments, the preparer container should prepare the
content as described above, then exit with an exit status of 0 if
preparation was successful, or nonzero if it was not.
53 changes: 28 additions & 25 deletions developing/setup.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,13 +34,14 @@ to shave the yak for Docker itself.
Git commit (server): 7c8fca2
OS/Arch (server): linux/amd64

#. We also use Docker Compose to orchestrate small numbers of local containers
to make development more convenient. Follow the `installation guide for Docker
Compose <https://docs.docker.com/compose/install/>`_.
#. We also use Docker Compose to orchestrate small numbers of local
containers to make development more convenient. Follow the
`installation guide for Docker Compose
<https://docs.docker.com/compose/install/>`_.

#. To contribute, you'll also need a reasonable `git <https://git-scm.com/>`_
client. It's likely that you already have one: open a terminal and type ``git
version`` to check.
#. To contribute, you'll also need a reasonable `git
<https://git-scm.com/>`_ client. It's likely that you already have
one: open a terminal and type ``git version`` to check.

Individual Service Development
------------------------------
Expand All @@ -56,12 +57,13 @@ clone:

docker-compose up

Compose will launch a container for the service you're focusing on right now, as
well as any upstream services or infrastructure that it depends on, and link
them all together correctly. You'll see the combined logs for all containers on
your terminal. As you edit source code in your editor of choice, the service
within the container will automatically reload with your changes, so you can
explore the effects live.
Compose will launch a container for the service you're focusing on
right now, as well as any upstream services or infrastructure that it
depends on, and link them all together correctly. You'll see the
combined logs for all containers on your terminal. As you edit source
code in your editor of choice, the service within the container will
automatically reload with your changes, so you can explore the effects
live.

.. note::

Expand All @@ -72,22 +74,23 @@ explore the effects live.
installed docker. For example, if you're using ``docker-machine``, running
``docker-machine ip dev`` will show you the IP.

Although your local source changes will take effect immediately, you may need to
periodically fetch newer versions of upstream containers, as development
progresses on the other parts of the system. To ensure that you have the latest
builds of each container, run ``docker-compose pull``. Also, if you need to
change the service's dependencies, you may need to rebuild your working
container with ``docker-compose build``.
Although your local source changes will take effect immediately, you
may need to periodically fetch newer versions of upstream containers,
as development progresses on the other parts of the system. To ensure
that you have the latest builds of each container, run
``docker-compose pull``. Also, if you need to change the service's
dependencies, you may need to rebuild your working container with
``docker-compose build``.

Compose can also be used to launch its containers in the background (with
``docker-compose up -d``), explore logs for individual containers rather than
aggregated, or run one-off processes in the context of any service container.
Consult the `compose documentation <https://docs.docker.com/compose/cli/>`_ to
see all of your options.

Each service's unit tests can also be executed within a Docker container for
convenience. As a convention, the following script will launch the container and
run all tests:
Each service's unit tests can also be executed within a Docker
container for convenience. As a convention, the following script will
launch the container and run all tests:

.. code-block:: bash

Expand All @@ -96,10 +99,10 @@ run all tests:
Integration Testing
-------------------

To verify that the entire Deconst system works together, use the **integrated**
repository. "Integrated" contains a compose file that executes a single "pod" of
related deconst services on your local host, so you can test all of the services
together.
To verify that the entire Deconst system works together, use the
**integrated** repository. "Integrated" contains a compose file that
executes a single "pod" of related deconst services on your local
host, so you can test all of the services together.

Clone the deconst/integrated repository and run ``script/up`` to begin:

Expand Down
Loading