diff --git a/developing/architecture.rst b/developing/architecture.rst index f4c39e2..5992487 100644 --- a/developing/architecture.rst +++ b/developing/architecture.rst @@ -19,16 +19,20 @@ additional worker hosts and by starting a greater number of pods on each host. This is how the world interacts with a Deconst cluster: -.. image:: /_images/deconst-external.png +.. image:: /_images/deconst-architecture.png + :alt: A diagram of the overall Deconst architecture -None of the service containers store any internal, persistent state: the sources -of truth for all Deconst state are Cloud Files containers, MongoDB collections, -or GitHub repositories. This means that you can adaptively destroy or launch -Deconst worker hosts without fear of losing information. +None of the service containers store any internal, persistent state: +the sources of truth for all Deconst state are Cloud Files containers, +MongoDB collections, or GitHub repositories. This means that you can +adaptively destroy or launch Deconst worker hosts without fear of +losing information. -Each pod includes the following arrangement of interlinked service containers: +Each pod includes the following arrangement of interlinked service +containers: .. image:: /_images/deconst-internal.png + :alt: A diagram of interlinked services On the build host, a dedicated `Strider CD `_ continuous integration server manages @@ -37,6 +41,7 @@ service containers that act as a :ref:`staging environment ` that can be used to preview content in context before it's merged and shipped. .. image:: /_images/deconst-build.png + :alt: A diagram of the Deconst build host Access to Strider is managed by membership in a GitHub organization or in teams within an organization, as configured in the instance's credentials file. @@ -82,6 +87,10 @@ Components envelope into an appropriate :term:`template` and send the final HTML back in an HTTP response. + .. image:: ../_images/deconst-presenter.png + :scale: 90 % + :alt: A diagram of the Deconst presenter service + nginx Reverse proxy that accepts requests from off of the host, terminates TLS, and delegates to the local :term:`presenter` and :term:`content service`. @@ -105,17 +114,19 @@ When a content consumer initiates an HTTPS request: to discover the :term:`content ID` of the content that should be rendered at that path. -#. Next, the presenter queries the :term:`content service` to acquire the - content for that ID. The content service locates the appropriate :term:`metadata - envelope`, all site-wide assets, and performs any necessary post-processing. +#. Next, the presenter queries the :term:`content service` to acquire + the content for that ID. The content service locates the + appropriate :term:`metadata envelope`, all site-wide assets, and + performs any necessary post-processing. #. If any :term:`addenda` are requested by the current envelope, each addenda envelope is fetched from the content service. -#. The presenter locates the Nunjucks :term:`template` that should be used to - decorate the raw content based on a regular expression match on the presented - URL. If no template is routed, this request is skipped and a null layout (that - renders the envelope's body directly) is used. +#. The presenter locates the Nunjucks :term:`template` that should be + used to decorate the raw content based on a regular expression + match on the presented URL. If no template is routed, this request + is skipped and a null layout (that renders the envelope's body + directly) is used. #. The presenter renders the metadata envelope using the layout. The resulting HTML document is returned to the user. @@ -126,54 +137,68 @@ Lifecycle of a Control Repository Update When a change is merged into the live branch of the :term:`control repository`: -#. A Strider build executes the asset :term:`preparer` on the latest commit of - the repository. Stylesheets, javascript, images, and fonts found within the - ``assets`` directory are compiled, concatenated, minified, and submitted to the - :term:`content service` to be fingerprinted, stored on the CDN-enabled asset - container, and made available as global assets to all metadata envelopes. +#. A Strider build executes the asset :term:`preparer` on the latest + commit of the repository. Stylesheets, javascript, images, and + fonts found within the ``assets`` directory are compiled, + concatenated, minified, and submitted to the :term:`content service` + to be fingerprinted, stored on the CDN-enabled asset container, + and made available as global assets to all metadata envelopes. + +#. Once all assets have been published, the preparer sends the latest + git commit SHA of the control repository to the + :term:`content service`, where it's stored in MongoDB. -#. Once all assets have been published, the preparer sends the latest git commit - SHA of the control repository to the :term:`content service`, where it's stored - in MongoDB. +#. Each entry within the ``content-repositories.json`` file is checked + against the list of :term:`strider` builds. If any new entries have + been added, a content build is created and configured with a newly + issued API key. -#. Each entry within the ``content-repositories.json`` file is checked against - the list of :term:`strider` builds. If any new entries have been added, a - content build is created and configured with a newly issued API key. +#. During each request, each :term:`presenter` queries its linked + :term:`content service` for the active control repository SHA. If + it doesn't match last-loaded control repository SHA, the presenter + triggers an asynchronous update. -#. During each request, each :term:`presenter` queries its linked :term:`content - service` for the active control repository SHA. If it doesn't match last-loaded - control repository SHA, the presenter triggers an asynchronous update. +#. If successful, the new content and template mappings, redirects, + and templates are atomically installed. Otherwise, the presenter + logs an error with the details and waits for further changes before + attempting to reload. -#. If successful, the new content and template mappings, redirects, and - templates are atomically installed. Otherwise, the presenter logs an error with - the details and waits for further changes before attempting to reload. Lifecycle of a Content Repository Update ---------------------------------------- -When a change is merged into the live branch of a :term:`content repository`: +.. image:: /_images/content-repo-update-lifecycle.png + :alt: A diagram of the content repository update lifecycle -#. A Strider build scans the latest commit of the repository for directories - containing ``_deconst.json`` files and executes the appropriate :term:`preparer` - within a Docker container that's given each context. -#. The preparer copies each referenced asset to an asset output directory within - the shared workspace container. The offset of the asset reference is saved in an - "asset_offsets" map. +When a change is merged into the live branch of a :term:`content +repository`: -#. The preparer generates a :term:`metadata envelope` for each page that would - be rendered, assigns it a :term:`content ID` using a configured base ID, and - writes it to the envelope output directory. +#. A Strider build scans the latest commit of the repository for + directories containing ``_deconst.json`` files and executes the + appropriate :term:`preparer` within a Docker container that's given + each context. -#. The submitter queries the :term:`content service` with the SHA-256 - fingerprints of each asset in the asset directory. If any assets are missing or - have changed, the submitter bulk-uploads them to the :term:`content service` - API. If more than 30MB of assets need to be uploaded, assets are uploaded in - batches of just over 30MB to avoid overwhelming the upload process. +#. The preparer copies each referenced asset to an asset output + directory within the shared workspace container. The offset of the + asset reference is saved in an "asset_offsets" map. -#. The submitter inserts the public CDN URLs of each asset into the body of each - metadata envelope at the recorded offsets and removes the "asset_offsets" key. +#. The preparer generates a :term:`metadata envelope` for each page + that would be rendered, assigns it a :term:`content ID` using a + configured base ID, and writes it to the envelope output directory. -#. The submitter queries the content service with the SHA-256 fingerprint of a - stable (key-sorted) representation of each envelope. Any envelopes that have - been changed are bulk-uploaded to the content service. +#. The submitter queries the :term:`content service` with the SHA-256 + fingerprints of each asset in the asset directory. If any assets + are missing or have changed, the submitter bulk-uploads them to the + :term:`content service` API. If more than 30MB of assets need to be + uploaded, assets are uploaded in batches of just over 30MB to avoid + overwhelming the upload process. + +#. The submitter inserts the public CDN URLs of each asset into the + body of each metadata envelope at the recorded offsets and removes + the "asset_offsets" key. + +#. The submitter queries the content service with the SHA-256 + fingerprint of a stable (key-sorted) representation of each + envelope. Any envelopes that have been changed are bulk-uploaded to + the content service. diff --git a/developing/envelope.rst b/developing/envelope.rst index 7e21cb1..3521742 100644 --- a/developing/envelope.rst +++ b/developing/envelope.rst @@ -3,13 +3,13 @@ Metadata Envelope Schema ======================== -Much of the deconst system involves the manipulation of :term:`metadata -envelopes`, the JSON documents produced by each :term:`preparer` that contain -the actual content to render. To be presented properly, envelopes must adhere to -a common schema. +Much of the deconst system involves the manipulation of +:term:`metadata envelopes`, the JSON documents produced by each +:term:`preparer` that contain the actual content to render. To be +presented properly, envelopes must adhere to a common schema. -This is an example envelope that demonstrates the full document structure, -including all optional fields: +This is an example envelope that demonstrates the full document +structure, including all optional fields: .. code-block:: json @@ -137,9 +137,10 @@ including all optional fields: character offsets into ``body`` that should be replaced by the full, public URL of the asset. -The documents retrieved from the content store consist of the requested envelope -and a number of additional attributes that are derived and injected at retrieval -time. The full content document looks like this: +The documents retrieved from the content store consist of the +requested envelope and a number of additional attributes that are +derived and injected at retrieval time. The full content document +looks like this: .. code-block:: json diff --git a/developing/index.rst b/developing/index.rst index ad64169..5bd6966 100644 --- a/developing/index.rst +++ b/developing/index.rst @@ -8,9 +8,9 @@ like "content IDs" and "presented URLs". .. toctree:: - setup - terminology - architecture - preparer - staging - envelope + setup + terminology + architecture + preparer + staging + envelope diff --git a/developing/preparer.rst b/developing/preparer.rst index fbdcb4c..2e7a14e 100644 --- a/developing/preparer.rst +++ b/developing/preparer.rst @@ -6,23 +6,24 @@ Writing a Preparer If you want to include content from a new :term:`content repository` format, you'll need to create a new :term:`preparer`. Generally, a preparer needs to: -#. Parse the markup language, configuration files, and other metadata for some - content format. When possible, you should use the format's native libraries and - tooling to do so. +#. Parse the markup language, configuration files, and other metadata + for some content format. When possible, you should use the format's + native libraries and tooling to do so. -#. Parse the ``_deconst.json`` file. Consult the :ref:`new content repository - section ` for its schema. +#. Parse the ``_deconst.json`` file. Consult the :ref:`new content + repository section ` for its schema. -#. Copy assets (usually images) to the directory specified by the environment - variable ``ASSET_DIR``. It's best to preserve as much of the local directory - structure as possible from the source repository, unless two assets in different - subdirectories have the same filename. +#. Copy assets (usually images) to the directory specified by the + environment variable ``ASSET_DIR``. It's best to preserve as much + of the local directory structure as possible from the source + repository, unless two assets in different subdirectories have the + same filename. #. Use the markup to produce rendered HTML. The preparer should use a - single-character placeholder for each asset URL. As it does so, it should - generate a map that associates the path of each asset relative to ``ASSET_DIR`` - to a collection of character offsets within the body text at which that asset is - referenced. + single-character placeholder for each asset URL. As it does so, it + should generate a map that associates the path of each asset + relative to ``ASSET_DIR`` to a collection of character offsets + within the body text at which that asset is referenced. As a rule, the rendered HTML *should omit any layouts* from the content repository itself and only render the page content, unadorned. In Deconst, @@ -56,14 +57,14 @@ Deconst preparer containers should respect the following configuration values: * ``ENVELOPE_DIR``: The preparer must write completed envelopes to this directory. -* ``CONTENT_ID_BASE``: *(optional)* If set, this should *override* the content - ID base specified in ``_deconst.json`` for this preparation run, preferably with - some kind of message if they differ. +* ``CONTENT_ID_BASE``: *(optional)* If set, this should *override* the + content ID base specified in ``_deconst.json`` for this preparation + run, preferably with some kind of message if they differ. * ``CONTENT_ROOT``: *(optional)* If specified, the preparer should prepare content mounted to a volume at this path within the container. Otherwise, it should default to preparing ``/usr/content-repo``. -When run with no arguments, the preparer container should prepare the content as -described above, then exit with an exit status of 0 if preparation was -successful, or nonzero if it was not. +When run with no arguments, the preparer container should prepare the +content as described above, then exit with an exit status of 0 if +preparation was successful, or nonzero if it was not. diff --git a/developing/setup.rst b/developing/setup.rst index 243fe3c..18762fb 100644 --- a/developing/setup.rst +++ b/developing/setup.rst @@ -34,13 +34,14 @@ to shave the yak for Docker itself. Git commit (server): 7c8fca2 OS/Arch (server): linux/amd64 -#. We also use Docker Compose to orchestrate small numbers of local containers - to make development more convenient. Follow the `installation guide for Docker - Compose `_. +#. We also use Docker Compose to orchestrate small numbers of local + containers to make development more convenient. Follow the + `installation guide for Docker Compose + `_. -#. To contribute, you'll also need a reasonable `git `_ - client. It's likely that you already have one: open a terminal and type ``git - version`` to check. +#. To contribute, you'll also need a reasonable `git + `_ client. It's likely that you already have + one: open a terminal and type ``git version`` to check. Individual Service Development ------------------------------ @@ -56,12 +57,13 @@ clone: docker-compose up -Compose will launch a container for the service you're focusing on right now, as -well as any upstream services or infrastructure that it depends on, and link -them all together correctly. You'll see the combined logs for all containers on -your terminal. As you edit source code in your editor of choice, the service -within the container will automatically reload with your changes, so you can -explore the effects live. +Compose will launch a container for the service you're focusing on +right now, as well as any upstream services or infrastructure that it +depends on, and link them all together correctly. You'll see the +combined logs for all containers on your terminal. As you edit source +code in your editor of choice, the service within the container will +automatically reload with your changes, so you can explore the effects +live. .. note:: @@ -72,12 +74,13 @@ explore the effects live. installed docker. For example, if you're using ``docker-machine``, running ``docker-machine ip dev`` will show you the IP. -Although your local source changes will take effect immediately, you may need to -periodically fetch newer versions of upstream containers, as development -progresses on the other parts of the system. To ensure that you have the latest -builds of each container, run ``docker-compose pull``. Also, if you need to -change the service's dependencies, you may need to rebuild your working -container with ``docker-compose build``. +Although your local source changes will take effect immediately, you +may need to periodically fetch newer versions of upstream containers, +as development progresses on the other parts of the system. To ensure +that you have the latest builds of each container, run +``docker-compose pull``. Also, if you need to change the service's +dependencies, you may need to rebuild your working container with +``docker-compose build``. Compose can also be used to launch its containers in the background (with ``docker-compose up -d``), explore logs for individual containers rather than @@ -85,9 +88,9 @@ aggregated, or run one-off processes in the context of any service container. Consult the `compose documentation `_ to see all of your options. -Each service's unit tests can also be executed within a Docker container for -convenience. As a convention, the following script will launch the container and -run all tests: +Each service's unit tests can also be executed within a Docker +container for convenience. As a convention, the following script will +launch the container and run all tests: .. code-block:: bash @@ -96,10 +99,10 @@ run all tests: Integration Testing ------------------- -To verify that the entire Deconst system works together, use the **integrated** -repository. "Integrated" contains a compose file that executes a single "pod" of -related deconst services on your local host, so you can test all of the services -together. +To verify that the entire Deconst system works together, use the +**integrated** repository. "Integrated" contains a compose file that +executes a single "pod" of related deconst services on your local +host, so you can test all of the services together. Clone the deconst/integrated repository and run ``script/up`` to begin: diff --git a/developing/staging.rst b/developing/staging.rst index b441a8f..810b917 100644 --- a/developing/staging.rst +++ b/developing/staging.rst @@ -19,33 +19,39 @@ To submit content to the staging :term:`content service`, run the normal #. Set ``CONTENT_SERVICE_URL`` to the staging environment's API endpoint. -#. Prepend a revision ID as the first URL path segment of your :term:`content - ID` base. For example, if your content repository's normal content ID base is - ``https://github.com/deconst/deconst-docs/``, set ``CONTENT_ID_BASE`` to +#. Prepend a revision ID as the first URL path segment of your + :term:`content ID` base. For example, if your content repository's + normal content ID base is + ``https://github.com/deconst/deconst-docs/``, set + ``CONTENT_ID_BASE`` to ``https://github.com/build-abcdef/deconst/deconst-docs/`` instead. -The revision ID is arbitrary, but it should be chosen to be relatively unique -among everyone who's staging changes so you don't overwrite one another's staged -content by mistake. Good examples include something containing part of your -current git SHA, your username, or a timestamp of some kind. +The revision ID is arbitrary, but it should be chosen to be relatively +unique among everyone who's staging changes so you don't overwrite one +another's staged content by mistake. Good examples include something +containing part of your current git SHA, your username, or a timestamp +of some kind. -To amend an existing revision's content, re-run the preparer with the same -revision ID. To append content from a different content repository to the same -staging environment, run its preparer with the revision ID. +To amend an existing revision's content, re-run the preparer with the +same revision ID. To append content from a different content +repository to the same staging environment, run its preparer with the +revision ID. Viewing Staged Content ---------------------- To see the content that you've just staged, visit the staging -:term:`presenter`'s address and prepend your revision ID to the URL path. For -example, if you just built content that's normally mapped to the path ``/docs/`` -to a staging server that's available at ``https://staging.example.com/`` with -the revision id ``user-smashwilson``, your staged content will be visible at +:term:`presenter`'s address and prepend your revision ID to the URL +path. For example, if you just built content that's normally mapped to +the path ``/docs/`` to a staging server that's available at +``https://staging.example.com/`` with the revision id +``user-smashwilson``, your staged content will be visible at ``https://staging.example.com/user-smashwilson/docs/``. The rest of the site will be *also* be visible beneath the parent -``/user-smashwilson/`` path exactly as it appears on the current production -site. Any links on any rendered page will be manipulated such that they will -point to the equivalent content within the same revision ID. This means that you -can click around the staging environment, using site navigation normally, -without accidentally jumping to the production endpoint instead. +``/user-smashwilson/`` path exactly as it appears on the current +production site. Any links on any rendered page will be manipulated +such that they will point to the equivalent content within the same +revision ID. This means that you can click around the staging +environment, using site navigation normally, without accidentally +jumping to the production endpoint instead.