From de95d7b6fb2e5e3c56dfc5ebc7ab828e7c35ac1e Mon Sep 17 00:00:00 2001 From: Tom Clark <> Date: Thu, 12 Sep 2019 21:54:53 +0100 Subject: [PATCH] DOC Restructured digital twins sections, fixed warnings and issues throughout, added variou scommentary and images --- .gitignore | 5 + docs/source/digital_twins.rst | 57 ++++++++ docs/source/gemini.rst | 16 --- .../images/digital_twin_component_basic.svg | 1 + docs/source/index.rst | 71 ++++------ docs/source/installation.rst | 10 +- docs/source/schema.rst | 17 +-- docs/source/schema_introducing_json.rst | 133 ++++++++++-------- 8 files changed, 184 insertions(+), 126 deletions(-) create mode 100644 docs/source/digital_twins.rst delete mode 100644 docs/source/gemini.rst create mode 100644 docs/source/images/digital_twin_component_basic.svg diff --git a/.gitignore b/.gitignore index 0e3da37..4d45ae1 100644 --- a/.gitignore +++ b/.gitignore @@ -105,3 +105,8 @@ venv.bak/ # jetbrains ides .idea/ + +# Sphinx related files +docs/source/doxyoutput/ +docs/source/library_api/ +docs/build/ diff --git a/docs/source/digital_twins.rst b/docs/source/digital_twins.rst new file mode 100644 index 0000000..999a3a9 --- /dev/null +++ b/docs/source/digital_twins.rst @@ -0,0 +1,57 @@ +.. _digital_twins: + +============= +Digital Twins +============= + +A digital twin is a virtual representation of a real life being - a physical asset like a wind turbine or car - or even +a human. + +There are three reasons why you might want to create a digital twin: + - Monitoring + - Prediction + - Optimisation + +On its own, a digital twin can be quite useful. For example, a twin might embody an AI-based analysis to predict power +output of a turbine. + +.. figure:: images/digital_twin_component_basic.svg + :width: 400px + :align: center + :figclass: align-center + :alt: A digital twin component + + A digital twin consists of some kind of analysis or processing task, which could be run many times per second, or + daily, down to occasionally or sometimes only once (the same as a "normal" analysis). + +Coupling digital twins is generally even more useful. You might wish to couple your turbine twin with a representation +of the local power grid, and a representation of a factory building to determine power demand... enabling you to +optimise your factory plant for lowest energy cost whilst intelligently selling surplus power to the grid. + +.. figure:: images/digital_twin_hierarchy.svg + :width: 350px + :align: center + :figclass: align-center + :alt: Hierarchy of digital twins + + A hierarchy of digital twins. Each blue circle represents a twin, coupled to its neighbours. Yellow nodes are where + schema are used to connect twins. + + +.. _gemini_principles: + +Gemini Principles +================= + +The Gemini Principles have been derived by the +`Centre for Digital Built Britain (CDBB) `_. +We strongly recommend you give them a read if embarking on a digital twins project. + +The aim of **twined** is to enable the following principles. In particular: + +#. Openness (open-source project to create schema for twins that can be run anywhere, anywhen) +#. Federation (encouraging a standardised way of connecting twins together) +#. Security (making sure schemas and data can be read safely) +#. Public Good (see our nano-rant about climate change in :ref:`reason_for_being`) + + diff --git a/docs/source/gemini.rst b/docs/source/gemini.rst deleted file mode 100644 index 67d75fc..0000000 --- a/docs/source/gemini.rst +++ /dev/null @@ -1,16 +0,0 @@ -.. _gemini_principles: - -================= -Gemini Principles -================= - -The Gemini Principles have been derived by the `Centre for Digital Built Britain (CDBB) `_. - -We strongly recommend you give them a read if embarking on a digital twins project. - -The aim of **twined** is to enable the following principles in particular: - -#. Openness (using an open-source project to create schema for applications/twins that can be run anywhere, anywhen) -#. Federation (encouraging a standardised way of connecting twins together) -#. Security (making sure schemas and data can be read safely) -#. Public Good (see our nano-rant about climate change in :ref:`reason_for_being`) diff --git a/docs/source/images/digital_twin_component_basic.svg b/docs/source/images/digital_twin_component_basic.svg new file mode 100644 index 0000000..b49800f --- /dev/null +++ b/docs/source/images/digital_twin_component_basic.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/source/index.rst b/docs/source/index.rst index 8c3d728..8574b0c 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -1,3 +1,9 @@ +.. ATTENTION:: + This library is in very early stages. Like the idea of it? Please + `star us on GitHub `_ and contribute via the + `issues board `_ and + `roadmap `_. + .. image:: https://codecov.io/gh/octue/twined/branch/master/graph/badge.svg :target: https://codecov.io/gh/octue/twined :alt: Code coverage @@ -8,70 +14,52 @@ :align: right ====== -twined +Twined ====== -**twined** is a library to help digital twins talk to one another. - +**twined** is a library to help :ref:`digital_twins` talk to one another. .. epigraph:: - "Twined" [t-why-nd] - encircled, twisted together, interwoven - - -.. ATTENTION:: - This library is in very early stages. Like the idea of it? Please - `star us on GitHub `_ and contribute via the - `issues board `_ and - `roadmap `_. - - -.. _digital_twins: -Digital Twins -============= + *"Twined" [t-why-nd] ~ encircled, twisted together, interwoven* A digital twin is a virtual representation of a real life being - a physical asset like a wind turbine or car - or even -a human. +a human. Like real things, digital twins need to interact, so can be connected together, but need a common communication +framework to do so. This is what is provided by **twined**. -.. figure:: images/digital_twin_hierarchy.png - :width: 500px +.. figure:: images/digital_twin_hierarchy.svg + :width: 350px :align: center :figclass: align-center :alt: Hierarchy of digital twins - A hierarchy of digital twins. Each blue circle represents a twin, coupled to its neighbours. Yellow nodes are where - schema are used to connect twins. - -There are three reasons why you might want to create a digital twin: - - Monitoring - - Prediction - - Optimisation - -On its own, a digital twin can be quite useful. For example, a twin might embody an AI-based analysis to predict power -output of a turbine. - -Coupling digital twins is generally even more useful. You might wish to couple your turbine twin with a representation -of the local power grid, and a representation of a factory building to determine power demand... enabling you to -optimise your factory plant for lowest energy cost whilst intelligently selling surplus power to the grid. + Digital twins connected in a hierarchy. Each blue circle represents a twin, coupled to its neighbours. + Yellow nodes are where schema are used to connect twins. .. _aims: + Aims ==== -**twined** provides a toolkit to help create and validate "schema" - descriptions of a digital twin, what data it +**twined** provides a toolkit to help create and validate ":ref:`schema`" - descriptions of a digital twin, what data it requires, what it does and how it works. The goals of **twined** are as follows: - Provide a clear framework for what a digital twin schema can and/or must contain - Provide functions to validate incoming data against a known schema - - Provide tools to create schema describing what you require - - Provide a function to check that a schema itself is valid + - Provide functions to check that a schema itself is valid + - Provide (or direct you to) tools to create schema describing what you require -Using schema, we can describe how digital twins connect and interact... building them together in hierarchies and -networks. You'll find an in depth description in the :ref:`schema` section. +Using :ref:`schema`, we can describe how digital twins connect and interact... building them together in hierarchies and +networks. + +The scope of **twined** is not large. Many other libraries will deal with hosting and deploying digital twins, still +more will deal with the actual analyses done within them. **twined** purely deals with parsing and checking the +information exchanged. .. _reason_for_being: + Raison d'etre ============= @@ -86,6 +74,7 @@ develop a wider ecosystem. .. _uses: + Uses ===== @@ -104,6 +93,7 @@ building digital twins using **twined**. .. _life_choices: + Life Choices ============ @@ -116,12 +106,13 @@ and are willing to consider sponsorship of development and maintenance of that l .. toctree:: - :maxdepth: 1 + :maxdepth: 2 :hidden: self - installation + digital_twins schema + installation examples license version_history diff --git a/docs/source/installation.rst b/docs/source/installation.rst index 7b5ca8e..0cbc136 100644 --- a/docs/source/installation.rst +++ b/docs/source/installation.rst @@ -4,16 +4,18 @@ Installation ============ -**twined** is available on pypi, so installation into your python virtual environment is dead simple: +**twined** is available on `pypi `_, so installation into your python virtual environment is dead +simple: -.. code-block:: +.. code-block:: py pip install twined -Don't have a virtual environment with pip? You probably should! pyenv is your friend. Google it. +Don't have a virtual environment with pip? You probably should! ``pyenv`` is your friend. Google it. .. _compilation: + Compilation ============ @@ -21,6 +23,7 @@ There is presently no need to compile **twined**, as it's written entirely in py .. _third_party_library_installation: + Third party library installation ================================ @@ -29,6 +32,7 @@ automatically installed during the installation above. .. _third_party_build_requirements: + Third party build requirements ============================== diff --git a/docs/source/schema.rst b/docs/source/schema.rst index 001d9f8..a35cb46 100644 --- a/docs/source/schema.rst +++ b/docs/source/schema.rst @@ -12,9 +12,9 @@ Requirements of digital twin schema A *schema* defines a digital twin, and has multiple roles. It: -#. defines what data is required by a digital twin, in order to run -#. defines what data will be returned by the twin following a successful run -#. defines the formats of these data, in such a way that incoming data can be validated +#. Defines what data is required by a digital twin, in order to run +#. Defines what data will be returned by the twin following a successful run +#. Defines the formats of these data, in such a way that incoming data can be validated If this weren't enough, the schema: @@ -24,17 +24,18 @@ If this weren't enough, the schema: #. Must be searchable/indexable Fortunately for digital twin developers, many of these requirements have already been seen for data interchange formats -developed for the web. **twined** uses `JSON` and `JSONSchema` to interchange data between digital twins. If you're not -already familiar with JSONSchema (or wish to know why **twined** uses `JSON` over the seemingly more appropriate `XML` -standard), see :ref:`introducing_json_schema`. +developed for the web. **twined** uses ``JSON`` and ``JSONSchema`` to interchange data between digital twins. + +If you're not already familiar with ``JSONSchema`` (or wish to know why **twined** uses ``JSON`` over the seemingly more +appropriate ``XML`` standard), see :ref:`introducing_json_schema`. .. _specifying_a_framework: Specifying a framework ====================== -We cannot simply expect many developers to create digital twins with a `JSONSchema` then to be able to connect them all -together. **twined** makes things slightly more specific +We cannot simply expect many developers to create digital twins with a ``JSONSchema`` then to be able to connect them all +together. **twined** makes things slightly more specific. .. toctree:: diff --git a/docs/source/schema_introducing_json.rst b/docs/source/schema_introducing_json.rst index 84c8644..8f20858 100644 --- a/docs/source/schema_introducing_json.rst +++ b/docs/source/schema_introducing_json.rst @@ -10,39 +10,51 @@ in recent years. ``JSONSchema`` is a way of specifying what a ``JSON`` document should contain. The Schema are, themselves, written in ``JSON``! -Whilst schema can become extremely complicated in some scenarios, they can also become quite succinct. See below for the -schema (and matching ``JSON``) for an integer and a string variable. - -.. tabs:: - - .. tab:: JSON - { - "id": 1, - "name": "Tom" - } - - .. tab:: Schema - { - "type": "object", - "title": "An id number and a name", - "properties": { - "id": { - "type": "integer", - "title": "An integer number", - "default": 0 - }, - "name": { - "type": "string", - "title": "A string name", - "default": "" - } - } - } - -.. list-table:: Some useful resources for JSON Schema +Whilst schema can become extremely complicated in some scenarios, they are best designed to be quite succinct. See below +for the schema (and matching ``JSON``) for an integer and a string variable. + +**JSON:** + +.. code-block:: json + + { + "id": 1, + "name": "Tom" + } + + +**Schema:** + +.. code-block:: json + + { + "type": "object", + "title": "An id number and a name", + "properties": { + "id": { + "type": "integer", + "title": "An integer number", + "default": 0 + }, + "name": { + "type": "string", + "title": "A string name", + "default": "" + } + } + } + + +.. _useful_resources: + +Useful resources +================ +.. list-table:: :widths: auto :header-rows: 1 + * - Link + - Resource * - https://jsonschema.net/ - Useful web tool for inferring schema from existing json * - https://jsoneditoronline.org @@ -56,12 +68,13 @@ schema (and matching ``JSON``) for an integer and a string variable. .. _human_readbility: + Human readability ================= Back in our :ref:`requirements` section, we noted it was important for humans to read and understand schema. -The actual documents themselves are pretty easy to read by technical users. For non technical users, readability can be +The actual documents themselves are pretty easy to read by technical users. But, for non technical users, readability can be enhanced even further by the ability to turn ``JSONSchema`` into web forms automatically. For our example above, we can autogenerate a web form straight from the schema: @@ -78,8 +91,9 @@ interface without writing a separate form component - great for ease and maintai .. _why_not_xml: -Why not XML -=========== + +Why not XML? +============ In a truly excellent `three-part blog `_, writer Seva Savris takes us through the ups and downs of ``JSON`` versus ``XML``; well worth a read if wishing to understand the respective technologies @@ -88,31 +102,32 @@ better. In short, both ``JSON`` and ``XML`` are generalised data interchange specifications and can both can do what we want here. We choose ``JSON`` because: -#. Textual representation is *much* more concise and easy to understand (very important where non-developers like - engineers and scientists must be expected to interpret schema) +#. Textual representation is much more concise and easy to understand (very important where non-developers like + engineers and scientists must be expected to interpret schema) #. `Attack vectors `_. Because entities in ``XML`` - are not necessarily primitives (unlike in ``JSON``), an ``XML`` document parser in its default state may leave a system - open to XXE injection attacks and DTD validation attacks, and therefore requires hardening. ``JSON`` documents are - similarly afflicated (just like any kind of serialized data) but default parsers, operating on the premise of only - deserializing to primitive types, are safe by default - it is only when nondefault parsering or deserialization - techniques (such as ``JSONP``) are used that the application becomes vulnerable. By utilising a default ``JSON`` parser - we can therefore significantly shrink the attack surface of the system. See - `this blog post `_ for further discussion. - -#. ``XML`` is powerful... perhaps *too* powerful. The standard can be adapted greatly, resulting in high encapsulation - and a high resilience to future unknowns. Both beneficial. However, this requires developers of twins to maintain - interfaces of very high complexity, adaptable to a much wider variety of input. To enable developers to progress, we - suggest handling changes and future unknowns through well-considered versioning, whilst keeping their API simple. - -#. ``XML`` allows baked-in validation of data and metadata. Whilst advantageous in some situations, this is not a - benefit here. We wish validation to be one-sided: validation of data accepted/generated by a digital twin should be - occur within (at) the boundaries of that twin. - -#. With ``JSON`` Schema, we can add the majority of the required validation capabilities (otherwise missing from JSON - but present in ``XML``) to ``JSON`` documents, with our schema specified where we need them. - -#. ``JSON`` is a much more compact expression than XML, significantly reducing memory and bandwidth requirements. Whilst - not a major issue for most modern PCS, sensors on the edge may have limited memory, and both memory and bandwidth at - scale are extremely expensive. Thus for extremely large networks of interconnected systems there could be significant - speed and cost savings. + are not necessarily primitives (unlike in ``JSON``), an ``XML`` document parser in its default state may leave a system + open to XXE injection attacks and DTD validation attacks, and therefore requires hardening. ``JSON`` documents are + similarly afflicated (just like any kind of serialized data) but default parsers, operating on the premise of only + deserializing to primitive types, are safe by default - it is only when nondefault parsering or deserialization + techniques (such as ``JSONP``) are used that the application becomes vulnerable. By utilising a default ``JSON`` parser + we can therefore significantly shrink the attack surface of the system. See + `this blog post `_ for further discussion. + +#. ``XML`` is powerful... perhaps too powerful. The standard can be adapted greatly, resulting in high encapsulation + and a high resilience to future unknowns. Both beneficial. However, this requires developers of twins to maintain + interfaces of very high complexity, adaptable to a much wider variety of input. To enable developers to progress, we + suggest handling changes and future unknowns through well-considered versioning, whilst keeping their API simple. + +#. ``XML`` allows baked-in validation of data and attributes. Whilst advantageous in some situations, this is not a + benefit here. We wish validation to be one-sided: validation of data accepted/generated by a digital twin should be + occur within (at) the boundaries of that twin. + +#. Required validation capabilities, built into ``XML`` are achievable with ``JSONSchema`` (otherwise missing from the + pure ``JSON`` standard) + +#. ``JSON`` is a more compact expression than XML, significantly reducing memory and bandwidth requirements. Whilst + not a major issue for most modern PCS, sensors on the edge may have limited memory, and both memory and bandwidth at + scale are extremely expensive. Thus for extremely large networks of interconnected systems there could be significant + speed and cost savings. +