Skip to content

Home

Jump to bottom
Greg Latuszek edited this page Mar 8, 2018 · 10 revisions

Welcome to the moler wiki!

Architecture - fundamental idea

Architectural idea behind this library is separation of 3 abstractions:

  1. commands/connection-observers
  2. concurrency
  3. I/O connections

This is to allow reusing the first one (main focus of Moler) with any concurrency/IO. That also means independent development of the 3 abstractions.

What I mean concurrency? Anything that allows given command to "run in background" and then "await till it is done". It may be realized by threads, twisted, asyncio, curio, ... .

But, at the end I want simplicity for end user. She/he should be able to just say "run in background" without knowing "how".

Architecture inspirations

As it comes to I/O I've been insiped by “bring-your-own-I/O” idea of hyper-h2 and h11. This idea minimizes dependency on I/O, allows code reuse under different I/O.

As it comes to layering concept I've been inspired by Cory Benfield's PYCON UK 2015: Simplicity Is A Feature

However, “bring-your-own-I/O” applies only to Layer-1. Next layers introduce some implementation of I/O provided by Moler but you can also switch them to your own. That way user gains next-level-of-simplicity "no need to write I/O, take what is given".

Architecture layers

We will go over Moler's architecture bottom-up.

Lowest layer gives you control over all details. You may combine all pieces as you wish but the price for it is more effort on your side, more code to glue them all. Up in layers means less code to write for the price of less freedom.

Layer 1 - glue it all manually (under your concurrency-I/O)

moler_layer1

  • External-IO-connection just passes bytes
  • Moler's connection perform encoding/decoding and dispatching to multiple observers doing "parallel" observation

Layer 2 - just use it

This layer introduces concept of runners - entities that are used internally by connection observers to realize "background run". moler_layer2 Allows to write code like (see layer_2 examples):

# using connection-observer as future:
net_down_detector.start()
# ...
net_down_time = net_down_detector.await_done(timeout=10)

# or using it as synchronous function
detect_network_up = net_up_detector  # function, so we want verb to express action
net_up_time = detect_network_up()

Concurrency here is used for two purposes:

  1. background run of commands/connection-observers
  2. self-running connections - after they are open they can read data as it comes (not delayed by lack of control) concurrency_variants External I/O variants are:
  • variants per I/O type: tcp, udp, ssh, process, shell, ...
  • multiplied by varians of concurrency: tcp on asyncio, tcp on thisted, socket tcp in thread, ... ext_IO_variants

Layer 3 - configurable choice

User will select concurrency/IO via configuration.

Choice is done on background via configuration. This alows to write code like

my_remote_machine = get_connection(type="tcp", host="hostname", port=87995)

and thats it!!! Why should user bother if it is asyncio-TCP or Twisted-TCP?

Configuration may be done by python code or read from config files.

Layer 4 - autodetected variants

Modules for selecting configuration (concurrency/IO variants) based on software available on given machine.

Like: "asyncio only if you have Python3".

Clone this wiki locally