archi@data

Data Architecture

Time Series, events, logging...

• Influxdata - TICK Stack: https://www.influxdata.com/time-series-platform/
  • Telegraf
    • RabbitMQ Input Plugin: https://github.com/influxdata/telegraf/tree/master/plugins/inputs/rabbitmq
    • MQTT Consumer Input Plugin: https://github.com/influxdata/telegraf/tree/master/plugins/inputs/mqtt_consumer
      • https://github.com/influxdata/telegraf/issues/4580
    • NB: runs on Windows: https://github.com/influxdata/telegraf/blob/master/docs/WINDOWS_SERVICE.md
  • InfluxDB
  • Chronograf
    • Grafana vs Chronograf: https://community.influxdata.com/t/grafana-and-chronograf-concurrents-what-is-the-difference/8282/2
    • Use cases:
      • Chronograf is better at data exploration and understanding schemas for InfluxDB data.
      • Typically we tell people to use Chronograf to explore and play around with their data. Once they find something they want to be able to check/monitor then you create a Grafana dashboard and/or alert based on that data. If you have to find new data inside Grafana, well it can be painful
  • Kapacitor
• InfluxDB + Grafana + Chronograf: https://medium.com/@xaviergeerinck/how-to-visualize-metrics-with-grafana-and-chronograf-by-using-influxdb-450a6b796710
• TIG Stack (Telegraf/InfluxDB/Grafana) - MQTT -> Telegraf -> InfluxDB:
  • https://xebia.com/blog/use-the-tig-stack-to-create-your-iot-home-automation-dashboards/
  • https://www.andymelichar.com/2017/02/26/iot-data-management-with-the-tig-stack/
• Grafana vs Kibana: https://logz.io/blog/grafana-vs-kibana/
  • Logs vs. metrics: Grafana is designed for analyzing and visualizing metrics (...). Kibana, on the other hand, runs on top of Elasticsearch and is used primarily for analyzing log messages.

Big Data, Streaming

Apache Kafka

• Kafka can be used for activity monitoring. The activity could belong to a website or physical sensors and devices. Producers can publish raw data from data sources that later can be used to find trends and pattern. From https://towardsdatascience.com/getting-started-with-apache-kafka-in-python-604b3250aa05
• Getting Started with Kafka & python: https://towardsdatascience.com/getting-started-with-apache-kafka-in-python-604b3250aa05
• Confluent's blog
  • Overview: https://www.confluent.io/blog/stream-data-platform-1/ and https://www.confluent.io/blog/stream-data-platform-2/
  • Kafka for Data Scientists: https://www.confluent.io/blog/putting-power-apache-kafka-hands-data-scientists/

QuasarDB

• https://quasardb.net/
  • .NET Client: https://github.com/bureau14/qdb-api-dotnet
  • Windows Administration: https://github.com/quasar/QuasarRAT
  • QuasarDB + Kafka:
    • https://doc.quasardb.net/master/api/kafka.html
    • https://github.com/bureau14/kafka-connect-qdb
    • https://jujucharms.com/u/bigdata-dev/kafka-quasardb/

Juju (Hadoop platform)

Juju: The reference implementation of the Anssr Data Platform, developed by Juju experts at Spicule.

• https://jujucharms.com/
  • https://jujucharms.com/big-data
  • https://jujucharms.com/how-it-works

Machining Data

• MTConnect vs OPC-UA: https://www.mmsonline.com/columns/the-difference-between-mtconnect-and-opc-ua

MTConnect

OPC-UA

MQTT

MongoDB

• Time-series and MongoDB:

  • https://www.mongodb.com/blog/post/time-series-data-and-mongodb-part-1-introduction. This is the intro article; nothing really useful here...
  • https://www.mongodb.com/blog/post/time-series-data-and-mongodb-part-2-schema-design-best-practices. Interesting concepts:
    • Use WiredTiger engine, not MMAPV1
    • data bucketting,
    • TTL indexes (https://docs.mongodb.com/manual/core/index-ttl/)
    • Collection per block of time
  • https://www.mongodb.com/blog/post/time-series-data-and-mongodb-part-3--querying-analyzing-and-presenting-timeseries-data. Querying:
    • Aggregation Framework
    • Read Only Views (materialized at run-time)

• Articles at red-gate:

  • MongoDB: https://www.red-gate.com/simple-talk/dotnet/net-development/how-to-program-with-mongodb-using-the-net-driver/
  • CQRS (using ASP.NET Core, RabbitMQ and MongoDB): https://www.red-gate.com/simple-talk/dotnet/c-programming/getting-started-with-cqrs-part-1/