Data-Infra-Projects

This is an attempt to list out all the interesting projects.

It is intended for anyone designing modern large scale architectures and need to choose tools/technoglogies/frameworks. The purpose is to help in making that choices with resources like comparisons/use-cases/features/maturity or really anything that helps in making an informed decision.

Abstractions

• Spark RDD
• Spark Dataframe
• DataFlow
• MapReduce
• Bulk Synchronous Parallel - BSP
• CRDT
• Merkle Tree
• DHT

Distributed Coordination

This are implementations/libraries to help write distributed applications which require some form of coordination.

• ZooKeeper
• Raft
• Serf
• Doozer

Infrastructure Management

• Yarn
• Mesos
• OpenStack
• CloudStack
• Ganeti
• CoreOS
• Kubernetes
• Eucalyptus

comparisons

• Serf vs *
• Serf vs Mesos
• CoreOs-Fleet vs Mesos/YARN

File Systems

• HDFS
• GlusterFS
• Ceph
• QFS

Distributed Databases

• Swift
• Riak
• Pinot

Infrastrcuture Logging/Monitoring

• Nagios
• Ganglia
• Chukwa
• Netflix Suro
• Zabbix
• Riemann
• Servo

Infrastructure Helpers

• Aurora
• Marathon
• Cronos
• Fleet
• Helix
• Slider

MultiCloud/CrossCloud utilities

• Fog
• Multistack
• jClouds
• Whirr

Virtualization

• LXC
• ZeroVM
• KVM
• XEN
• Solaris Zones
• BSD Jails

Virtualization++

• Docker
• CloudFoundry
• Redhat Project Atomic
• OSv

Generalized Data Processing

• Hadoop
• Spark
• REEF
• Flink (previously know as Stratosphere)
• Tez
• Dryad
• Hyracks
• Naiad
• Vespa

comparisons

• Tez vs Dryad
• Hadoop vs Spark - Too many differences, no good link.

Largescale Distributed ML

• MLBase/MLlib
• Vowpal Wabbit
• Jubatus
• Mahout
• Hama
• h2o
• Oryx

pub-sub / messaging

• Kafka
• RabbitMQ
• ZeroMQ
• ActiveMQ
• hornetq

Data Ingest

• Sqoop
• Flume
• Gobblin

Data change management

• SpinalTap

Graph Storing and/or Processing

• Turi (previously knowns as GraphLab)
• Giraph
• Neo4j
• Cassovary

SQL Engines

• Hive
• Impala
• Spark-SQL
• Tajo
• Presto

Stream Processing

• Storm
• Spark Streaming
• Samza
• Borealis

Security

• Scumblr

Performance Analysis

• Dr. Elephant

Workflow engines/DAG-executors/Pipelines

• Oozie
• Luigi
• Azkaban
• Cascading
• Hmake
• Crunch (Modeled after FlumeJava)

Comparisons

• Oozie vs Crunch
• Oozie vs Azkaban
• Feature comparison

Configuration Management

• Chef
• Puppet
• Salt
• Ansible
• Vagrant
• Capistrano
• Fabric
• Bosh

Service Discovery

• etcd
• confd
• Vulcand
• Frontrunner
• Consul
• SkyDNS
• Synapse - part of SmartStack combined with Nerve

Comparison

• Consul vs others

Testing

• Jespen
• Simian Army

Visualization

• White Elephent
• Ambrose
• Lipstick
• Hue - Hadoop Web UI
• Inviso
• Timberlake

Libraries

• Zoie
• Norbert - cluster manager and networking layer built on top of Zookeeper.
• Okapi - Large-scale ML & graph analytics on Giraph
• Scalding - A Scala API for Cascading
• SummingBird - Streaming MapReduce with Scalding and Storm
• Curator - set of Java libraries that make using Apache ZooKeeper much easier
• Turbine - Low latency high throughput aggregator for real time streams
• DataFu - Collection of MapReduce lib
• Twill (Previsously known as Weave) - YARN application writing lib

Search

• Lucene+Solr
• ElasticSearch

others

• Nutch - web crawler
• Ambari - Hadoop Deployment + Management
• Bigtop - Hadoop Packaging
• Skuld
• Camus - LinkedIn's Kafka to HDFS pipeline.
• Kiji - collect, analyze and serve data in real time on Apache Hadoop and HBase