[WIP] Support DOT inputs (and multigraphs); much improved boundary node splitting in pattern identification; add back frayed rope support; lots of other improvements

[fedarko]

(The interactive visualization side of things still needs to be updated, so this isn't ready yet, but the DOT / stats export options both work.)

Changes

• MetagenomeScope now accepts DOT files produced by Flye and LJA.

  • This means that MetagenomeScope can now visualize de Bruijn graphs (DBGs) without having to load them from GFA / FASTG files (meaning that we can actually draw the literal structure of DBGs -- GFA and FASTG files implicitly convert edges to nodes and vice versa).

  • Nodes in these graphs are drawn as small circles (which matches existing DBG visualizations in the literature, including AGB's).

  • Added support for parallel edges (closes Need to set up decomposed graph as a MultiDiGraph #202, closes Explicitly handle/test duplicate edges in the decomposed graph #187, closes Add defined behavior for duplicate edges #75), whether they're in the input assembly graph file or in the decomposed graph. (Note that the FASTG and GFA parsers will throw errors if they see parallel edges, so I've opened Support duplicate (aka parallel) edges in GFA / FASTG files? #239 accordingly.)

  • Since "bubbles" often manifest in DBGs as "bulges" (in which there exist multiple edges between the same pair of nodes), I expanded the validation code to accept bulges as "bubbles".

    • Just for reference: we say there exists a bulge between two nodes X and Y if there multiple edges X → Y, X has no outgoing edges to other nodes, and Y has no incoming edges from other nodes.

• Refactor the assembly graph code.

  • Rather than store node and edge metadata (e.g. length, orientation, multiplicity, ...) in the NetworkX graph structures, we instead create Node and Edge objects that we use to store this information. The nodes and edges in the NX graphs contain IDs that point to these objects. (Amusingly, this is actually pretty close to the nodeid2obj structures that the earliest versions of MetagenomeScope created.) This makes the code much cleaner. Closes Centralize data storage in the Python codebase #204.

  • Overhauled hierarchical pattern identification.

    • Added back frayed ropes as a "top-level-only" pattern type (closes Disallow frayed ropes from being used in other types of patterns in the decomposition #200).

    • Support "end-to-start" cyclic bubbles (Identifying cyclic bubbles? #241) and frayed ropes (closes Identify cyclic frayed ropes? #242). (Need to do some more testing and also there's one silly issue if these are in isolated components, so I'm leaving Identifying cyclic bubbles? #241 open.)

    • Add automatic boundary node splitting for bubbles, chains, cyclic chains,
      and frayed ropes by default; unneeded split nodes are then merged back
      afterwards. Closes Auto-duplicate bubbles' boundary nodes, then trim "unneeded" duplicates at the end of pattern detection #167, and also addresses the un-checked boxes in Hierarchical decomposition improvements #164.
      This makes the pattern decompositions so much nicer than before.

    • Replace the idea of "duplicate nodes" for boundary node splitting (where
      one of them was that gross shade of pink) with the idea of left/right
      splitting -- if a given node is split, then its left split node is marked
      as a "left split" node and its right split node is marked as a "right split"
      node. This means that, rather than adjusting node colors, we can adjust
      the shape of nodes accordingly (which is a much more intuitive way of
      showing this). Closes Alternate, more intuitive representation of duplicate bubble boundary nodes #206.

• Additional output options.

  • Add the ability to save a DOT file of the assembly graph (representing patterns as clusters, like the ~2016-era version of MetagenomeScope).

    • I'd like to eventually also support saving XDOT files that use the exact node/edge coordinates used in the visualization interface (so, after applying backfilling), but that will take some extra work.
    • Part of the motivation behind this feature: if users want fancy images of the assembly graph (e.g. SVG) that Cytoscape.js can't export right now, then they can just extract them using DOT (which can create SVGs).

  • Add the ability to save statistics about each connected component of the assembly graph to a TSV file. This helps give users a high-level overview of large graphs -- I think it'll be handy when e.g. working with large graphs on a remote server, where visualizations may be impractical.

  • Made the original output option (saving an interactive visualization) optional. Now, the user can specify anywhere from zero to all three of these output options. (If you don't select any output options, MetagenomeScope will just identify patterns, output the number of patterns identified, and call it a day.)

• Documentation updates.

  • Added developer documentation.
  • Lots of updates to the README (vignettes, FAQs, expanded filetype table, ...)

• Other stuff.

  • Test on Python 3.6, 3.7, 3.8, 3.9, and 3.10 in addition to just 3.6.
  • Add an option to disallow pattern identification (--no-patterns).
  • Add explicit options to disable large component removal (setting -maxn / -maxe to 0 removes the checks).
    • Given the tests that I added for this, I think it's safe to close Detect prohibitively large components and skip (initial) layout/rendering for them #137 now. (Maybe test more specifically that output is formatted nicely...? eh)
  • [WIP] Add options to allow users to specify arbitrary node/edge metadata in input TSV files (--node-metadata / --edge-metadata). Closes Support general node / edge metadata #243.
  • Add a version option to the CLI (-v / --version).
  • Lots of other changes.

Things to address before merging this in

• Update the layout code and everything downstream (the interactive viz stuff) to work with these changes.
• Ideally, more tests.

Ignore all that, show me some pretty pictures

These are all produced by running mgsc -i [graph file] -od [dot output], and then visualizing the resulting DOT file with Graphviz. Note that this process doesn't perform backfilling, so these layouts will look a bit different from what's shown in the visualization interface.

Flye yeast assembly graph from AGB's GitHub repository	Simple chain of two bubbles	Cyclic chain of 3 bubbles in the Bandage E. coli graph