-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
24 changed files
with
1,009 additions
and
15 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,162 @@ | ||
| <!DOCTYPE html> | ||
| <html class="writer-html5" lang="en"> | ||
| <head> | ||
| <meta charset="utf-8" /> | ||
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> | ||
| <title>mltb2.db — MLTB2 documentation</title> | ||
| <link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=80d5e7a1" /> | ||
| <link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=19f00094" /> | ||
| <link rel="stylesheet" type="text/css" href="../../_static/copybutton.css?v=76b2166b" /> | ||
| <link rel="stylesheet" type="text/css" href="../../_static/css/custom.css?v=de248184" /> | ||
|
|
||
|
|
||
| <!--[if lt IE 9]> | ||
| <script src="../../_static/js/html5shiv.min.js"></script> | ||
| <![endif]--> | ||
|
|
||
| <script src="../../_static/jquery.js?v=5d32c60e"></script> | ||
| <script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script> | ||
| <script data-url_root="../../" id="documentation_options" src="../../_static/documentation_options.js?v=b3ba4146"></script> | ||
| <script src="../../_static/doctools.js?v=888ff710"></script> | ||
| <script src="../../_static/sphinx_highlight.js?v=4825356b"></script> | ||
| <script src="../../_static/clipboard.min.js?v=a7894cd8"></script> | ||
| <script src="../../_static/copybutton.js?v=f281be69"></script> | ||
| <script src="../../_static/js/theme.js"></script> | ||
| <link rel="index" title="Index" href="../../genindex.html" /> | ||
| <link rel="search" title="Search" href="../../search.html" /> | ||
| </head> | ||
|
|
||
| <body class="wy-body-for-nav"> | ||
| <div class="wy-grid-for-nav"> | ||
| <nav data-toggle="wy-nav-shift" class="wy-nav-side"> | ||
| <div class="wy-side-scroll"> | ||
| <div class="wy-side-nav-search" > | ||
|
|
||
|
|
||
|
|
||
| <a href="../../index.html" class="icon icon-home"> | ||
| MLTB2 | ||
| </a> | ||
| <div role="search"> | ||
| <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get"> | ||
| <input type="text" name="q" placeholder="Search docs" aria-label="Search docs" /> | ||
| <input type="hidden" name="check_keywords" value="yes" /> | ||
| <input type="hidden" name="area" value="default" /> | ||
| </form> | ||
| </div> | ||
| </div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu"> | ||
| <ul> | ||
| <li class="toctree-l1"><a class="reference internal" href="../../api-reference.html">API Reference</a></li> | ||
| <li class="toctree-l1"><a class="reference internal" href="../../genindex.html">Index</a></li> | ||
| <li class="toctree-l1"><a class="reference external" href="https://github.com/telekom/mltb2">GitHub Repository</a></li> | ||
| <li class="toctree-l1"><a class="reference external" href="https://github.com/telekom/mltb2/blob/main/LICENSE">License</a></li> | ||
| <li class="toctree-l1"><a class="reference external" href="https://www.telekom.com/imprint">Imprint</a></li> | ||
| </ul> | ||
|
|
||
| </div> | ||
| </div> | ||
| </nav> | ||
|
|
||
| <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" > | ||
| <i data-toggle="wy-nav-top" class="fa fa-bars"></i> | ||
| <a href="../../index.html">MLTB2</a> | ||
| </nav> | ||
|
|
||
| <div class="wy-nav-content"> | ||
| <div class="rst-content"> | ||
| <div role="navigation" aria-label="Page navigation"> | ||
| <ul class="wy-breadcrumbs"> | ||
| <li><a href="../../index.html" class="icon icon-home" aria-label="Home"></a></li> | ||
| <li class="breadcrumb-item"><a href="../index.html">Module code</a></li> | ||
| <li class="breadcrumb-item active">mltb2.db</li> | ||
| <li class="wy-breadcrumbs-aside"> | ||
| </li> | ||
| </ul> | ||
| <hr/> | ||
| </div> | ||
| <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article"> | ||
| <div itemprop="articleBody"> | ||
|
|
||
| <h1>Source code for mltb2.db</h1><div class="highlight"><pre> | ||
| <span></span><span class="c1"># Copyright (c) 2023 Philip May</span> | ||
| <span class="c1"># This software is distributed under the terms of the MIT license</span> | ||
| <span class="c1"># which is available at https://opensource.org/licenses/MIT</span> | ||
|
|
||
| <span class="sd">"""Database utils module."""</span> | ||
|
|
||
| <span class="kn">from</span> <span class="nn">abc</span> <span class="kn">import</span> <span class="n">ABC</span><span class="p">,</span> <span class="n">abstractmethod</span> | ||
| <span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclass</span> | ||
| <span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">Callable</span><span class="p">,</span> <span class="n">Sequence</span> | ||
|
|
||
|
|
||
| <div class="viewcode-block" id="AbstractBatchDataManager"><a class="viewcode-back" href="../../api-reference/db.html#mltb2.db.AbstractBatchDataManager">[docs]</a><span class="k">class</span> <span class="nc">AbstractBatchDataManager</span><span class="p">(</span><span class="n">ABC</span><span class="p">):</span> | ||
| <span class="w"> </span><span class="sd">"""Abstract base class for batch processing of database data.</span> | ||
|
|
||
| <span class="sd"> This class (respectively an implementation of it) is intended to be</span> | ||
| <span class="sd"> used in conjunction with the :class:`BatchDataProcessor`.</span> | ||
| <span class="sd"> """</span> | ||
|
|
||
| <div class="viewcode-block" id="AbstractBatchDataManager.load_batch"><a class="viewcode-back" href="../../api-reference/db.html#mltb2.db.AbstractBatchDataManager.load_batch">[docs]</a> <span class="nd">@abstractmethod</span> | ||
| <span class="k">def</span> <span class="nf">load_batch</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="n">Sequence</span><span class="p">:</span> | ||
| <span class="w"> </span><span class="sd">"""Load a batch of data from the database."""</span> | ||
| <span class="k">pass</span></div> | ||
|
|
||
| <div class="viewcode-block" id="AbstractBatchDataManager.save_batch"><a class="viewcode-back" href="../../api-reference/db.html#mltb2.db.AbstractBatchDataManager.save_batch">[docs]</a> <span class="nd">@abstractmethod</span> | ||
| <span class="k">def</span> <span class="nf">save_batch</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">batch</span><span class="p">:</span> <span class="n">Sequence</span><span class="p">)</span> <span class="o">-></span> <span class="kc">None</span><span class="p">:</span> | ||
| <span class="w"> </span><span class="sd">"""Save a batch of data to the database."""</span> | ||
| <span class="k">pass</span></div></div> | ||
|
|
||
|
|
||
| <div class="viewcode-block" id="BatchDataProcessor"><a class="viewcode-back" href="../../api-reference/db.html#mltb2.db.BatchDataProcessor">[docs]</a><span class="nd">@dataclass</span> | ||
| <span class="k">class</span> <span class="nc">BatchDataProcessor</span><span class="p">:</span> | ||
| <span class="w"> </span><span class="sd">"""Process batches of data from a database.</span> | ||
|
|
||
| <span class="sd"> Args:</span> | ||
| <span class="sd"> data_manager: The data manager to load and save batches of data.</span> | ||
| <span class="sd"> process_batch_callback: A callback function that processes one batch of data.</span> | ||
| <span class="sd"> """</span> | ||
|
|
||
| <span class="n">data_manager</span><span class="p">:</span> <span class="n">AbstractBatchDataManager</span> | ||
| <span class="n">process_batch_callback</span><span class="p">:</span> <span class="n">Callable</span><span class="p">[[</span><span class="n">Sequence</span><span class="p">],</span> <span class="n">Sequence</span><span class="p">]</span> | ||
|
|
||
| <div class="viewcode-block" id="BatchDataProcessor.run"><a class="viewcode-back" href="../../api-reference/db.html#mltb2.db.BatchDataProcessor.run">[docs]</a> <span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="kc">None</span><span class="p">:</span> | ||
| <span class="w"> </span><span class="sd">"""Run the batch data processing.</span> | ||
|
|
||
| <span class="sd"> This is done until the data manager returns an empty batch.</span> | ||
| <span class="sd"> For each batch the ``process_batch_callback`` is called.</span> | ||
| <span class="sd"> Data is loaded by using an implementation of the :class:`AbstractBatchDataManager`.</span> | ||
| <span class="sd"> """</span> | ||
| <span class="k">while</span> <span class="kc">True</span><span class="p">:</span> | ||
| <span class="n">batch</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">data_manager</span><span class="o">.</span><span class="n">load_batch</span><span class="p">()</span> | ||
| <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">batch</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> | ||
| <span class="k">break</span> | ||
| <span class="n">new_batch</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">process_batch_callback</span><span class="p">(</span><span class="n">batch</span><span class="p">)</span> | ||
| <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">new_batch</span><span class="p">)</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span> | ||
| <span class="bp">self</span><span class="o">.</span><span class="n">data_manager</span><span class="o">.</span><span class="n">save_batch</span><span class="p">(</span><span class="n">new_batch</span><span class="p">)</span></div></div> | ||
| </pre></div> | ||
|
|
||
| </div> | ||
| </div> | ||
| <footer> | ||
|
|
||
| <hr/> | ||
|
|
||
| <div role="contentinfo"> | ||
| <p>© Copyright 2023, Philip May, Deutsche Telekom AG.</p> | ||
| </div> | ||
|
|
||
|
|
||
|
|
||
| </footer> | ||
| </div> | ||
| </div> | ||
| </section> | ||
| </div> | ||
| <script> | ||
| jQuery(function () { | ||
| SphinxRtdTheme.Navigation.enable(true); | ||
| }); | ||
| </script> | ||
|
|
||
| </body> | ||
| </html> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| .. _aramgodb_code_doc: | ||
|
|
||
| :mod:`~mltb2.arangodb` | ||
| ====================== | ||
|
|
||
| .. automodule:: mltb2.arangodb |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| .. _db_code_doc: | ||
|
|
||
| :mod:`~mltb2.db` | ||
| ================ | ||
|
|
||
| .. automodule:: mltb2.db |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.