-
Notifications
You must be signed in to change notification settings - Fork 5
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
1 changed file
with
47 additions
and
81 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,113 +1,79 @@ | ||
| # Dialogy | ||
|
|
||
| [](https://travis-ci.com/Vernacular-ai/dialogy) | ||
| [](https://coveralls.io/github/Vernacular-ai/dialogy?branch=master) | ||
| [](https://www.codacy.com/gh/Vernacular-ai/dialogy/dashboard?utm_source=github.com&utm_medium=referral&utm_content=Vernacular-ai/dialogy&utm_campaign=Badge_Grade) | ||
| [](https://app.travis-ci.com/skit-ai/dialogy) | ||
| [](https://coveralls.io/github/skit-ai/dialogy?branch=master) | ||
| [](https://www.codacy.com/gh/skit-ai/dialogy/dashboard?utm_source=github.com&utm_medium=referral&utm_content=Vernacular-ai/dialogy&utm_campaign=Badge_Grade) | ||
| [](https://badge.fury.io/py/dialogy) | ||
|
|
||
| Dialogy is a batteries-included 🔋 opinionated framework to build machine-learning solutions for speech applications. | ||
|
|
||
| - Plugin-based: Makes it easy to import/export components to other projects. 🔌 | ||
| - Stack-agnostic: No assumptions made on ML stack; your choice of machine learning library will not be affected by using Dialogy. 👍 | ||
| - Progressive: Minimal boilerplate writing to let you focus on your machine learning problems. 🤏 | ||
|
|
||
| [Documentation](https://vernacular-ai.github.io/dialogy/) | ||
| Dialogy is a library for building SLU applications. | ||
| [Documentation](https://skit-ai.github.io/dialogy/) | ||
|
|
||
| ## Installation | ||
|
|
||
| ```shell | ||
| pip install dialogy | ||
| ``` | ||
|
|
||
| ## Test | ||
|
|
||
| ```shell | ||
| make test | ||
| ``` | ||
|
|
||
| ## Examples | ||
|
|
||
| Using `dialogy` to run a classifier on a new input. | ||
|
|
||
| ```python | ||
| import pickle | ||
| from dialogy.workflow import Workflow | ||
| from dialogy.preprocessing import merge_asr_output | ||
|
|
||
|
|
||
| def access(workflow): | ||
| return workflow.input | ||
| Dialogy's CLI supports building and migration of projects. | ||
|
|
||
| ```txt | ||
| dialogy -h | ||
| usage: dialogy [-h] {create,update,train} ... | ||
| def mutate(workflow, value): | ||
| workflow.input = value | ||
| positional arguments: | ||
| {create,update,train} | ||
| Dialogy project utilities. | ||
| create Create a new project. | ||
| update Migrate an existing project to the latest template version. | ||
| train Train a workflow. | ||
|
|
||
| def vectorizer(workflow): | ||
| vectorizer = TfidfVectorizer() | ||
| workflow.input = vectorizer.transform(workflow.input) | ||
|
|
||
|
|
||
| class TfidfMLPClfWorkflow(Workflow): | ||
| def __init__(self): | ||
| super(TfidfMLPClfWorkflow).__init__() | ||
| self.model = None | ||
|
|
||
| def load_model(self, model_path): | ||
| with open(model_path, "rb") as binary: | ||
| self.model = binary.load() | ||
|
|
||
| def inference(self): | ||
| self.output = self.model.predict(self.input) | ||
|
|
||
|
|
||
| preprocessors = [merge_asr_output(access=access, mutate=mutate), vectorizer] | ||
| workflow = TfidfMLPClfWorkflow(preprocessors=preprocessors, postprocessors=[]) | ||
| output = workflow.run([[{"transcript": "hello world", "confidence": 0.97}]]) # output -> _greeting_ | ||
| optional arguments: | ||
| -h, --help show this help message and exit | ||
| ``` | ||
|
|
||
| Refer to the source for [`merge_asr_output`](https://vernacular-ai.github.io/dialogy/dialogy/preprocess/text/merge_asr_output.html) and [`Plugins`](https://vernacular-ai.github.io/dialogy/dialogy/plugin/plugin.html) to understand this example better. | ||
| ## Project Creation | ||
|
|
||
| ## Note | ||
| ```txt | ||
| dialogy create -h | ||
| usage: dialogy create [-h] [--template TEMPLATE] [--dry-run] [--namespace NAMESPACE] [--master] project | ||
| - Popular workflow sub-classes will be accepted after code-review. | ||
| ## FAQs | ||
| positional arguments: | ||
| project A directory with this name will be created at the root of command invocation. | ||
| ### Training boilerplate | ||
| optional arguments: | ||
| -h, --help show this help message and exit | ||
| --template TEMPLATE | ||
| --dry-run Make no change to the directory structure. | ||
| --namespace NAMESPACE | ||
| The github/gitlab user or organization name where the template project lies. | ||
| --master Download the template's master branch (HEAD) instead of the latest tag. | ||
| ``` | ||
|
|
||
| ❌. This is not an end-to-end automated model training framework. That said, no one wants to write boilerplate code, | ||
| unfortunately, it doesn't align with the objectives of this project. Also, it is hard to accomodate for different needs | ||
| like: | ||
| ## Concepts | ||
|
|
||
| - library dependencies | ||
| - hardware support | ||
| - Need for visualizations/reports during/after training. | ||
| There are a few key concepts to build a machine-learning `Workflow`(s) using Dialogy. | ||
| All the effects comprising pre-processing, classification, scoring, ranking, etc are governed by `Plugin`(s). | ||
|
|
||
| Any rigidity here would lead to distractions both to maintainers and users of this project. [`Plugins`](https://vernacular-ai.github.io/dialogy/dialogy/plugin/plugin.html) and custom | ||
| [Workflow](https://vernacular-ai.github.io/dialogy/dialogy/workflow/workflow.html) are certainly welcome and can take care of recipe-based needs. | ||
| ### Workflow | ||
|
|
||
| ### Common Evaluation Plans | ||
| A workflow has these objectives. | ||
|
|
||
| ❌. Evaluation of models is hard to standardize. if you have a fairly common need, feel free to contribute your `workflow`, `plugins`. | ||
| 1. Allow interactions between different plugins. | ||
|
|
||
| ### Benefits | ||
| 2. Isolating plugins from each other. A plugin can't access another in the chain. | ||
|
|
||
| - ✅. This project offers a conduit for an untrained model. This means once a [Workflow](https://vernacular-ai.github.io/dialogy/dialogy/workflow/workflow.html) is ready you can use it anywhere: | ||
| evaluation scripts, serving your models via an API, custom training/evaluation scripts, combining another workflow, etc. | ||
| 3. Storing the information till the end of the execution of plugin chain. | ||
|
|
||
| - ✅. If your application needs spoken language understanding, you should find little need to write data processing functions. | ||
| ### Plugin | ||
|
|
||
| - ✅. Little to no learning curve, if you know python you know how to use this project. | ||
| A plugin transforms data. Depending on its utility, a plugin may have phases of operation. | ||
|
|
||
| ## Contributions | ||
| - Bulk transformation during training phase. | ||
|
|
||
| - Go through the docs. | ||
| - Inference or Transformation | ||
|
|
||
| - Name your branch as "purpose/short-description". examples: | ||
| - "feature/hippos_can_fly" | ||
| - "fix/breathing_water_instead_of_fire" | ||
| - "docs/chapter_on_mighty_sphinx" | ||
| - "refactor/limbs_of_mummified_pharao" | ||
| - "test/my_patience" | ||
| ## Test | ||
|
|
||
| - Make sure tests are added are passing. Run `make test lint` at project root. Coverage is important aim for 100% unless reviewer agrees. | ||
| ```shell | ||
| make test | ||
| ``` |