Skip to content

Commit

Permalink
deploy: 97e4cbf
Browse files Browse the repository at this point in the history
  • Loading branch information
@scarmuega
scarmuega committed Mar 18, 2024
1 parent 652aea4 commit 8efb6b3
Show file tree
Hide file tree
Showing 11 changed files with 891 additions and 9 deletions.
2 changes: 1 addition & 1 deletion 404.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@

<nav id="sidebar" class="sidebar" aria-label="Table of contents">
<div class="sidebar-scrollbox">
<ol class="chapter"><li class="chapter-item expanded "><a href="introduction.html"><strong aria-hidden="true">1.</strong> Introduction</a></li><li class="chapter-item expanded "><a href="reducers/index.html"><strong aria-hidden="true">2.</strong> Reducers</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="reducers/typescript.html"><strong aria-hidden="true">2.1.</strong> Typescript</a></li><li class="chapter-item expanded "><a href="reducers/rust.html"><strong aria-hidden="true">2.2.</strong> Rust</a></li></ol></li></ol>
<ol class="chapter"><li class="chapter-item expanded "><a href="introduction.html"><strong aria-hidden="true">1.</strong> Introduction</a></li><li class="chapter-item expanded "><a href="reducers/index.html"><strong aria-hidden="true">2.</strong> Reducers</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="reducers/typescript.html"><strong aria-hidden="true">2.1.</strong> Typescript</a></li><li class="chapter-item expanded "><a href="reducers/rust.html"><strong aria-hidden="true">2.2.</strong> Rust</a></li><li class="chapter-item expanded "><a href="reducers/golang.html"><strong aria-hidden="true">2.3.</strong> Golang</a></li><li class="chapter-item expanded "><a href="reducers/python.html"><strong aria-hidden="true">2.4.</strong> Python</a></li></ol></li></ol>
</div>
<div id="sidebar-resize-handle" class="sidebar-resize-handle">
<div class="sidebar-resize-indicator"></div>
Expand Down
2 changes: 1 addition & 1 deletion index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@

<nav id="sidebar" class="sidebar" aria-label="Table of contents">
<div class="sidebar-scrollbox">
<ol class="chapter"><li class="chapter-item expanded "><a href="introduction.html" class="active"><strong aria-hidden="true">1.</strong> Introduction</a></li><li class="chapter-item expanded "><a href="reducers/index.html"><strong aria-hidden="true">2.</strong> Reducers</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="reducers/typescript.html"><strong aria-hidden="true">2.1.</strong> Typescript</a></li><li class="chapter-item expanded "><a href="reducers/rust.html"><strong aria-hidden="true">2.2.</strong> Rust</a></li></ol></li></ol>
<ol class="chapter"><li class="chapter-item expanded "><a href="introduction.html" class="active"><strong aria-hidden="true">1.</strong> Introduction</a></li><li class="chapter-item expanded "><a href="reducers/index.html"><strong aria-hidden="true">2.</strong> Reducers</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="reducers/typescript.html"><strong aria-hidden="true">2.1.</strong> Typescript</a></li><li class="chapter-item expanded "><a href="reducers/rust.html"><strong aria-hidden="true">2.2.</strong> Rust</a></li><li class="chapter-item expanded "><a href="reducers/golang.html"><strong aria-hidden="true">2.3.</strong> Golang</a></li><li class="chapter-item expanded "><a href="reducers/python.html"><strong aria-hidden="true">2.4.</strong> Python</a></li></ol></li></ol>
</div>
<div id="sidebar-resize-handle" class="sidebar-resize-handle">
<div class="sidebar-resize-indicator"></div>
Expand Down
2 changes: 1 addition & 1 deletion introduction.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@

<nav id="sidebar" class="sidebar" aria-label="Table of contents">
<div class="sidebar-scrollbox">
<ol class="chapter"><li class="chapter-item expanded "><a href="introduction.html" class="active"><strong aria-hidden="true">1.</strong> Introduction</a></li><li class="chapter-item expanded "><a href="reducers/index.html"><strong aria-hidden="true">2.</strong> Reducers</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="reducers/typescript.html"><strong aria-hidden="true">2.1.</strong> Typescript</a></li><li class="chapter-item expanded "><a href="reducers/rust.html"><strong aria-hidden="true">2.2.</strong> Rust</a></li></ol></li></ol>
<ol class="chapter"><li class="chapter-item expanded "><a href="introduction.html" class="active"><strong aria-hidden="true">1.</strong> Introduction</a></li><li class="chapter-item expanded "><a href="reducers/index.html"><strong aria-hidden="true">2.</strong> Reducers</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="reducers/typescript.html"><strong aria-hidden="true">2.1.</strong> Typescript</a></li><li class="chapter-item expanded "><a href="reducers/rust.html"><strong aria-hidden="true">2.2.</strong> Rust</a></li><li class="chapter-item expanded "><a href="reducers/golang.html"><strong aria-hidden="true">2.3.</strong> Golang</a></li><li class="chapter-item expanded "><a href="reducers/python.html"><strong aria-hidden="true">2.4.</strong> Python</a></li></ol></li></ol>
</div>
<div id="sidebar-resize-handle" class="sidebar-resize-handle">
<div class="sidebar-resize-indicator"></div>
Expand Down
216 changes: 215 additions & 1 deletion print.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@

<nav id="sidebar" class="sidebar" aria-label="Table of contents">
<div class="sidebar-scrollbox">
<ol class="chapter"><li class="chapter-item expanded "><a href="introduction.html"><strong aria-hidden="true">1.</strong> Introduction</a></li><li class="chapter-item expanded "><a href="reducers/index.html"><strong aria-hidden="true">2.</strong> Reducers</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="reducers/typescript.html"><strong aria-hidden="true">2.1.</strong> Typescript</a></li><li class="chapter-item expanded "><a href="reducers/rust.html"><strong aria-hidden="true">2.2.</strong> Rust</a></li></ol></li></ol>
<ol class="chapter"><li class="chapter-item expanded "><a href="introduction.html"><strong aria-hidden="true">1.</strong> Introduction</a></li><li class="chapter-item expanded "><a href="reducers/index.html"><strong aria-hidden="true">2.</strong> Reducers</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="reducers/typescript.html"><strong aria-hidden="true">2.1.</strong> Typescript</a></li><li class="chapter-item expanded "><a href="reducers/rust.html"><strong aria-hidden="true">2.2.</strong> Rust</a></li><li class="chapter-item expanded "><a href="reducers/golang.html"><strong aria-hidden="true">2.3.</strong> Golang</a></li><li class="chapter-item expanded "><a href="reducers/python.html"><strong aria-hidden="true">2.4.</strong> Python</a></li></ol></li></ol>
</div>
<div id="sidebar-resize-handle" class="sidebar-resize-handle">
<div class="sidebar-resize-indicator"></div>
Expand Down Expand Up @@ -225,6 +225,220 @@ <h3 id="section-reducer-1"><a class="header" href="#section-reducer-1">Section:
<h2 id="run-code-1"><a class="header" href="#run-code-1">Run code</a></h2>
<p>To run a custom WASM reducer, it will be necessary to trigger scrolls enabling the <code>wasm</code> feature</p>
<pre><code class="language-sh">cargo run --features=wasm -- daemon --config ./examples/wasm/daemon.toml
</code></pre>
<div style="break-before: page; page-break-before: always;"></div><h1 id="building-a-golang-reducer"><a class="header" href="#building-a-golang-reducer">Building a Golang Reducer</a></h1>
<p>This guide explains how to use Golang to build a custom reducer for Scrolls.</p>
<h2 id="how-it-works"><a class="header" href="#how-it-works">How it works</a></h2>
<p>To build our custom reducer we'll leverage an <a href="https://github.com/txpipe/oura">Oura</a> pipeline that sync data from a Cardano node, filters the data through a Wasm plugin and finally persists the records in an Sqlite db.</p>
<h2 id="requirements"><a class="header" href="#requirements">Requirements</a></h2>
<ul>
<li>Scrolls SDK</li>
<li><a href="https://tinygo.org/getting-started/install/">tinygo</a></li>
<li>sqlite</li>
<li>make</li>
<li><a href="https://github.com/txpipe/oura">Oura</a></li>
</ul>
<h2 id="procedure"><a class="header" href="#procedure">Procedure</a></h2>
<h3 id="1-create-the-project-scaffold"><a class="header" href="#1-create-the-project-scaffold">1. Create the project scaffold</a></h3>
<p>There's some boilerplate code required to setup our reducer. The <em>Scrolls SDK</em> cli provides a command to automatically generate the basic file structure that can be later customized to specific needs.</p>
<p>Run the following command in your shell to scaffold a new Golang reducer.</p>
<pre><code class="language-sh">scrolls-sdk scrolls-sdk scaffold --template reducer-oura-golang {NAME}
</code></pre>
<p>where:</p>
<ul>
<li><code>{NAME}</code> is the name of your reducer</li>
</ul>
<h3 id="2-customize-your-db-schema"><a class="header" href="#2-customize-your-db-schema">2. Customize your DB schema</a></h3>
<p>Your custom reducer will require a custom DB schema that reflects your particular requirements. In this scenario we're using Sqlite as our relational data persistence mechanism.</p>
<p>Edit the <code>init.sql</code> file inside your reducer code to define your schema using SQL:</p>
<pre><code class="language-sql">CREATE TABLE my_reducer (
slot INTEGER NOT NULL,
{{custom fields}}
);

CREATE INDEX idx_my_reducer_slot ON my_reducer(slot);
</code></pre>
<p>where:</p>
<ul>
<li><code>{{custom fields}}</code> is the definition of the custom fields required by your reducer</li>
</ul>
<h3 id="3-edit-your-golang-reducer-logic"><a class="header" href="#3-edit-your-golang-reducer-logic">3. Edit your Golang reducer logic</a></h3>
<p>The core of your reducer is the business logic you use to map blocks &amp; transactions from the chain into relevant data for your use case. In this scenario we're using Golang code that compiles to WASM.</p>
<p>Edit the <code>main.go</code> file inside your reducer code define your business logic:</p>
<pre><code class="language-go">package main

import (
"github.com/extism/go-pdk"
)

//export map_u5c_tx
func map_u5c_tx() int32 {
// unmarshal the U5C Tx data provided by the host
var param map[string]interface{}
err := pdk.InputJSON(&amp;param)

if err != nil {
pdk.SetError(err)
return 1
}

// you can log info to see it in the debug output of
// the pipeline:
//pdk.Log(pdk.LogInfo, fmt.Sprintf("%v", param))

// Here is where you get to do something interesting
// with the data. In this example, we just extract the
// fee data from the Tx:
// fee := param["fee"].(interface{})

// As default, we just resend the exact same payload
output := param

// Use this method to return the mapped value back to
// the Oura pipeline.
err = pdk.OutputJSON(output)

if err != nil {
pdk.SetError(err)
return 1
}

// return 0 for a successful operation and 1 for
// failure.
return 0
}

// you need to keep the main entry point even if we
// don't use it
func main() {}
</code></pre>
<h3 id="4-edit-the-oura-config-file"><a class="header" href="#4-edit-the-oura-config-file">4. Edit the <em>Oura</em> config file</a></h3>
<p>The Cardano node, the WASM plugin and the Sqlite DB is connected together via an <em>Oura</em> pipeline. This pipeline will ensure that data goes through each the required steps (stages) in performant and resilient way.</p>
<p>Edit the <code>oura.toml</code> in your custom reducer folder to configure your pipeline:</p>
<pre><code class="language-toml">[source]
type = "N2N"
peers = ["relays-new.cardano-mainnet.iohk.io:3001"]

[[filters]]
type = "SplitBlock"

[[filters]]
type = "ParseCbor"

[[filters]]
type = "WasmPlugin"
path = "plugin.wasm"

[sink]
type = "SqlDb"
connection = "sqilte:./scrolls.db"
apply_template = "INSERT INTO {{reducer}} (slot, {{custom fields}}) VALUES ('\{{point.slot}}', '{{custom values}}');"
undo_template = "DELETE FROM {{reducer}} WHERE slot = \{{point.slot}}"
reset_template = "DELETE FROM {{reducer}} WHERE slot &gt; \{{point.slot}}"
</code></pre>
<p>where:</p>
<ul>
<li><code>{{custom fields}}</code> is the list of custom fields in your schema</li>
<li><code>{{custom values}}</code> is the expressions to access the custom values in the object returned by your custom logic</li>
</ul>
<ol start="5">
<li>Run your pipeline</li>
</ol>
<p>Now that everything has been configured, you can start your indexing pipeline. This requires that you compile your Golang code into wasm and that the sqlite db is created with the corresponding schema. With all requirements in place, the Oura process can be started.</p>
<p>The <code>Makefile</code> provided in your custom reducer files provides a shortcut to trigger the pipeline making sure that all requirements are in place.</p>
<p>Run the following command:</p>
<pre><code class="language-sh">make run
</code></pre>
<div style="break-before: page; page-break-before: always;"></div><h1 id="building-a-python-reducer"><a class="header" href="#building-a-python-reducer">Building a Python Reducer</a></h1>
<p>This guide explains how to use Python to build a custom reducer for Scrolls.</p>
<h2 id="how-it-works-1"><a class="header" href="#how-it-works-1">How it works</a></h2>
<p>To build our custom reducer we'll leverage an <a href="https://github.com/txpipe/oura">Oura</a> pipeline that sync data from a Cardano node, filters the data through a Rust-based Python interpreter plugin and finally persists the records in an Sqlite db.</p>
<h2 id="requirements-1"><a class="header" href="#requirements-1">Requirements</a></h2>
<ul>
<li>Scrolls SDK</li>
<li><a href="https://github.com/RustPython/RustPython">RustPython</a></li>
<li>sqlite</li>
<li>make</li>
<li><a href="https://github.com/txpipe/oura">Oura</a></li>
</ul>
<h2 id="procedure-1"><a class="header" href="#procedure-1">Procedure</a></h2>
<h3 id="1-create-the-project-scaffold-1"><a class="header" href="#1-create-the-project-scaffold-1">1. Create the project scaffold</a></h3>
<p>There's some boilerplate code required to setup our reducer. The <em>Scrolls SDK</em> cli provides a command to automatically generate the basic file structure that can be later customized to specific needs.</p>
<p>Run the following command in your shell to scaffold a new Python reducer.</p>
<pre><code class="language-sh">scrolls-sdk scrolls-sdk scaffold --template reducer-oura-python {NAME}
</code></pre>
<p>where:</p>
<ul>
<li><code>{NAME}</code> is the name of your reducer</li>
</ul>
<h3 id="2-customize-your-db-schema-1"><a class="header" href="#2-customize-your-db-schema-1">2. Customize your DB schema</a></h3>
<p>Your custom reducer will require a custom DB schema that reflects your particular requirements. In this scenario we're using Sqlite as our relational data persistence mechanism.</p>
<p>Edit the <code>init.sql</code> file inside your reducer code to define your schema using SQL:</p>
<pre><code class="language-sql">CREATE TABLE my_reducer (
slot INTEGER NOT NULL,
{{custom fields}}
);

CREATE INDEX idx_my_reducer_slot ON my_reducer(slot);
</code></pre>
<p>where:</p>
<ul>
<li><code>{{custom fields}}</code> is the definition of the custom fields required by your reducer</li>
</ul>
<h3 id="3-edit-your-python-reducer-logic"><a class="header" href="#3-edit-your-python-reducer-logic">3. Edit your Python reducer logic</a></h3>
<p>The core of your reducer is the business logic you use to map blocks &amp; transactions from the chain into relevant data for your use case. In this scenario we're using Python code that will be interpreted using <em>RustPython</em>.</p>
<p>Edit the <code>main.py</code> file inside your reducer code define your business logic:</p>
<pre><code class="language-python">def map_u5c_tx(tx):
# the Tx param holds the data of the tx to map


# Here is where you get to do something interesting
# with the data. In this example, we just extract the
# fee data from the Tx:
# fee = tx["fee"]

# As default, we just resend the exact same payload
output := param

# Return a new Dict that holds the data that will
# continue down through the Oura pipeline
return output
}
</code></pre>
<h3 id="4-edit-the-oura-config-file-1"><a class="header" href="#4-edit-the-oura-config-file-1">4. Edit the <em>Oura</em> config file</a></h3>
<p>The Cardano node, the WASM plugin and the Sqlite DB is connected together via an <em>Oura</em> pipeline. This pipeline will ensure that data goes through each the required steps (stages) in performant and resilient way.</p>
<p>Edit the <code>oura.toml</code> in your custom reducer folder to configure your pipeline:</p>
<pre><code class="language-toml">[source]
type = "N2N"
peers = ["relays-new.cardano-mainnet.iohk.io:3001"]

[[filters]]
type = "SplitBlock"

[[filters]]
type = "ParseCbor"

[[filters]]
type = "PythonPlugin"
path = "main.py"

[sink]
type = "SqlDb"
connection = "sqilte:./scrolls.db"
apply_template = "INSERT INTO {{reducer}} (slot, {{custom fields}}) VALUES ('\{{point.slot}}', '{{custom values}}');"
undo_template = "DELETE FROM {{reducer}} WHERE slot = \{{point.slot}}"
reset_template = "DELETE FROM {{reducer}} WHERE slot &gt; \{{point.slot}}"
</code></pre>
<p>where:</p>
<ul>
<li><code>{{custom fields}}</code> is the list of custom fields in your schema</li>
<li><code>{{custom values}}</code> is the expressions to access the custom values in the object returned by your custom logic</li>
</ul>
<h3 id="5-run-your-pipeline"><a class="header" href="#5-run-your-pipeline">5. Run your pipeline</a></h3>
<p>Now that everything has been configured, you can start your indexing pipeline. This requires that that the sqlite db is created with the corresponding schema. With all requirements in place, the Oura process can be started.</p>
<p>The <code>Makefile</code> provided in your custom reducer files provides a shortcut to trigger the pipeline making sure that all requirements are in place.</p>
<p>Run the following command:</p>
<pre><code class="language-sh">make run
</code></pre>

</main>
Expand Down
Loading

0 comments on commit 8efb6b3

Please sign in to comment.