Data: BANKS, Character table, Feature docs
Features:
Two quotes from Consider Phlebas by Iain M. Banks
authorgap
letters
number
otype
punc
terminator
title
oslots
Text-Fabric API: names N F E L T S C TF directly usable
" + ], + "text/plain": [ + "
| n | word | word |
|---|---|---|
| 1 | Consider Phlebas 1:1 Everything | Consider Phlebas 1:1 everything |
| 2 | Consider Phlebas 1:1 Everything | Consider Phlebas 1:1 everything |
| 3 | Consider Phlebas 1:1 us, | Consider Phlebas 1:1 us, |
| 4 | Consider Phlebas 1:1 everything | Consider Phlebas 1:1 Everything |
| 5 | Consider Phlebas 1:1 everything | Consider Phlebas 1:1 everything |
| 6 | Consider Phlebas 1:1 us, | Consider Phlebas 1:1 us, |
| 7 | Consider Phlebas 1:1 everything | Consider Phlebas 1:1 Everything |
| 8 | Consider Phlebas 1:1 everything | Consider Phlebas 1:1 everything |
| 9 | Consider Phlebas 1:1 we | Consider Phlebas 1:2 we |
| 10 | Consider Phlebas 1:1 we | Consider Phlebas 1:2 we |
result 1
" + ], + "text/plain": [ + "Consider Phlebas 1:1
sentence
line
Everything
about
us,
line
everything
around
us,
line
everything
we
know
and
can
know
of
line
is
composed
ultimately
of
patterns
of
nothing;
line
that’s
the
bottom
line,
the
final
truth.
result 2
" + ], + "text/plain": [ + "Consider Phlebas 1:1
sentence
line
Everything
about
us,
line
everything
around
us,
line
everything
we
know
and
can
know
of
line
is
composed
ultimately
of
patterns
of
nothing;
line
that’s
the
bottom
line,
the
final
truth.
result 3
" + ], + "text/plain": [ + "Consider Phlebas 1:1
sentence
line
Everything
about
us,
line
everything
around
us,
line
everything
we
know
and
can
know
of
line
is
composed
ultimately
of
patterns
of
nothing;
line
that’s
the
bottom
line,
the
final
truth.
result 4
" + ], + "text/plain": [ + "Consider Phlebas 1:1
sentence
line
Everything
about
us,
line
everything
around
us,
line
everything
we
know
and
can
know
of
line
is
composed
ultimately
of
patterns
of
nothing;
line
that’s
the
bottom
line,
the
final
truth.
result 5
" + ], + "text/plain": [ + "Consider Phlebas 1:1
sentence
line
Everything
about
us,
line
everything
around
us,
line
everything
we
know
and
can
know
of
line
is
composed
ultimately
of
patterns
of
nothing;
line
that’s
the
bottom
line,
the
final
truth.
+
+
+
+---
+Start with [convert](https://nbviewer.jupyter.org/github/annotation/banks/blob/master/programs/convert.ipynb)
+
+---
+
+
+# The Banks example corpus as app
+
+```python
+%load_ext autoreload
+%autoreload 2
+```
+
+```python
+from tf.app import use
+```
+
+We do not only load the main corpus data, but also the additional *sim* (similarity) feature that is in a
+module.
+
+For the very last version, use `hot`.
+
+For the latest release, use `latest`.
+
+If you have cloned the repos (TF app and data), use `clone`.
+
+If you do not want/need to upgrade, leave out the checkout specifiers.
+
+```python
+A = use(
+ "banks:clone",
+ checkout="clone",
+ mod="annotation/banks/sim/tf:clone",
+ hoist=globals(),
+)
+# A = use('banks:hot', checkout="hot", mod='annotation/banks/sim/tf:hot', hoist=globals())
+# A = use('banks:latest', checkout="latest", mod='annotation/banks/sim/tf:latest', hoist=globals())
+# A = use('banks', mod='annotation/banks/sim/tf', hoist=globals())
+```
+
+# Use the similarity edge feature
+
+We print all similar pairs of words that are at least 50% similar but not 100%.
+
+```python
+query = """
+word
+
+#
+#
+#
+# ---
+# Start with [convert](https://nbviewer.jupyter.org/github/annotation/banks/blob/master/programs/convert.ipynb)
+#
+# ---
+
+# # The Banks example corpus as app
+
+# %load_ext autoreload
+# %autoreload 2
+
+from tf.app import use
+
+# We do not only load the main corpus data, but also the additional *sim* (similarity) feature that is in a
+# module.
+#
+# For the very last version, use `hot`.
+#
+# For the latest release, use `latest`.
+#
+# If you have cloned the repos (TF app and data), use `clone`.
+#
+# If you do not want/need to upgrade, leave out the checkout specifiers.
+
+A = use(
+ "banks:clone",
+ checkout="clone",
+ mod="annotation/banks/sim/tf:clone",
+ hoist=globals(),
+)
+# A = use('banks:hot', checkout="hot", mod='annotation/banks/sim/tf:hot', hoist=globals())
+# A = use('banks:latest', checkout="latest", mod='annotation/banks/sim/tf:latest', hoist=globals())
+# A = use('banks', mod='annotation/banks/sim/tf', hoist=globals())
+
+# # Use the similarity edge feature
+#
+# We print all similar pairs of words that are at least 50% similar but not 100%.
+
+query = """
+word
+![](\"images/tf-small.png\")
\n",
+ "![](\"images/phblogo.png\")
\n",
+ "![](\"images/dans.png\"/)
\n",
+ "\n",
+ "---\n",
+ "Start with [convert](https://nbviewer.jupyter.org/github/annotation/banks/blob/master/programs/convert.ipynb)\n",
+ "\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Compose\n",
+ "\n",
+ "This is about combining multiple TF datasets into one, and then tweaking it further.\n",
+ "\n",
+ "In the previous chapters of this tutorial you have learned how to add new features to an existing dataset.\n",
+ "\n",
+ "Here you learn how you can combine dozens of slightly heterogeneous TF data sets,\n",
+ "and apply structural tweaks to the node types and features later on.\n",
+ "\n",
+ "The incentive to write these composition functions into Text-Fabric came from Ernst Boogert while he was\n",
+ "converting between 100 and 200 works by the Church Fathers (Patristics).\n",
+ "The conversion did a very good job in getting all the information from TEI files with different structures into TF,\n",
+ "one dataset per work.\n",
+ "\n",
+ "Then the challenge became to combine them into one big dataset, and to merge several node types into one type,\n",
+ "and several features into one.\n",
+ "\n",
+ "See [patristics](https://github.com/pthu/patristics)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%load_ext autoreload\n",
+ "%autoreload 2"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The new functions are `collect()` and `modify()`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from tf.fabric import Fabric\n",
+ "from tf.dataset import modify\n",
+ "from tf.volumes import collect"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Corpus\n",
+ "\n",
+ "We use two copies of our example corpus Banks, present in this repository."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "## Collect\n",
+ "\n",
+ "The *collect* function takes any number of directory locations, and considers each location to be the\n",
+ "host of a TF data set.\n",
+ "\n",
+ "You can pass this list straight to the `collect()` function as the `locations` parameter,\n",
+ "or you can add names to the individual corpora.\n",
+ "In that case, you pass an iterable of (`name`, `location`) pairs into the `locations` parameter.\n",
+ "\n",
+ "Here we give the first copy the name `banks`, and the second copy the name `river`.\n",
+ "\n",
+ "We also specify the output location."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "PREFIX = \"combine/input\"\n",
+ "SUFFIX = \"tf/0.2\"\n",
+ "\n",
+ "locations = (\n",
+ " (\"banks\", f\"{PREFIX}/banks1/{SUFFIX}\"),\n",
+ " (\"rivers\", f\"{PREFIX}/banks2/{SUFFIX}\"),\n",
+ ")\n",
+ "\n",
+ "COMBINED = \"combine/_temp/riverbanks\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We are going to call the `collect()` function.\n",
+ "\n",
+ "But first we clear the output location.\n",
+ "\n",
+ "Note how you can mix a bash-shell command with your Python code."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ " 0.00s Loading volume banks from combine/input/banks1/tf/0.2 ...\n",
+ "This is Text-Fabric 9.1.3\n",
+ "Api reference : https://annotation.github.io/text-fabric/tf/cheatsheet.html\n",
+ "\n",
+ "10 features found and 0 ignored\n",
+ " 0.00s loading features ...\n",
+ " 0.01s All features loaded/computed - for details use TF.isLoaded()\n",
+ " | 0.00s Feature overview: 8 for nodes; 1 for edges; 1 configs; 8 computed\n",
+ " 0.00s loading features ...\n",
+ " 0.00s All additional features loaded - for details use TF.isLoaded()\n",
+ " 0.02s Loading volume rivers from combine/input/banks2/tf/0.2 ...\n",
+ "This is Text-Fabric 9.1.3\n",
+ "Api reference : https://annotation.github.io/text-fabric/tf/cheatsheet.html\n",
+ "\n",
+ "10 features found and 0 ignored\n",
+ " 0.00s loading features ...\n",
+ " 0.01s All features loaded/computed - for details use TF.isLoaded()\n",
+ " | 0.00s Feature overview: 8 for nodes; 1 for edges; 1 configs; 8 computed\n",
+ " 0.00s loading features ...\n",
+ " 0.00s All additional features loaded - for details use TF.isLoaded()\n",
+ " 0.04s inspect metadata ...\n",
+ "WARNING: otext.structureFeatures metadata varies across volumes\n",
+ "WARNING: otext.structureTypes metadata varies across volumes\n",
+ "WARNING: author.compiler metadata varies across volumes\n",
+ "WARNING: author.purpose metadata varies across volumes\n",
+ "WARNING: letters.description metadata varies across volumes\n",
+ " 0.04s metadata sorted out\n",
+ " 0.04s check nodetypes ...\n",
+ " | volume banks\n",
+ " | volume rivers\n",
+ " 0.04s node types ok\n",
+ " 0.04s Collect nodes from volumes ...\n",
+ " | 0.00s Check against overlapping slots ...\n",
+ " | | banks : 99 slots\n",
+ " | | rivers : 99 slots\n",
+ " | 0.00s no overlap\n",
+ " | 0.00s Group non-slot nodes by type\n",
+ " | | banks : 100- 117\n",
+ " | | rivers : 100- 117\n",
+ " | 0.00s Mapping nodes from volume to/from work ...\n",
+ " | | book : 199 - 200\n",
+ " | | chapter : 201 - 204\n",
+ " | | line : 205 - 228\n",
+ " | | sentence : 229 - 234\n",
+ " | 0.01s The new work has 236 nodes of which 198 slots\n",
+ " 0.05s collection done\n",
+ " 0.05s remap features ...\n",
+ " 0.05s remapping done\n",
+ " 0.05s write work as TF data set\n",
+ " 0.07s writing done\n",
+ " 0.07s done\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "True"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "output = COMBINED\n",
+ "\n",
+ "!rm -rf {output}\n",
+ "\n",
+ "collect(\n",
+ " locations,\n",
+ " output,\n",
+ " volumeType=\"volume\",\n",
+ " volumeFeature=\"title\",\n",
+ " featureMeta=dict(\n",
+ " otext=dict(\n",
+ " sectionTypes=\"volume,chapter,line\",\n",
+ " sectionFeatures=\"title,number,number\",\n",
+ " **{\"fmt:text-orig-full\": \"{letters} \"},\n",
+ " ),\n",
+ " ),\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "This function is a bit verbose in its output, but a lot happens under the hood, and if your dataset is large,\n",
+ "it may take several minutes. It is pleasant to see the progress under those circumstances.\n",
+ "\n",
+ "But for now, we pass `silent=True`, to make everything a bit more quiet."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "WARNING: otext.structureFeatures metadata varies across volumes\n",
+ "WARNING: otext.structureTypes metadata varies across volumes\n",
+ "WARNING: author.compiler metadata varies across volumes\n",
+ "WARNING: author.purpose metadata varies across volumes\n",
+ "WARNING: letters.description metadata varies across volumes\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "True"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "output = COMBINED\n",
+ "\n",
+ "!rm -rf {output}\n",
+ "\n",
+ "collect(\n",
+ " locations,\n",
+ " output,\n",
+ " volumeType=\"volume\",\n",
+ " volumeFeature=\"title\",\n",
+ " featureMeta=dict(\n",
+ " otext=dict(\n",
+ " sectionTypes=\"volume,chapter,line\",\n",
+ " sectionFeatures=\"title,number,number\",\n",
+ " **{\"fmt:text-orig-full\": \"{letters} \"},\n",
+ " ),\n",
+ " ),\n",
+ " silent=True,\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "There you are, on your file system you see the combined dataset:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "total 88\n",
+ "-rw-r--r-- 1 dirk staff 559 Nov 4 16:04 author.tf\n",
+ "-rw-r--r-- 1 dirk staff 524 Nov 4 16:04 gap.tf\n",
+ "-rw-r--r-- 1 dirk staff 1619 Nov 4 16:04 letters.tf\n",
+ "-rw-r--r-- 1 dirk staff 548 Nov 4 16:04 number.tf\n",
+ "-rw-r--r-- 1 dirk staff 681 Nov 4 16:04 oslots.tf\n",
+ "-rw-r--r-- 1 dirk staff 1062 Nov 4 16:04 otext.tf\n",
+ "-rw-r--r-- 1 dirk staff 485 Nov 4 16:04 otype.tf\n",
+ "-rw-r--r-- 1 dirk staff 2747 Nov 4 16:04 ovolume.tf\n",
+ "-rw-r--r-- 1 dirk staff 640 Nov 4 16:04 punc.tf\n",
+ "-rw-r--r-- 1 dirk staff 494 Nov 4 16:04 terminator.tf\n",
+ "-rw-r--r-- 1 dirk staff 563 Nov 4 16:04 title.tf\n"
+ ]
+ }
+ ],
+ "source": [
+ "!ls -l {output}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If we compare that with one of the input:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "total 80\n",
+ "-rw-r--r-- 1 dirk staff 359 May 20 2019 author.tf\n",
+ "-rw-r--r-- 1 dirk staff 409 May 20 2019 gap.tf\n",
+ "-rw-r--r-- 1 dirk staff 911 May 20 2019 letters.tf\n",
+ "-rw-r--r-- 1 dirk staff 421 May 20 2019 number.tf\n",
+ "-rw-r--r-- 1 dirk staff 419 May 20 2019 oslots.tf\n",
+ "-rw-r--r-- 1 dirk staff 572 May 20 2019 otext.tf\n",
+ "-rw-r--r-- 1 dirk staff 372 May 30 2019 otype.tf\n",
+ "-rw-r--r-- 1 dirk staff 457 May 20 2019 punc.tf\n",
+ "-rw-r--r-- 1 dirk staff 377 May 20 2019 terminator.tf\n",
+ "-rw-r--r-- 1 dirk staff 361 May 20 2019 title.tf\n"
+ ]
+ }
+ ],
+ "source": [
+ "!ls -l {PREFIX}/banks1/{SUFFIX}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "then we see the same files (with the addition of *ovolume.tf*\n",
+ "but smaller file sizes.\n",
+ "\n",
+ "## Result\n",
+ "\n",
+ "Let's have a look inside, and note that we use the TF function `loadAll()`\n",
+ "which loads all loadable features."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "This is Text-Fabric 9.1.3\n",
+ "Api reference : https://annotation.github.io/text-fabric/tf/cheatsheet.html\n",
+ "\n",
+ "11 features found and 0 ignored\n",
+ " 0.00s loading features ...\n",
+ " | 0.00s T otype from combine/_temp/riverbanks\n",
+ " | 0.00s T oslots from combine/_temp/riverbanks\n",
+ " | 0.00s Dataset without structure sections in otext:no structure functions in the T-API\n",
+ " | 0.00s T number from combine/_temp/riverbanks\n",
+ " | 0.00s T punc from combine/_temp/riverbanks\n",
+ " | 0.00s T gap from combine/_temp/riverbanks\n",
+ " | 0.00s T terminator from combine/_temp/riverbanks\n",
+ " | 0.00s T title from combine/_temp/riverbanks\n",
+ " | 0.00s T letters from combine/_temp/riverbanks\n",
+ " | | 0.00s C __levels__ from otype, oslots, otext\n",
+ " | | 0.00s C __order__ from otype, oslots, __levels__\n",
+ " | | 0.00s C __rank__ from otype, __order__\n",
+ " | | 0.00s C __levUp__ from otype, oslots, __rank__\n",
+ " | | 0.00s C __levDown__ from otype, __levUp__, __rank__\n",
+ " | | 0.00s C __boundary__ from otype, oslots, __rank__\n",
+ " | | 0.00s C __sections__ from otype, oslots, otext, __levUp__, __levels__, title, number, number\n",
+ " 0.03s All features loaded/computed - for details use TF.isLoaded()\n",
+ " | 0.00s Feature overview: 9 for nodes; 1 for edges; 1 configs; 8 computed\n",
+ " 0.00s loading features ...\n",
+ " | 0.00s T author from combine/_temp/riverbanks\n",
+ " | 0.00s T ovolume from combine/_temp/riverbanks\n",
+ " 0.01s All additional features loaded - for details use TF.isLoaded()\n"
+ ]
+ }
+ ],
+ "source": [
+ "TF = Fabric(locations=COMBINED)\n",
+ "api = TF.loadAll(silent=False)\n",
+ "docs = api.makeAvailableIn(globals())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We look up the section of the first word:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "('banks', 1, 1)"
+ ]
+ },
+ "execution_count": 12,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "T.sectionFromNode(1)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The component sets had 99 words each. So what is the section of word 100?"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "('rivers', 1, 1)"
+ ]
+ },
+ "execution_count": 13,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "T.sectionFromNode(100)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Right, that's the first word of the second component.\n",
+ "\n",
+ "Here is an overview of all the node types in the combined set.\n",
+ "\n",
+ "The second field is the average length in words for nodes of that type, the remaining fields give\n",
+ "the first and last node of that type."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "(('book', 99.0, 199, 200),\n",
+ " ('volume', 99.0, 235, 236),\n",
+ " ('chapter', 49.5, 201, 204),\n",
+ " ('sentence', 33.0, 229, 234),\n",
+ " ('line', 7.666666666666667, 205, 228),\n",
+ " ('word', 1, 1, 198))"
+ ]
+ },
+ "execution_count": 14,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "C.levels.data"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The combined data set consists of the concatenation of all slot nodes of the component data sets.\n",
+ "\n",
+ "Note that the individual components have got a top node, of type `volume`.\n",
+ "This is the effect of specifying `componentType='volume'`.\n",
+ "\n",
+ "There is also a feature for volumes, named `title`, that contains their name, or if we haven't passed their names\n",
+ "in the `locations` parameter, their location.\n",
+ "This is the effect of `componentFeature='title'`.\n",
+ "\n",
+ "Let's check.\n",
+ "\n",
+ "We use the new `.items()` method on features."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "dict_items([(199, 'Consider Phlebas'), (200, 'Consider Phlebas'), (235, 'banks'), (236, 'rivers')])"
+ ]
+ },
+ "execution_count": 15,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "F.title.items()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We see several things:\n",
+ "\n",
+ "* the volume nodes indeed got the component name in the feature `title`\n",
+ "* the other nodes that already had a title, the `book` nodes, still have the same value for `title` as before.\n",
+ "\n",
+ "### The merging principle\n",
+ "\n",
+ "This is a general principle that we see over and over again: when we combine data, we merge as much as possible.\n",
+ "\n",
+ "That means that when you create new features, you may use the names of old features, and the new information for that\n",
+ "feature will be merged with the old information of that feature."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Modify\n",
+ "\n",
+ "Although combining has its complications, the most complex operation is `modify()` because it can do many things.\n",
+ "\n",
+ "It operates on a single TF dataset, and it produces a modified dataset as a fresh \"copy\".\n",
+ "\n",
+ "Despite the name, no actual modification takes place on the input dataset."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "location = f\"{PREFIX}/banks1/{SUFFIX}\"\n",
+ "\n",
+ "MODIFIED = \"_temp/mudbanks\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now we take the first local copy of the Banks dataset as our input, for a lot of different operations."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Here is the list what `modify()` can do.\n",
+ "The order is important, because all operations are executed in this order:\n",
+ "\n",
+ "1. **merge features**: several input features are combined into a single output feature and then deleted;\n",
+ "2. **delete features**: several features are be deleted\n",
+ "3. **add features**: several node/edge features with their data are added to the dataset\n",
+ "4. **merge types**: several input node types are combined into a single output node type;\n",
+ " the input nodetypes are deleted, but not their nodes: they are now part of the output node type;\n",
+ "5. **delete types**: several node types are deleted, *with their nodes*, and all features\n",
+ " will be remapped to accomodate for this;\n",
+ "6. **add types**: several new node types with additional feature data for them are added after the last node;\n",
+ " features do not have to be remapped for this; the new node types may be arbitrary intervals of integers and\n",
+ " have no relationship with the existing nodes.\n",
+ "7. **modify metadata**: the metadata of all features can be tweaked, including everything that is in the\n",
+ " `otext` feature, such as text formats and section structure definitions.\n",
+ "\n",
+ "Modify will perform as many sanity checks as possible before it starts working, so that the chances are good that\n",
+ "the modified dataset will load properly.\n",
+ "It will adapt the value type of features to the values encountered, and it will deduce whether edges have values or not.\n",
+ "\n",
+ "If a modified dataset does not load, while the original dataset did load, it is a bug, and I welcome a\n",
+ "[GitHub issue](https://github.com/annotation/text-fabric/issues)\n",
+ "for it."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Only meta data\n",
+ "\n",
+ "We start with the last one, the most simple one."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "otext = dict(\n",
+ " sectionTypes=\"book,chapter\",\n",
+ " sectionFeatures=\"title,number\",\n",
+ " **{\"fmt:text-orig-full\": \"{letters} \"},\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We use `silent=True` from now on, but if you work with larger datasets, it is recommended to set `silent=False` or\n",
+ "to leave it out altogether."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 20,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "True"
+ ]
+ },
+ "execution_count": 20,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "test = \"meta\"\n",
+ "output = f\"{MODIFIED}.{test}\"\n",
+ "\n",
+ "!rm -rf {output}\n",
+ "\n",
+ "modify(\n",
+ " location,\n",
+ " output,\n",
+ " featureMeta=dict(otext=otext),\n",
+ " silent=True,\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Result"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 21,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "TF = Fabric(locations=f\"{MODIFIED}.{test}\", silent=True)\n",
+ "api = TF.loadAll(silent=True)\n",
+ "docs = api.makeAvailableIn(globals())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We have now only 2 section levels. If we ask for some sections, we see that we only get 2 components in the tuple."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 22,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "('Consider Phlebas', 1)"
+ ]
+ },
+ "execution_count": 22,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "T.sectionFromNode(1)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 23,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "('Consider Phlebas', 2)"
+ ]
+ },
+ "execution_count": 23,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "T.sectionFromNode(99)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Merge features\n",
+ "\n",
+ "We are going to do some tricky mergers on features that are involved in the section structure and the\n",
+ "text formats, so we take care to modify those by means of the `featureMeta` parameter."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 24,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "otext = dict(\n",
+ " sectionTypes=\"book,chapter\",\n",
+ " sectionFeatures=\"heading,heading\",\n",
+ " structureTypes=\"book,chapter\",\n",
+ " structureFeatures=\"heading,heading\",\n",
+ " **{\n",
+ " \"fmt:text-orig-full\": \"{content} \",\n",
+ " \"fmt:text-orig-fake\": \"{fake} \",\n",
+ " \"fmt:line-default\": \"{content:XXX}{terminator} \",\n",
+ " },\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We want sectional headings in one feature, `heading`, instead of in `title` for books and `number` for chapters.\n",
+ "\n",
+ "We also make a `content` feature that gives the `letters` of a word unless there is punctuation: then it gives `punc`.\n",
+ "\n",
+ "And we make the opposite: `fake`: it prefers `punc` over `letters`.\n",
+ "\n",
+ "Note that `punc` and `letters` will be deleted after the merge as a whole is completed, so that it is indeed\n",
+ "possible for features to be the input of multiple mergers."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 25,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "True"
+ ]
+ },
+ "execution_count": 25,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "test = \"merge.f\"\n",
+ "output = f\"{MODIFIED}.{test}\"\n",
+ "\n",
+ "!rm -rf {output}\n",
+ "\n",
+ "modify(\n",
+ " location,\n",
+ " output,\n",
+ " mergeFeatures=dict(\n",
+ " heading=(\"title number\"), content=(\"punc letters\"), fake=(\"letters punc\")\n",
+ " ),\n",
+ " featureMeta=dict(\n",
+ " otext=otext,\n",
+ " ),\n",
+ " silent=True,\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Result"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 26,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "TF = Fabric(locations=f\"{MODIFIED}.{test}\", silent=True)\n",
+ "api = TF.loadAll(silent=True)\n",
+ "docs = api.makeAvailableIn(globals())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We inspect the new `heading` feature for a book and a chapter."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 27,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "'Consider Phlebas'"
+ ]
+ },
+ "execution_count": 27,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "b = F.otype.s(\"book\")[0]\n",
+ "F.heading.v(b)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 28,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "'1'"
+ ]
+ },
+ "execution_count": 28,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "c = F.otype.s(\"chapter\")[0]\n",
+ "F.heading.v(c)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "And here is an overview of all node features: `title` and `number` are gone, together with `punc` and `letters`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 29,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "['author', 'content', 'fake', 'gap', 'heading', 'otype', 'terminator']"
+ ]
+ },
+ "execution_count": 29,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "Fall()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We have modified the standard text format, `text-orig-full`. It now uses the `content` feature,\n",
+ "and indeed, we do not see punctuation anymore."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 30,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "'Everything about us everything around us everything we know '"
+ ]
+ },
+ "execution_count": 30,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "T.text(range(1, 10))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "On the other hand, `text-orig-fake` uses the `fake` feature, and we see that the words in front\n",
+ "of punctuation have disappeared."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 31,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "'Everything about , everything around , everything we know '"
+ ]
+ },
+ "execution_count": 31,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "T.text(range(1, 10), fmt=\"text-orig-fake\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Delete features\n",
+ "\n",
+ "We just remove two features from the dataset: `author` and `terminator`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 32,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ " | Missing for text API: features: terminator\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "False"
+ ]
+ },
+ "execution_count": 32,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "test = \"delete.f\"\n",
+ "output = f\"{MODIFIED}.{test}\"\n",
+ "\n",
+ "!rm -rf {output}\n",
+ "\n",
+ "modify(\n",
+ " location,\n",
+ " output,\n",
+ " deleteFeatures=\"author terminator\",\n",
+ " silent=True,\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Oops. `terminator` is used in a text-format, so if we delete it, the dataset will not load properly.\n",
+ "\n",
+ "Let's not delete `terminator` but `gap`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 33,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "True"
+ ]
+ },
+ "execution_count": 33,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "test = \"delete.f\"\n",
+ "output = f\"{MODIFIED}.{test}\"\n",
+ "\n",
+ "modify(\n",
+ " location,\n",
+ " output,\n",
+ " deleteFeatures=\"author gap\",\n",
+ " silent=True,\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Result"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 34,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "TF = Fabric(locations=f\"{MODIFIED}.{test}\", silent=True)\n",
+ "api = TF.loadAll(silent=True)\n",
+ "docs = api.makeAvailableIn(globals())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 35,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "['letters', 'number', 'otype', 'punc', 'terminator', 'title']"
+ ]
+ },
+ "execution_count": 35,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "Fall()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Indeed, `gap` is gone."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 36,
+ "metadata": {},
+ "outputs": [
+ {
+ "ename": "AttributeError",
+ "evalue": "'NodeFeatures' object has no attribute 'gap'",
+ "output_type": "error",
+ "traceback": [
+ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+ "\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)",
+ "\u001b[0;32m\n",
+ "\n",
+ "\n",
+ "\n",
+ "---\n",
+ "Start with [convert](https://nbviewer.jupyter.org/github/annotation/banks/blob/master/programs/convert.ipynb)\n",
+ "\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Getting data from online repos\n",
+ "\n",
+ "We show the various automatic ways by which you can get data that is out there on GitHub to your computer.\n",
+ "\n",
+ "The work horse is the function `checkoutRepo()` in `tf.applib.repo`.\n",
+ "\n",
+ "Text-Fabric uses this function for all operations where data flows from GitHub to your computer.\n",
+ "\n",
+ "There are quite some options, and here we explain all the `checkout` options, i.e. the selection of\n",
+ "data from the history.\n",
+ "\n",
+ "See also the [documentation](https://annotation.github.io/text-fabric/tf/advanced/repo.html)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%load_ext autoreload\n",
+ "%autoreload 2"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Leading example\n",
+ "\n",
+ "We use markdown display from IPython purely for presentation.\n",
+ "It is not needed to run `checkoutRepo()`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from tf.advanced.helpers import dm\n",
+ "from tf.advanced.repo import checkoutRepo"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We work with our tiny example TF app: `banks`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "lines_to_next_cell": 2
+ },
+ "outputs": [],
+ "source": [
+ "ORG = \"annotation\"\n",
+ "REPO = \"banks\"\n",
+ "MAIN = \"tf\"\n",
+ "MOD = \"sim/tf\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "`MAIN`points to the main data, `MOD` points to a module of data: the similarity feature."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "lines_to_next_cell": 2
+ },
+ "source": [
+ "## Presenting the results\n",
+ "\n",
+ "The function `do()` just formats the results of a `checkoutRepo()` run.\n",
+ "\n",
+ "The result of such a run, after the progress messages, is a tuple.\n",
+ "For the explanation of the tuple, read the [docs](https://annotation.github.io/text-fabric/tf/advanced/repo.html)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def do(task):\n",
+ " md = f\"\"\"\n",
+ "commit | release | local | base | subdir\n",
+ "--- | --- | --- | --- | ---\n",
+ "`{task[0]}` | `{task[1]}` | `{task[2]}` | `{task[3]}` | `{task[4]}`\n",
+ "\"\"\"\n",
+ " dm(md)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "incorrectly_encoded_metadata": "toc-hr-collapsed=false"
+ },
+ "source": [
+ "## All the checkout options\n",
+ "\n",
+ "We discuss the meaning and effects of the values you can pass to the `checkout` option."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### `clone`\n",
+ "\n",
+ "> Look whether the appropriate folder exists under your `~/github` directory.\n",
+ "\n",
+ "This is merely a check whether your data exists in the expected location.\n",
+ "\n",
+ "* No online checks take place.\n",
+ "* No data is moved or copied.\n",
+ "\n",
+ "**NB**: you cannot select releases and commits in your *local* GitHub clone.\n",
+ "The data will be used as it is found on your file system.\n",
+ "\n",
+ "**When to use**\n",
+ "\n",
+ "> If you are developing new feature data.\n",
+ "\n",
+ "When you develop your data in a repository, your development is private as long as you\n",
+ "do not push to GitHub.\n",
+ "\n",
+ "You can test your data, even without locally committing your data.\n",
+ "\n",
+ "But, if you are ready to share your data, everything is in place, and you only\n",
+ "have to commit and push, and pass the location on github to others, like\n",
+ "\n",
+ "```\n",
+ "myorg/myrepo/subfolder\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "data: ~/github/annotation/banks/tf/0.2"
+ ],
+ "text/plain": [
+ "
+#
+#
+#
+# ---
+# Start with [convert](https://nbviewer.jupyter.org/github/annotation/banks/blob/master/programs/convert.ipynb)
+#
+# ---
+
+# # Getting data from online repos
+#
+# We show the various automatic ways by which you can get data that is out there on GitHub to your computer.
+#
+# The work horse is the function `checkoutRepo()` in `tf.applib.repo`.
+#
+# Text-Fabric uses this function for all operations where data flows from GitHub to your computer.
+#
+# There are quite some options, and here we explain all the `checkout` options, i.e. the selection of
+# data from the history.
+#
+# See also the [documentation](https://annotation.github.io/text-fabric/tf/advanced/repo.html).
+
+# %load_ext autoreload
+# %autoreload 2
+
+# ## Leading example
+#
+# We use markdown display from IPython purely for presentation.
+# It is not needed to run `checkoutRepo()`.
+
+from tf.advanced.helpers import dm
+from tf.advanced.repo import checkoutRepo
+
+# We work with our tiny example TF app: `banks`.
+
+ORG = "annotation"
+REPO = "banks"
+MAIN = "tf"
+MOD = "sim/tf"
+
+
+# `MAIN`points to the main data, `MOD` points to a module of data: the similarity feature.
+
+# ## Presenting the results
+#
+# The function `do()` just formats the results of a `checkoutRepo()` run.
+#
+# The result of such a run, after the progress messages, is a tuple.
+# For the explanation of the tuple, read the [docs](https://annotation.github.io/text-fabric/tf/advanced/repo.html).
+
+
+def do(task):
+ md = f"""
+commit | release | local | base | subdir
+--- | --- | --- | --- | ---
+`{task[0]}` | `{task[1]}` | `{task[2]}` | `{task[3]}` | `{task[4]}`
+"""
+ dm(md)
+
+
+# + [markdown] toc-hr-collapsed=false
+# ## All the checkout options
+#
+# We discuss the meaning and effects of the values you can pass to the `checkout` option.
+# -
+
+# ### `clone`
+#
+# > Look whether the appropriate folder exists under your `~/github` directory.
+#
+# This is merely a check whether your data exists in the expected location.
+#
+# * No online checks take place.
+# * No data is moved or copied.
+#
+# **NB**: you cannot select releases and commits in your *local* GitHub clone.
+# The data will be used as it is found on your file system.
+#
+# **When to use**
+#
+# > If you are developing new feature data.
+#
+# When you develop your data in a repository, your development is private as long as you
+# do not push to GitHub.
+#
+# You can test your data, even without locally committing your data.
+#
+# But, if you are ready to share your data, everything is in place, and you only
+# have to commit and push, and pass the location on github to others, like
+#
+# ```
+# myorg/myrepo/subfolder
+# ```
+
+do(checkoutRepo(org=ORG, repo=REPO, folder=MAIN, version="0.2", checkout="clone"))
+
+# We show what happens if you do not have a local github clone in `~/github`.
+
+# + language="sh"
+#
+# mv ~/github/annotation/banks/tf ~/github/annotation/banks/tfxxx
+# -
+
+do(checkoutRepo(org=ORG, repo=REPO, folder=MAIN, version="0.2", checkout="clone"))
+
+# Note that no attempt is made to retrieve online data.
+
+# + language="sh"
+#
+# mv ~/github/annotation/banks/tfxxx ~/github/annotation/banks/tf
+# -
+
+# ### `local`
+#
+# > Look whether the appropriate folder exists under your `~/text-fabric-data` directory.
+#
+# This is merely a check whether your data exists in the expected location.
+#
+# * No online checks take place.
+# * No data is moved or copied.
+#
+# **When to use**
+#
+# > If you are using data created and shared by others, and if the data
+# is already on your system.
+#
+# You can be sure that no updates are downloaded, and that everything works the same as the last time
+# you ran your program.
+#
+# If you do not already have the data, you have to pass `latest` or `hot` or `''` which will be discussed below.
+
+do(checkoutRepo(org=ORG, repo=REPO, folder=MAIN, version="0.2", checkout="local"))
+
+# You see this data because earlier I have downloaded release `v2.0`, which is a tag for
+# the commit with hash `9713e71c18fd296cf1860d6411312f9127710ba7`.
+
+# If you do not have any corresponding data in your `~/text-fabric-data`, you get this:
+
+# + language="sh"
+#
+# mv ~/text-fabric-data/annotation/banks/tf ~/text-fabric-data/annotation/banks/tfxxx
+# -
+
+do(checkoutRepo(org=ORG, repo=REPO, folder=MAIN, version="0.2", checkout="local"))
+
+# + language="sh"
+#
+# mv ~/text-fabric-data/annotation/banks/tfxxx ~/text-fabric-data/annotation/banks/tf
+# -
+
+# ### `''` (default)
+#
+# This is about when you omit the `checkout` parameter, or pass `''` to it.
+#
+# The destination for local data is your `~/text-fabric-data` folder.
+#
+# If you have already a local copy of the data, that will be used.
+#
+# If not:
+#
+# > Note that if your local data is outdated, no new data will be downloaded.
+# You need `latest` or `hot` for that.
+#
+# But what is the latest online copy? In this case we mean:
+#
+# * the latest *release*, and from that release an appropriate attached zip file
+# * but if there is no such zip file, we take the files from the corresponding commit
+# * but if there is no release at all, we take the files from the *latest commit*.
+#
+# **When to use**
+#
+# > If you need data created/shared by other people and you want to be sure that you always have the
+# same copy that you initially downloaded.
+#
+# * If the data provider makes releases after important modifications, you will get those.
+# * If the data provider is experimenting after the latest release, and commits them to GitHub,
+# you do not get those.
+#
+# However, with `hot`, you `can` get the latest commit, to be discussed below.
+
+do(checkoutRepo(org=ORG, repo=REPO, folder=MAIN, version="0.2", checkout=""))
+
+# Note that no data has been downloaded, because it has detected that there is already local data on your computer.
+
+# If you do not have any checkout of this data on your computer, the data will be downloaded.
+
+# + language="sh"
+#
+# rm -rf ~/text-fabric-data/annotation/banks/tf
+# -
+
+do(checkoutRepo(org=ORG, repo=REPO, folder=MAIN, version="0.2", checkout=""))
+
+# #### Note about versions and releases
+#
+# The **version** of the data is not necessarily the same concept as the **release** of it.
+#
+# It is possible to keep the versions and the releases strictly parallel,
+# but in text conversion workflows it can be handy to make a distinction between them,
+# e.g. as follows:
+#
+# > the version is a property of the input data
+# > the release is a property of the output data
+#
+# When you create data from sources using conversion algorithms,
+# you want to increase the version if you get new input data, e.g. as a result of corrections
+# made by the author.
+#
+# But if you modify your conversion algorithm, while still running it on the same input data,
+# you may release the new output data as a **new release** of the **same version**.
+#
+# Likewise, when the input data stays the same, but you have corrected typos in the metadata,
+# you can make a **new release** of the **same version** of the data.
+#
+# The conversion delivers the features under a specific version,
+# and Text-Fabric supports those versions: users of TF can select the version they work with.
+#
+# Releases are made in the version control system (git and GitHub).
+# The part of Text-Fabric that auto-downloads data is aware of releases.
+# But once the data has been downloaded in place, there is no machinery in Text-Fabric to handle
+# different releases.
+#
+# Yet the release tag and commit hash are passed on to the point where it comes to recording
+# the provenance of the data.
+
+# #### Download a different version
+#
+# We download version `0.1` of the data.
+
+do(checkoutRepo(org=ORG, repo=REPO, folder=MAIN, version="0.1", checkout=""))
+
+# Several observations:
+#
+# * we obtained the older version from the *latest* release, which is still release `v2.0`
+# * the download looks different from when we downloaded version `0.2`;
+# this is because the data producer has zipped the `0.2` data and has attached it to release `v2.0`,
+# but he forgot, or deliberately refused, to attach version `0.1` to that release;
+# so it has been retrieved directly from the files in the corresponding commit, which is
+# `9713e71c18fd296cf1860d6411312f9127710ba7`.
+
+# For the verification, an online check is needed. The verification consists of checking the release tag and/or commit hash.
+#
+# If there is no online connection, you get this:
+
+# + language="sh"
+#
+# networksetup -setairportpower en0 off
+# -
+
+do(checkoutRepo(org=ORG, repo=REPO, folder=MAIN, version="0.1", checkout="latest"))
+
+# or if you do not have local data:
+
+# + language="sh"
+#
+# mv ~/text-fabric-data/annotation/banks/tf/0.1 ~/text-fabric-data/annotation/banks/tf/0.1xxx
+# -
+
+do(checkoutRepo(org=ORG, repo=REPO, folder=MAIN, version="0.1", checkout="latest"))
+
+# + language="sh"
+#
+# mv ~/text-fabric-data/annotation/banks/tf/0.1xxx ~/text-fabric-data/annotation/banks/tf/0.1
+
+# + language="sh"
+#
+# networksetup -setairportpower en0 on
+# -
+
+# ### `latest`
+#
+# > The latest online release will be identified,
+# and if you do not have that copy locally, it will be downloaded.
+#
+# **When to use**
+#
+# > If you need data created/shared by other people and you want to be sure that you always have the
+# latest *stable* version of that data, unreleased data is not good enough.
+#
+# One of the difference with `checkout=''` is that if there are no releases, you will not get data.
+
+do(checkoutRepo(org=ORG, repo=REPO, folder=MAIN, version="0.2", checkout="latest"))
+
+# There is no sim/tf data in any release commit, so if we look it up, it should fail.
+
+do(checkoutRepo(org=ORG, repo=REPO, folder=MOD, version="0.2", checkout="latest"))
+
+# But with `checkout=''` it will only be found if you do not have local data already:
+
+do(checkoutRepo(org=ORG, repo=REPO, folder=MOD, version="0.2", checkout=""))
+
+# In that case there is only one way: `hot`:
+
+do(checkoutRepo(org=ORG, repo=REPO, folder=MOD, version="0.2", checkout="hot"))
+
+# ### `hot`
+#
+# > The latest online commit will be identified,
+# and if you do not have that copy locally, it will be downloaded.
+#
+# **When to use**
+#
+# > If you need data created/shared by other people and you want to be sure that you always have the
+# latest version of that data, whether released or not.
+#
+# The difference with `checkout=''` is that if there are releases,
+# you will now get data that may be newer than the latest release.
+
+do(checkoutRepo(org=ORG, repo=REPO, folder=MAIN, version="0.2", checkout="hot"))
+
+# Observe that data has been downloaded, and that we have now data corresponding to a different commit hash,
+# and not corresponding to a release.
+#
+# If we now ask for the latest *stable* data, the data will be downloaded anew.
+
+do(checkoutRepo(org=ORG, repo=REPO, folder=MAIN, version="0.2", checkout="latest"))
+
+# ### `v1.0` a specific release
+#
+# > Look for a specific online release to get data from.
+#
+# **When to use**
+#
+# > When you want to replicate something, and need data from an earlier point in the history.
+
+do(checkoutRepo(org=ORG, repo=REPO, folder=MAIN, version="0.1", checkout="v1.0"))
+
+# We might try to get version `0.2` from this release.
+
+do(checkoutRepo(org=ORG, repo=REPO, folder=MAIN, version="0.2", checkout="v1.0"))
+
+# At that early point in the history there is not yet a version `0.2` of the data.
+
+# ### `a81746c` a specific commit
+#
+# > Look for a specific online commit to get data from.
+#
+# **When to use**
+#
+# > When you want to replicate something, and need data from an earlier point in the history, and there is no
+# release for that commit.
+
+do(
+ checkoutRepo(
+ org=ORG,
+ repo=REPO,
+ folder=MAIN,
+ version="0.1",
+ checkout="a81746c5f9627637db4dae04c2d5348bda9e511a",
+ )
+)
+
+# + [markdown] toc-hr-collapsed=false
+# ## *source* and *dest*: an alternative for `~/github` and `~/text-fabric-data`
+#
+# Everything so far uses the hard-wired `~/github` and `~/text-fabric-data` directories.
+# But you can change that:
+#
+# * pass *source* as a replacement for `~/github`.
+# * pass *dest* as a replacement for `~/text-fabric-data`.
+#
+# **When to use**
+#
+# > if you do not want to interfere with the `~/text-fabric-data` directory.
+#
+# Text-Fabric manages the `~/text-fabric-data` directory,
+# and if you are experimenting outside Text-Fabric
+# you may not want to touch its data directory.
+#
+# > if you want to clone data into your `~/github` directory.
+#
+# Normally, TF uses your `~/github` directory as a source of information,
+# and never writes into it.
+# But if you explicitly pass `dest=~/github`, things change: downloads will
+# arrive under `~/github`. Use this with care.
+#
+# > if you work with cloned data outside your `~/github` directory,
+#
+# you can let the system look in *source* instead of `~/github`.
+# -
+
+# We customize source and destination directories:
+#
+# * we put them both under `~/Downloads`
+# * we give them different names
+
+MY_GH = "~/Downloads/repoclones"
+MY_TFD = "~/Downloads/textbase"
+
+# Download a fresh copy of the data to `~/Downloads/textbase` instead.
+
+do(
+ checkoutRepo(
+ org=ORG,
+ repo=REPO,
+ folder=MAIN,
+ version="0.2",
+ checkout="",
+ source=MY_GH,
+ dest=MY_TFD,
+ )
+)
+
+# Lookup the same data locally.
+
+do(
+ checkoutRepo(
+ org=ORG,
+ repo=REPO,
+ folder=MAIN,
+ version="0.2",
+ checkout="",
+ source=MY_GH,
+ dest=MY_TFD,
+ )
+)
+
+# We copy the local github data to the custom location:
+
+# + language="sh"
+#
+# mkdir -p ~/Downloads/repoclones/annotation
+# cp -R ~/github/annotation/banks ~/Downloads/repoclones/annotation/banks
+# -
+
+# Lookup the data in this alternative directory.
+
+do(
+ checkoutRepo(
+ org=ORG,
+ repo=REPO,
+ folder=MAIN,
+ version="0.2",
+ checkout="clone",
+ source=MY_GH,
+ dest=MY_TFD,
+ )
+)
+
+# Note that the directory trees under the customised *source* and *dest* locations have exactly the same shape as before.
+
+# ## Conclusion
+#
+# With the help of `checkoutRepo()` you will be able to make local copies of online data in an organized way.
+#
+# This will help you when
+#
+# * you use other people's data
+# * develop your own data
+# * share and publish your data
+# * go back in history.
+
+# ---
+# All chapters:
+#
+# * [use](use.ipynb)
+# * [share](share.ipynb)
+# * [app](app.ipynb)
+# * *repo*
+# * [compose](compose.ipynb)
+#
+# ---
diff --git a/tutorial/share.ipynb b/tutorial/share.ipynb
new file mode 100644
index 0000000..9f97d86
--- /dev/null
+++ b/tutorial/share.ipynb
@@ -0,0 +1,100 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "---\n",
+ "Start with [convert](https://nbviewer.jupyter.org/github/annotation/banks/blob/master/programs/convert.ipynb)\n",
+ "\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Share the Banks example corpus"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "True\n",
+ "Create release data for annotation/banks/tf\n",
+ "Found 2 versions\n",
+ "zip files end up in /Users/dirk/Downloads/annotation-release/banks\n",
+ "zipping annotation/banks 0.1 with 10 features ==> tf-0.1.zip\n",
+ "zipping annotation/banks 0.2 with 10 features ==> tf-0.2.zip\n"
+ ]
+ }
+ ],
+ "source": [
+ "%%sh\n",
+ "\n",
+ "text-fabric-zip annotation/banks/tf"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Share the similarity feature as a module\n",
+ "\n",
+ "We do not attach the similarity data to a release, since we intend to update this data quite a few times\n",
+ "after tweaking the parameters of the algorithm by which we create it.\n",
+ "\n",
+ "That means that users will get this data from the latest commit.\n",
+ "\n",
+ "So we make sure that the similarity data is committed and pushed to GitHub."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "All chapters:\n",
+ "\n",
+ "* [use](use.ipynb)\n",
+ "* *share*\n",
+ "* [app](app.ipynb)\n",
+ "* [repo](repo.ipynb)\n",
+ "* [compose](compose.ipynb)\n",
+ "\n",
+ "---"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.7"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/tutorial/use.ipynb b/tutorial/use.ipynb
new file mode 100644
index 0000000..a2f9cff
--- /dev/null
+++ b/tutorial/use.ipynb
@@ -0,0 +1,790 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "---\n",
+ "Start with [convert](https://nbviewer.jupyter.org/github/annotation/banks/blob/master/programs/convert.ipynb)\n",
+ "\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Use the Banks example corpus"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Load TF\n",
+ "\n",
+ "We are going to load the new data: all features.\n",
+ "\n",
+ "We start a new instance of the TF machinery."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%load_ext autoreload\n",
+ "%autoreload 2"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "\n",
+ "from tf.fabric import Fabric"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "This is Text-Fabric 8.3.0\n",
+ "Api reference : https://annotation.github.io/text-fabric/cheatsheet.html\n",
+ "\n",
+ "10 features found and 0 ignored\n"
+ ]
+ }
+ ],
+ "source": [
+ "GH_BASE = os.path.expanduser(\"~/github\")\n",
+ "ORG = \"annotation\"\n",
+ "REPO = \"banks\"\n",
+ "FOLDER = \"tf\"\n",
+ "TF_DIR = f\"{GH_BASE}/{ORG}/{REPO}/{FOLDER}\"\n",
+ "\n",
+ "VERSION = \"0.2\"\n",
+ "\n",
+ "TF_PATH = f\"{TF_DIR}/{VERSION}\"\n",
+ "TF = Fabric(locations=TF_PATH)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We ask for a list of all features:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "('author',\n",
+ " 'gap',\n",
+ " 'letters',\n",
+ " 'number',\n",
+ " 'otype',\n",
+ " 'punc',\n",
+ " 'terminator',\n",
+ " 'title',\n",
+ " 'oslots')"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "allFeatures = TF.explore(silent=True, show=True)\n",
+ "loadableFeatures = allFeatures[\"nodes\"] + allFeatures[\"edges\"]\n",
+ "loadableFeatures"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We load all features:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ " 0.00s loading features ...\n",
+ " 0.02s All features loaded/computed - for details use loadLog()\n"
+ ]
+ }
+ ],
+ "source": [
+ "api = TF.load(loadableFeatures, silent=False)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "You see that all files are marked with a `T`.\n",
+ "\n",
+ "That means that Text-Fabric loads the features by reading the plain text `.tf` files.\n",
+ "But after reading, it makes a binary equivalent and stores it as a `.tfx`\n",
+ "file in the hidden `.tf` directory next to it.\n",
+ "\n",
+ "Furthermore, you see some lines marked with `C`. Here Text-Fabric is computing derived data,\n",
+ "mostly about sections, the order of nodes, and the relative positions of nodes with respect to the slots they\n",
+ "are linked to.\n",
+ "\n",
+ "The results of this pre-computation are also stored in that hidden `.tf` directory.\n",
+ "\n",
+ "The next time, Text-Fabric loads the data from their binary `.tfx` files, which is much faster.\n",
+ "And the pre-computation step will be skipped.\n",
+ "\n",
+ "If the binary files get outdated Text-Fabric will recompile and recompute everything automatically.\n",
+ "\n",
+ "So let's load again."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "This is Text-Fabric 8.3.0\n",
+ "Api reference : https://annotation.github.io/text-fabric/cheatsheet.html\n",
+ "\n",
+ "10 features found and 0 ignored\n",
+ " 0.00s loading features ...\n",
+ " 0.01s All features loaded/computed - for details use loadLog()\n"
+ ]
+ }
+ ],
+ "source": [
+ "TF = Fabric(locations=TF_PATH)\n",
+ "api = TF.load(loadableFeatures, silent=False)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Where there were `T`s before, there are now `B`s."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Hoisting\n",
+ "\n",
+ "We can access all TF data programmatically by using `A.api.Features`, or `A.api.F` (same thing) and a bunch of\n",
+ "other API members.\n",
+ "\n",
+ "But if we working with a single data source, we hoist those API members to the global namespace.\n",
+ "\n",
+ "Now you can directly refer to `F` and friends."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[('Computed',\n",
+ " 'computed-data',\n",
+ " ('C Computed', 'Call AllComputeds', 'Cs ComputedString')),\n",
+ " ('Features', 'edge-features', ('E Edge', 'Eall AllEdges', 'Es EdgeString')),\n",
+ " ('Fabric', 'loading', ('TF',)),\n",
+ " ('Locality', 'locality', ('L Locality',)),\n",
+ " ('Nodes', 'navigating-nodes', ('N Nodes',)),\n",
+ " ('Features',\n",
+ " 'node-features',\n",
+ " ('F Feature', 'Fall AllFeatures', 'Fs FeatureString')),\n",
+ " ('Search', 'search', ('S Search',)),\n",
+ " ('Text', 'text', ('T Text',))]"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "api.makeAvailableIn(globals())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "As a result, you have an overview of the names you can use."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Exploration\n",
+ "\n",
+ "Finally, let's explore this set by means of Text-Fabric.\n",
+ "\n",
+ "### Frequency list\n",
+ "\n",
+ "We can get ordered frequency lists for the values of all features.\n",
+ "\n",
+ "First the words:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "(('the', 8),\n",
+ " ('of', 5),\n",
+ " ('and', 4),\n",
+ " ('in', 3),\n",
+ " ('we', 3),\n",
+ " ('everything', 2),\n",
+ " ('know', 2),\n",
+ " ('most', 2),\n",
+ " ('ones', 2),\n",
+ " ('patterns', 2),\n",
+ " ('us', 2),\n",
+ " ('Besides', 1),\n",
+ " ('Culture', 1),\n",
+ " ('Everything', 1),\n",
+ " ('So', 1),\n",
+ " ('a', 1),\n",
+ " ('about', 1),\n",
+ " ('aid', 1),\n",
+ " ('any', 1),\n",
+ " ('around', 1),\n",
+ " ('as', 1),\n",
+ " ('barbarian', 1),\n",
+ " ('bottom', 1),\n",
+ " ('can', 1),\n",
+ " ('care', 1),\n",
+ " ('climbing', 1),\n",
+ " ('composed', 1),\n",
+ " ('control', 1),\n",
+ " ('dead', 1),\n",
+ " ('elegant', 1),\n",
+ " ('enjoyable', 1),\n",
+ " ('final', 1),\n",
+ " ('find', 1),\n",
+ " ('free', 1),\n",
+ " ('games', 1),\n",
+ " ('good', 1),\n",
+ " ('harness', 1),\n",
+ " ('have', 1),\n",
+ " ('high', 1),\n",
+ " ('humans', 1),\n",
+ " ('impossible', 1),\n",
+ " ('is', 1),\n",
+ " ('it', 1),\n",
+ " ('languages', 1),\n",
+ " ('left', 1),\n",
+ " ('life', 1),\n",
+ " ('line', 1),\n",
+ " ('make', 1),\n",
+ " ('mattered', 1),\n",
+ " ('mountains', 1),\n",
+ " ('not', 1),\n",
+ " ('nothing', 1),\n",
+ " ('our', 1),\n",
+ " ('over', 1),\n",
+ " ('own', 1),\n",
+ " ('problems', 1),\n",
+ " ('really', 1),\n",
+ " ('romance', 1),\n",
+ " ('safety', 1),\n",
+ " ('societies', 1),\n",
+ " ('sports', 1),\n",
+ " ('studying', 1),\n",
+ " ('such', 1),\n",
+ " ('take', 1),\n",
+ " ('terms', 1),\n",
+ " ('that', 1),\n",
+ " ('that’s', 1),\n",
+ " ('things', 1),\n",
+ " ('those', 1),\n",
+ " ('to', 1),\n",
+ " ('truth', 1),\n",
+ " ('ultimately', 1),\n",
+ " ('where', 1),\n",
+ " ('why', 1),\n",
+ " ('without', 1))"
+ ]
+ },
+ "execution_count": 8,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "F.letters.freqList()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "For the node types we can get info by calling this:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {
+ "lines_to_next_cell": 2
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "(('book', 99.0, 100, 100),\n",
+ " ('chapter', 49.5, 101, 102),\n",
+ " ('sentence', 33.0, 115, 117),\n",
+ " ('line', 7.666666666666667, 103, 114),\n",
+ " ('word', 1, 1, 99))"
+ ]
+ },
+ "execution_count": 9,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "C.levels.data"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "It means that chapters are 49.5 words long on average, and that the chapter nodes are 101 and 102.\n",
+ "\n",
+ "And you see that we have 99 words."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Add to the banks corpus\n",
+ "\n",
+ "We are going to make a relationship between each pair of words, and we annotate each related pair with how similar they are.\n",
+ "\n",
+ "We measure the similarity by looking at the distinct letters in each word (lowercase), and computing the percentage of\n",
+ "how many letters they have in common with respect to how many letters they jointly have.\n",
+ "\n",
+ "This will become a symmetric edge feature. Symmetric means, that if a and b are similar, then b and a as well, with the\n",
+ "same similarity.\n",
+ "\n",
+ "We only store one copy of each symmetric pair of edges.\n",
+ "\n",
+ "We can then use\n",
+ "[`E.sim.b(node)`](https://annotation.github.io/text-fabric/tf/core/edgefeature.html)\n",
+ "to find all nodes that are parallel to node.\n",
+ "\n",
+ "If words do not have letters in common, their similarity is 0, and we do not make an edge."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "lines_to_next_cell": 2
+ },
+ "source": [
+ "# Preparation\n",
+ "\n",
+ "We pre-compute all letter sets for all words."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def makeSet(w):\n",
+ " return set(F.letters.v(w).lower())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {
+ "lines_to_end_of_cell_marker": 2,
+ "lines_to_next_cell": 2
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "99 words\n"
+ ]
+ }
+ ],
+ "source": [
+ "words = {}\n",
+ "\n",
+ "for w in F.otype.s(\"word\"):\n",
+ " words[w] = makeSet(w)\n",
+ "\n",
+ "nWords = len(words)\n",
+ "print(f\"{nWords} words\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {
+ "lines_to_next_cell": 2
+ },
+ "outputs": [],
+ "source": [
+ "def sim(wSet, vSet):\n",
+ " return int(round(100 * len(wSet & vSet) / len(wSet | vSet)))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "lines_to_next_cell": 2
+ },
+ "source": [
+ "# Compute all similarities\n",
+ "\n",
+ "We are going to perform all comparisons.\n",
+ "\n",
+ "Since there are 99 words, this will amount to only 5000 comparisons.\n",
+ "\n",
+ "For a big corpus, this amount will quickly grow with the number of items to be compared.\n",
+ "\n",
+ "See for example the similarities in the\n",
+ "[Quran](https://nbviewer.jupyter.org/github/q-ran/quran/blob/master/programs/parallels.ipynb)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def computeSim():\n",
+ " similarity = {}\n",
+ "\n",
+ " wordNodes = sorted(words.keys())\n",
+ " nWords = len(wordNodes)\n",
+ "\n",
+ " nComparisons = nWords * (nWords - 1) // 2\n",
+ "\n",
+ " print(f\"{nComparisons} comparisons to make\")\n",
+ "\n",
+ " TF.indent(reset=True)\n",
+ "\n",
+ " co = 0\n",
+ " si = 0\n",
+ " stop = False\n",
+ " for i in range(nWords):\n",
+ " nodeI = wordNodes[i]\n",
+ " wordI = words[nodeI]\n",
+ " for j in range(i + 1, nWords):\n",
+ " nodeJ = wordNodes[j]\n",
+ " wordJ = words[nodeJ]\n",
+ " s = sim(wordI, wordJ)\n",
+ " co += 1\n",
+ " if s:\n",
+ " similarity[(nodeI, nodeJ)] = sim(wordI, wordJ)\n",
+ " si += 1\n",
+ " if stop:\n",
+ " break\n",
+ "\n",
+ " TF.info(f\"{co:>4} comparisons and {si:>4} similarities\")\n",
+ " return similarity"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "4851 comparisons to make\n",
+ " 0.01s 4851 comparisons and 3332 similarities\n"
+ ]
+ }
+ ],
+ "source": [
+ "similarity = computeSim()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "7\n",
+ "100\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(min(similarity.values()))\n",
+ "print(max(similarity.values()))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "eq = [x for x in similarity.items() if x[1] >= 100]\n",
+ "neq = [x for x in similarity.items() if x[1] <= 50]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "((1, 4), 100)\n",
+ "((1, 2), 8)\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(eq[0])\n",
+ "print(neq[0])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "58\n",
+ "3247\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(len(eq))\n",
+ "print(len(neq))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "1 Everything\n",
+ "4 everything\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(eq[0][0][0], F.letters.v(eq[0][0][0]))\n",
+ "print(eq[0][0][1], F.letters.v(eq[0][0][1]))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 20,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "1 Everything\n",
+ "2 about\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(neq[0][0][0], F.letters.v(neq[0][0][0]))\n",
+ "print(neq[0][0][1], F.letters.v(neq[0][0][1]))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Add parallels to the TF dataset\n",
+ "\n",
+ "We now add this information to the Banks dataset as an *edge feature*."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 21,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "metaData = {\n",
+ " \"\": {\n",
+ " \"name\": \"Banks (similar words)\",\n",
+ " \"converters\": \"Dirk Roorda\",\n",
+ " \"sourceUrl\": \"https://nbviewer.jupyter.org/github/annotation/tutorials/blob/master/text-fabric/use.ipynb\",\n",
+ " \"version\": \"0.2\",\n",
+ " },\n",
+ " \"sim\": {\n",
+ " \"valueType\": \"int\",\n",
+ " \"edgeValues\": True,\n",
+ " \"description\": \"similarity between words, as a percentage of the common material wrt the combined material\",\n",
+ " },\n",
+ "}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 22,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "simData = {}\n",
+ "for ((f, t), d) in similarity.items():\n",
+ " simData.setdefault(f, {})[t] = d"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 23,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "FOLDER_SIM = \"sim/tf\"\n",
+ "path = f\"{ORG}/{REPO}/{FOLDER_SIM}\"\n",
+ "location = f\"{GH_BASE}/{path}\"\n",
+ "module = VERSION"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 24,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ " 0.00s Exporting 0 node and 1 edge and 0 config features to ~/github/annotation/banks/sim/tf/0.2:\n",
+ " | 0.01s T sim to ~/github/annotation/banks/sim/tf/0.2\n",
+ " 0.01s Exported 0 node features and 1 edge features and 0 config features to ~/github/annotation/banks/sim/tf/0.2\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "True"
+ ]
+ },
+ "execution_count": 24,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "TF.save(\n",
+ " edgeFeatures=dict(sim=simData), metaData=metaData, location=location, module=module\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "All chapters:\n",
+ "\n",
+ "* *use*\n",
+ "* [share](share.ipynb)\n",
+ "* [app](app.ipynb)\n",
+ "* [repo](repo.ipynb)\n",
+ "* [compose](compose.ipynb)\n",
+ "\n",
+ "---\n",
+ "\n",
+ "CC-BY Dirk Roorda"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.7"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/tutorial/use.py b/tutorial/use.py
new file mode 100644
index 0000000..eabd6ad
--- /dev/null
+++ b/tutorial/use.py
@@ -0,0 +1,272 @@
+# ---
+# jupyter:
+# jupytext:
+# text_representation:
+# extension: .py
+# format_name: light
+# format_version: '1.5'
+# jupytext_version: 1.11.4
+# kernelspec:
+# display_name: Python 3
+# language: python
+# name: python3
+# ---
+
+#
+#
+#
+#
+# ---
+# Start with [convert](https://nbviewer.jupyter.org/github/annotation/banks/blob/master/programs/convert.ipynb)
+#
+# ---
+
+# # Use the Banks example corpus
+
+# ## Load TF
+#
+# We are going to load the new data: all features.
+#
+# We start a new instance of the TF machinery.
+
+# %load_ext autoreload
+# %autoreload 2
+
+# +
+import os
+
+from tf.fabric import Fabric
+
+# +
+GH_BASE = os.path.expanduser("~/github")
+ORG = "annotation"
+REPO = "banks"
+FOLDER = "tf"
+TF_DIR = f"{GH_BASE}/{ORG}/{REPO}/{FOLDER}"
+
+VERSION = "0.2"
+
+TF_PATH = f"{TF_DIR}/{VERSION}"
+TF = Fabric(locations=TF_PATH)
+# -
+
+# We ask for a list of all features:
+
+allFeatures = TF.explore(silent=True, show=True)
+loadableFeatures = allFeatures["nodes"] + allFeatures["edges"]
+loadableFeatures
+
+# We load all features:
+
+api = TF.load(loadableFeatures, silent=False)
+
+# You see that all files are marked with a `T`.
+#
+# That means that Text-Fabric loads the features by reading the plain text `.tf` files.
+# But after reading, it makes a binary equivalent and stores it as a `.tfx`
+# file in the hidden `.tf` directory next to it.
+#
+# Furthermore, you see some lines marked with `C`. Here Text-Fabric is computing derived data,
+# mostly about sections, the order of nodes, and the relative positions of nodes with respect to the slots they
+# are linked to.
+#
+# The results of this pre-computation are also stored in that hidden `.tf` directory.
+#
+# The next time, Text-Fabric loads the data from their binary `.tfx` files, which is much faster.
+# And the pre-computation step will be skipped.
+#
+# If the binary files get outdated Text-Fabric will recompile and recompute everything automatically.
+#
+# So let's load again.
+
+TF = Fabric(locations=TF_PATH)
+api = TF.load(loadableFeatures, silent=False)
+
+# Where there were `T`s before, there are now `B`s.
+
+# ### Hoisting
+#
+# We can access all TF data programmatically by using `A.api.Features`, or `A.api.F` (same thing) and a bunch of
+# other API members.
+#
+# But if we working with a single data source, we hoist those API members to the global namespace.
+#
+# Now you can directly refer to `F` and friends.
+
+api.makeAvailableIn(globals())
+
+# As a result, you have an overview of the names you can use.
+
+# ## Exploration
+#
+# Finally, let's explore this set by means of Text-Fabric.
+#
+# ### Frequency list
+#
+# We can get ordered frequency lists for the values of all features.
+#
+# First the words:
+
+F.letters.freqList()
+
+# For the node types we can get info by calling this:
+
+C.levels.data
+
+
+# It means that chapters are 49.5 words long on average, and that the chapter nodes are 101 and 102.
+#
+# And you see that we have 99 words.
+
+# # Add to the banks corpus
+#
+# We are going to make a relationship between each pair of words, and we annotate each related pair with how similar they are.
+#
+# We measure the similarity by looking at the distinct letters in each word (lowercase), and computing the percentage of
+# how many letters they have in common with respect to how many letters they jointly have.
+#
+# This will become a symmetric edge feature. Symmetric means, that if a and b are similar, then b and a as well, with the
+# same similarity.
+#
+# We only store one copy of each symmetric pair of edges.
+#
+# We can then use
+# [`E.sim.b(node)`](https://annotation.github.io/text-fabric/tf/core/edgefeature.html)
+# to find all nodes that are parallel to node.
+#
+# If words do not have letters in common, their similarity is 0, and we do not make an edge.
+
+# # Preparation
+#
+# We pre-compute all letter sets for all words.
+
+
+def makeSet(w):
+ return set(F.letters.v(w).lower())
+
+
+# +
+words = {}
+
+for w in F.otype.s("word"):
+ words[w] = makeSet(w)
+
+nWords = len(words)
+print(f"{nWords} words")
+
+
+# -
+
+
+def sim(wSet, vSet):
+ return int(round(100 * len(wSet & vSet) / len(wSet | vSet)))
+
+
+# # Compute all similarities
+#
+# We are going to perform all comparisons.
+#
+# Since there are 99 words, this will amount to only 5000 comparisons.
+#
+# For a big corpus, this amount will quickly grow with the number of items to be compared.
+#
+# See for example the similarities in the
+# [Quran](https://nbviewer.jupyter.org/github/q-ran/quran/blob/master/programs/parallels.ipynb).
+
+
+def computeSim():
+ similarity = {}
+
+ wordNodes = sorted(words.keys())
+ nWords = len(wordNodes)
+
+ nComparisons = nWords * (nWords - 1) // 2
+
+ print(f"{nComparisons} comparisons to make")
+
+ TF.indent(reset=True)
+
+ co = 0
+ si = 0
+ stop = False
+ for i in range(nWords):
+ nodeI = wordNodes[i]
+ wordI = words[nodeI]
+ for j in range(i + 1, nWords):
+ nodeJ = wordNodes[j]
+ wordJ = words[nodeJ]
+ s = sim(wordI, wordJ)
+ co += 1
+ if s:
+ similarity[(nodeI, nodeJ)] = sim(wordI, wordJ)
+ si += 1
+ if stop:
+ break
+
+ TF.info(f"{co:>4} comparisons and {si:>4} similarities")
+ return similarity
+
+
+similarity = computeSim()
+
+print(min(similarity.values()))
+print(max(similarity.values()))
+
+eq = [x for x in similarity.items() if x[1] >= 100]
+neq = [x for x in similarity.items() if x[1] <= 50]
+
+print(eq[0])
+print(neq[0])
+
+print(len(eq))
+print(len(neq))
+
+print(eq[0][0][0], F.letters.v(eq[0][0][0]))
+print(eq[0][0][1], F.letters.v(eq[0][0][1]))
+
+print(neq[0][0][0], F.letters.v(neq[0][0][0]))
+print(neq[0][0][1], F.letters.v(neq[0][0][1]))
+
+# # Add parallels to the TF dataset
+#
+# We now add this information to the Banks dataset as an *edge feature*.
+
+metaData = {
+ "": {
+ "name": "Banks (similar words)",
+ "converters": "Dirk Roorda",
+ "sourceUrl": "https://nbviewer.jupyter.org/github/annotation/tutorials/blob/master/text-fabric/use.ipynb",
+ "version": "0.2",
+ },
+ "sim": {
+ "valueType": "int",
+ "edgeValues": True,
+ "description": "similarity between words, as a percentage of the common material wrt the combined material",
+ },
+}
+
+simData = {}
+for ((f, t), d) in similarity.items():
+ simData.setdefault(f, {})[t] = d
+
+FOLDER_SIM = "sim/tf"
+path = f"{ORG}/{REPO}/{FOLDER_SIM}"
+location = f"{GH_BASE}/{path}"
+module = VERSION
+
+TF.save(
+ edgeFeatures=dict(sim=simData), metaData=metaData, location=location, module=module
+)
+
+# ---
+# All chapters:
+#
+# * *use*
+# * [share](share.ipynb)
+# * [app](app.ipynb)
+# * [repo](repo.ipynb)
+# * [compose](compose.ipynb)
+#
+# ---
+#
+# CC-BY Dirk Roorda