documentation update

The-Book-of-CLAMS/About-My-Music.qmd

@@ -29,33 +29,23 @@ Wilson [@narushima2019microtonality] and Nick Collins [@collins2008errant;
 - Spectral music. This is another advanced synthesis methodology; a recent
 reference is [@lazzarini2021spectral]
 
-## Examples
 My current home for published music is on Bandcamp at
-<https://algocompsynth.bandcamp.com/>. The test case for CLAMS is a work in
-progress called "A Musical Clambake", submitted as a proposal to the
-[Hybrid Live Coding Interfaces Workshop
-2023](https://hybrid-livecode.pubpub.org/workshop2023):
-
-> "A Musical Clambake": A five minute algorithmic microtonal video produced
-on a Pimoroni PicoVision
-(https://shop.pimoroni.com/products/picovision?variant=41048911904851) using
-the CLAMS (https://algocompsynth.github.io/CLAMS) live coding system. "A
-Musical Clambake" revisits the birth of live coding, made possible by
-inexpensive personal computers and the Forth programming language, and
-explores what happens when the computer is a musical collaborator via dynamic
-stochastic synthesis.
-
-I am also planning to submit "A Musical Clambake" to the [Pimoroni PicoVision
-Demoscene
-Competition](https://shop.pimoroni.com/pages/win-a-500-gift-card-in-our-picovision-competition).
+<https://algocompsynth.bandcamp.com/>. I may move to Gumroad, which allows
+publishing a broader range of digital media artefacts than music and music
+videos.
 
 ## Other AlgoCompSynth projects
 
 1. [AlgoCompSynth-One](https://github.com/AlgoCompSynth/AlgoCompSynth-One):
 This is a platform for doing high-performance digital signal processing and
 musical AI on NVIDIA GPUs. I currently support Windows 11 WSL Ubuntu 22.04 LTS
-and NVIDIA Jetson JetPack 5. If I can make everything work I will be
-supporting native Windows 11.
+and NVIDIA Jetson JetPack 5.
+
+    If I can make everything work I will be supporting native Windows 11.
+    `AlgoCompSynth-One` is based on
+    [Mamba](https://mamba.readthedocs.io/en/latest/installation/mamba-installation.html),
+    and Mamba runs on Windows, so unless other dependencies can't be found, it
+    should be possible.
 
 2. [eikosany](https://algocompsynth.github.io/eikosany/): This is an R
 package for algorithmic composition with musical scales derived by Erv Wilson
@@ -66,3 +56,7 @@ package to perform the computations described in _Tuning, Timbre, Spectrum,
 Scale_ [@sethares2013tuning]. This is a superset of any such algorithms that
 will be deployed in CLAMS; only calculations that need to be performed
 during a performance need to be deployed in CLAMS.
+
+There is a fairly large overlap in functionality between `eikosany` and
+`consonaR`. There's a plan for refactoring the two, but that work is on hold
+until I get more of `CLAMS` done.

The-Book-of-CLAMS/CLAMS-Forth-Overview.qmd

@@ -13,48 +13,91 @@ are Stephen Smith's _RP2040 Assembly Language Programming_ [@smith2021rp2040],
 and, of course, _Raspberry Pi Pico C/C++ SDK_ [@raspberrypi2023a].
 
 2. Optimized for speed: CLAMS-Forth will be written in RP2040 assembly
-and will provide an RP2040 assembler. CLAMS-Forth will use subroutine
-threading and will allow both calling and inlining `CODE` words
-written in assembly.
+and will provide an RP2040 assembler.
 
-3. Cooperative multitasking: The RP2040 has two cores, and each core has a
-divide coprocessor and two interpolators. In addition, the RP2040 has
-two programmable input / output (PIO) blocks. Cooperative multitasking
-is the Forth way to exploit this available concurrency and is
-well-supported by the SDK.
-
-4. An enhanced Forth virtual machine, providing registers for indexing and
-intermendiate results as defined in Stephen Pelc's "Updating the Forth
-virtual machine" [@pelc2008updating]
-
-5. A high-speed block floating point digital signal processing library.
+3. Digital signal processing: A high-speed block floating point digital signal processing library will be provided.
 
 ## Desiderata
 
 1. Forth 2012 standard compatibility is a strong desire but not an absolute
-requirement. Most of the CORE word set and some of the CORE EXT word set
-will be implemented, but a few tricky or risky CORE words will be omitted. The
-Search-Order and Programming-Tools word sets will be implemented.
+requirement. Much of the CORE word set and some of the CORE EXT word set
+will be implemented, but a number of tricky, risky or little-used CORE words
+will be omitted. Many of the two-cell operations, for example, aren't really
+needed in a Forth running on a 32-bit architecture like the RP2040.
+
+    Parts of the Search-Order and Programming-Tools word sets will be
+    implemented. I want to have the Block word set using blocks in flash memory,
+    but that's not going to be in the first release.
 
-Custom word sets will be provided for cooperative multitasking, high-speed
-digital signal processing, and SDK / hardware access. ***All access to
-the hardware / system level operations will be performed via C language calls to
-the SDK.***
+    Custom word sets will be provided for high-speed digital signal processing
+    and SDK / hardware access. ***All access to the hardware / system level
+    operations will be performed via C language calls to the SDK.***
 
-2. Portability to other boards with the RP2040 microcontroller is possible,
+2. Cooperative multitasking: The RP2040 has two cores, and each core has a
+divide coprocessor and two interpolators. In addition, the RP2040 has two
+programmable input / output (PIO) blocks. Cooperative multitasking is the Forth
+way to exploit this available concurrency and is well-supported by the SDK.
+
+3. Portability to other boards with the RP2040 microcontroller is possible,
 but is not on the roadmap yet. As with CLAMS itself, the initial target is the
 Pimoroni PicoVision.
 
-3. Floating point support is desirable, but is a fairly low priority.
-Floating point arithmetic is convenient, and the RP2040 SDK provides
-optimized floating point libraries, but there's no hardware floating
-point arithmetic on the RP2040. So it's not obvious how useful
-this capability would be.
+4. Floating point support is desirable, but is a fairly low priority. Floating
+point arithmetic is convenient, and the RP2040 SDK provides optimized floating
+point libraries, but there's no hardware floating point arithmetic on the
+RP2040. It's too slow for real-time digital signal processing, so it's not
+obvious how useful this capability would be.
 
 ## Design / architecture
 
-The top-level design / architecture are based on Dr. Chen-Hanson Ting's
+The top-level design and architecture are based on Dr. Chen-Hanson Ting's
 _eForth Overview_ [@pintaske2018eforth]. In eForth, a small number of primitive
 words are implemented with a macro assembler, then the rest of the system is
-built in Forth on top of those words. The lower-level design will follow
-_Irreducible Complexity: EForth for Discovery_ [@pintaske2019irreducible].
+built in Forth on top of those words. Brad Rodriguez' CamelForth
+[@pintaske2018moving] also provides some implementation guidance.
+
+The original plan for CLAMS-Forth was to use subroutine threading and inline
+code. There is a partial implementation, but I decided to switch to a more
+"traditional" but less efficient direct threaded implementation. There are two
+main reasons:
+
+1. The branch instructions in the RP2040 are program counter relative, and the
+available target distances are limited by the instruction formats. In particular,
+the function call (`BL`) instruction has an available range of plus or minus
+16 megabytes. While this is enough within the RP2040's flash or SRAM, it is
+*not* enough to cross the address space distance between SRAM and flash.
+
+    CLAMS-Forth will have a dictionary split between flash and SRAM. The entire
+    system dictionary - Forth virtual machine, text interpreter and compiler -
+    will be in flash. Only words created by the user at run time will be in
+    SRAM.
+
+    Dr. Ting's _eForth for Discovery_ [@pintaske2019irreducible] gets around
+    this limitation by copying the system dictionary from flash to RAM at boot
+    time. But the hardware he used has more RAM and a more complete instruction
+    set than the RP2040. The RP2040 only has 264 kilobytes of SRAM and much of
+    that will be used for audio buffers. The more code and tables safely stashed
+    away in the 2 megabytes of flash, the better.
+
+2. Even with a dictionary entirely in SRAM at run time, constructing a `BL`
+function call instruction while compiling a user "colon" definition is a tricky process, difficult to document,
+understand and maintain. If you're interested in the details, see 
+[@pintaske2019irreducible, section 3.5.5 ]. I decided to switch to direct
+threading, rather than spend an extra week coding both a flash-to-SRAM
+relocation and an algorithm to compute a relative branch address that is then
+split into four separate bit fields of two 16-bit `thumb` instructions.
+
+## Status / roadmap
+
+1. There is a partial implementation of a subroutine-threaded version with
+inlining. I am leaving that in the repository in a development branch but won't
+be working on it unless I find that the direct-threaded version cannot meet
+speed requirements. The subroutine-threaded version is in branch `forth-stc`.
+
+2. The direct-threaded version is nearing a release. I expect to have the
+system dictionary, text interpreter and compiler done by January 1, 2024. It is
+in the branch `forth-dtc`.
+
+3. The next step is to implement synthesis algorithms. The general process will
+be to prototype them in Forth, converting to assembler when needed. That will
+create the requirements for the digital signal processing library.

The-Book-of-CLAMS/_quarto.yml

@@ -5,7 +5,7 @@ project:
 book:
   title: "The Book of CLAMS"
   author: "M. Edward (Ed) Borasky"
-  date: "11/3/2023"
+  date: "12/15/2023"
   chapters:
     - index.qmd
     - About-My-Music.qmd

The-Book-of-CLAMS/index.qmd

@@ -14,8 +14,8 @@ you will need to buy additional hardware and port some code.
 ## How does it work?
 
 `CLAMS` is a domain-specific language built on a Forth compiler /
-interpreter. The user connects to the board via a serial connection and
-enters `CLAMS` / Forth code interactively.
+interpreter. The user connects to the board via a USB or UART serial connection
+and enters `CLAMS` / Forth code interactively.
 
 ## Why Forth?
 

docs/About-My-Music.html

@@ -153,8 +153,7 @@ <h2 id="toc-title">Table of contents</h2>
 
   <ul>
   <li><a href="#algocompsynth" id="toc-algocompsynth" class="nav-link active" data-scroll-target="#algocompsynth"><span class="header-section-number">1.1</span> AlgoCompSynth</a></li>
-  <li><a href="#examples" id="toc-examples" class="nav-link" data-scroll-target="#examples"><span class="header-section-number">1.2</span> Examples</a></li>
-  <li><a href="#other-algocompsynth-projects" id="toc-other-algocompsynth-projects" class="nav-link" data-scroll-target="#other-algocompsynth-projects"><span class="header-section-number">1.3</span> Other AlgoCompSynth projects</a></li>
+  <li><a href="#other-algocompsynth-projects" id="toc-other-algocompsynth-projects" class="nav-link" data-scroll-target="#other-algocompsynth-projects"><span class="header-section-number">1.2</span> Other AlgoCompSynth projects</a></li>
   </ul>
 </nav>
     </div>
@@ -190,22 +189,17 @@ <h2 data-number="1.1" class="anchored" data-anchor-id="algocompsynth"><span clas
 <li><p>Physical modeling synthesis. A comprehensive reference can be found at <span class="citation" data-cites="paspweb2010">(<a href="references.html#ref-paspweb2010" role="doc-biblioref">Smith accessed 2023-10-21</a>)</span>.</p></li>
 <li><p>Spectral music. This is another advanced synthesis methodology; a recent reference is <span class="citation" data-cites="lazzarini2021spectral">(<a href="references.html#ref-lazzarini2021spectral" role="doc-biblioref">Lazzarini 2021</a>)</span></p></li>
 </ul>
+<p>My current home for published music is on Bandcamp at <a href="https://algocompsynth.bandcamp.com/" class="uri">https://algocompsynth.bandcamp.com/</a>. I may move to Gumroad, which allows publishing a broader range of digital media artefacts than music and music videos.</p>
 </section>
-<section id="examples" class="level2" data-number="1.2">
-<h2 data-number="1.2" class="anchored" data-anchor-id="examples"><span class="header-section-number">1.2</span> Examples</h2>
-<p>My current home for published music is on Bandcamp at <a href="https://algocompsynth.bandcamp.com/" class="uri">https://algocompsynth.bandcamp.com/</a>. The test case for CLAMS is a work in progress called “A Musical Clambake”, submitted as a proposal to the <a href="https://hybrid-livecode.pubpub.org/workshop2023">Hybrid Live Coding Interfaces Workshop 2023</a>:</p>
-<blockquote class="blockquote">
-<p>“A Musical Clambake”: A five minute algorithmic microtonal video produced on a Pimoroni PicoVision (https://shop.pimoroni.com/products/picovision?variant=41048911904851) using the CLAMS (https://algocompsynth.github.io/CLAMS) live coding system. “A Musical Clambake” revisits the birth of live coding, made possible by inexpensive personal computers and the Forth programming language, and explores what happens when the computer is a musical collaborator via dynamic stochastic synthesis.</p>
-</blockquote>
-<p>I am also planning to submit “A Musical Clambake” to the <a href="https://shop.pimoroni.com/pages/win-a-500-gift-card-in-our-picovision-competition">Pimoroni PicoVision Demoscene Competition</a>.</p>
-</section>
-<section id="other-algocompsynth-projects" class="level2" data-number="1.3">
-<h2 data-number="1.3" class="anchored" data-anchor-id="other-algocompsynth-projects"><span class="header-section-number">1.3</span> Other AlgoCompSynth projects</h2>
+<section id="other-algocompsynth-projects" class="level2" data-number="1.2">
+<h2 data-number="1.2" class="anchored" data-anchor-id="other-algocompsynth-projects"><span class="header-section-number">1.2</span> Other AlgoCompSynth projects</h2>
 <ol type="1">
-<li><p><a href="https://github.com/AlgoCompSynth/AlgoCompSynth-One">AlgoCompSynth-One</a>: This is a platform for doing high-performance digital signal processing and musical AI on NVIDIA GPUs. I currently support Windows 11 WSL Ubuntu 22.04 LTS and NVIDIA Jetson JetPack 5. If I can make everything work I will be supporting native Windows 11.</p></li>
+<li><p><a href="https://github.com/AlgoCompSynth/AlgoCompSynth-One">AlgoCompSynth-One</a>: This is a platform for doing high-performance digital signal processing and musical AI on NVIDIA GPUs. I currently support Windows 11 WSL Ubuntu 22.04 LTS and NVIDIA Jetson JetPack 5.</p>
+<p>If I can make everything work I will be supporting native Windows 11. <code>AlgoCompSynth-One</code> is based on <a href="https://mamba.readthedocs.io/en/latest/installation/mamba-installation.html">Mamba</a>, and Mamba runs on Windows, so unless other dependencies can’t be found, it should be possible.</p></li>
 <li><p><a href="https://algocompsynth.github.io/eikosany/">eikosany</a>: This is an R package for algorithmic composition with musical scales derived by Erv Wilson and students of his theories.</p></li>
 <li><p><a href="https://algocompsynth.github.io/consonaR/">consonaR</a>: This is an R package to perform the computations described in <em>Tuning, Timbre, Spectrum, Scale</em> <span class="citation" data-cites="sethares2013tuning">(<a href="references.html#ref-sethares2013tuning" role="doc-biblioref">Sethares 2013</a>)</span>. This is a superset of any such algorithms that will be deployed in CLAMS; only calculations that need to be performed during a performance need to be deployed in CLAMS.</p></li>
 </ol>
+<p>There is a fairly large overlap in functionality between <code>eikosany</code> and <code>consonaR</code>. There’s a plan for refactoring the two, but that work is on hold until I get more of <code>CLAMS</code> done.</p>
 
 
 <div id="refs" class="references csl-bib-body hanging-indent" role="list" style="display: none">