Merge pull request #327 from sshanks-kx/refactor

refactor

docs/architecture/index.md

@@ -7,9 +7,9 @@ keywords: hdb, kdb+, q, rdb, tick, tickerplant, streaming
 
 A kdb+ tick based archecture can be used to capture, process and analyse vasts amount of real-time and historical data.
 
-The following diagram illustrates the components that are often found in a vanilla kdb+ tick setup:
+The following diagram illustrates the components that are often found in a vanilla kdb+ tick setup:<br>
 
-![architecture](../img/architecture.png)
+![architecture](../img/tick_arch.svg)
 
 ## Components
 

docs/architecture/rq.md

@@ -30,6 +30,8 @@ If known/common queries can be designed, the RDB can load additional scripts to
 
 ### End-of-day
 
+![RDB_end-of-day](../img/rdb_end_of_day.svg)
+
 The end-of-day event is governed by the tickerplant process. The tickerplant calls the RDB [`.u.end`](#uend) function when this event occurs.
 The main end-of-day event for an RDB is to save todays data from memory to disk, clear its tables and uses IPC to instruct the HDB to be aware of a new days dataset for it to access.
 
@@ -42,6 +44,10 @@ The function [`.u.rep`](#urep) is then used to [replay the log](../kb/logging.md
 
 !!! Note "The RDB should be able to access the tickerplant log from a directory on the same machine. The RDB/tickerplant can be changed to reside on different hosts but this increases the resources needed to transmit the log file contents over the network."
 
+The following diagram shows the steps taken by an RDB to recover from a TP log:
+
+![RDB_end-of-day](../img/TP_log_recovery.svg)
+
 ## Usage
 
 ```bash
@@ -84,12 +90,12 @@ Where
 
 ### .u.end
 
-Perform end-of-day actions of saving tables to disk, clearing tables and running reload on HDB instance to make it aware of new day of data.
+Perform [end-of-day actions](end-of-day) of saving tables to disk, clearing tables and running reload on HDB instance to make it aware of new day of data.
 
 ```q
 .u.end[x]
 ```
-Where x is the date that has ended.
+Where x is the date that has ended, as a date atom type.
 
 Actions performed:
 
@@ -103,7 +109,7 @@ Actions performed:
 
 ### .u.rep
 
-Initialise RDB by creating tables, which is then populated with any existing tickerplant log. Will set the HDB directory to use at end-of-day.
+Initialise RDB by creating tables, which is then [populated with any existing tickerplant log](#recovery). Sets the HDB directory to be used at end-of-day.
 
 ```q
 .u.rep[x;y]

docs/architecture/uq.md

@@ -134,13 +134,13 @@ Where
 * `y` is list of syms used to filter table data, with empty sym representing for all table data
 
 If `x` is empty symbol, client is subscribed to all known tables using `y` criteria. This is achieved by calling .u.sub for each table in [`.u.t`](#variables). 
-It then returns a list of all the return values provided by .u.sub (i.e. a list of pairs comprising of table name and table definition).
-
-An error is returned if the table does not exist.
-
 For the subscribing  client, any previous registered in the given tables are removed prior to reinstating new criteria provided i.e. calls [`.u.del`](#udel).
+Calls [`.u.add`](#uadd) to record the client subscription.
 
-Calls [`.u.add`](#uadd) to record the client subscription and passes the returned values to client.
+Returns 
+* a two item list if x is an indivial table name. First item is the table name subscribed to as a symbol. Second item is an empty table (table schema).
+* a list of two item lists as described above for each individual table, if x is an empty symbol (i.e. subscribe to all tables)
+* an error if the table does not exist.
 
 ### .u.end
 

docs/basics/funsql.md

@@ -10,6 +10,12 @@ keywords: delete, exec, functional, kdb+, q, select, sql, update
 
 The functional forms of [`delete`](../ref/delete.md), [`exec`](../ref/exec.md), [`select`](../ref/select.md) and [`update`](../ref/update.md) are particularly useful for programmatically-generated queries, such as when column names are dynamically produced. 
 
+Functional form is an alternative to using a [qSQL template](qsql.md) to construct a query. For example, the folowing are equivalent:
+```q
+q)select n from t
+q)?[t;();0b;(enlist `n)!enlist `n]
+```
+
 !!! info "Performance"
 
     The q interpreter parses `delete`, `exec`, `select`, and `update` into their equivalent functional forms, so there is no performance difference.

docs/basics/peach.md

@@ -138,7 +138,7 @@ Each secondary thread has its own heap, a minimum of 64MB.
 
 Since V2.7 2011.09.21, [`.Q.gc[]`](../ref/dotq.md#gc-garbage-collect) in the main thread collects garbage in the secondary threads too.
 
-Automatic garbage collection within each thread (triggered by a wsfull, or hitting the artificial heap limit as specified with [`-w`](cmdline.md#-w-workspace) on the command line) is executed only for that particular thread, not across all threads.
+Automatic garbage collection within each thread (triggered by a [`wsfull`](../basics/errors.md#wsfull)), or hitting the artificial heap limit as specified with [`-w`](cmdline.md#-w-workspace) on the command line) is executed only for that particular thread, not across all threads.
 
 Symbols are internalized from a single memory area common to all threads.
 

docs/basics/qsql.md

@@ -5,8 +5,6 @@ keywords: exec, delete, kdb+, q, query, select, sql, update, upsert
 ---
 # QSQL query templates
 
-
-
 <div markdown="1" class="typewriter">
 [delete](../ref/delete.md)  delete rows or columns from a table
 [exec](../ref/exec.md)    return columns from a table, possibly with new columns
@@ -17,9 +15,6 @@ keywords: exec, delete, kdb+, q, query, select, sql, update, upsert
 The query templates of qSQL share a query syntax that varies from the [syntax of q](syntax.md) and closely resembles [conventional SQL](https://www.w3schools.com/sql/).
 For many use cases involving ordered data it is significantly more expressive.
 
-:fontawesome-brands-wikipedia-w:
-[Structured Query Language](https://en.wikipedia.org/wiki/SQL "Wikipedia")
-
 
 ## Template syntax
 

docs/basics/syscmds.md

@@ -342,10 +342,10 @@ Executing [`.Q.gc[]`](../ref/dotq.md#qgc-garbage-collect) additionally attempts
 Since V3.3 2015.08.23 (Linux only) unused pages in the heap are dropped from RSS during `.Q.gc[]`.
 
 When q is denied additional address space from the OS, it invokes `.Q.gc[]` and retries the request to the OS. 
-Should that fail, it will exit with `'wsfull`.
+If the subsequent attempt fail, the request exits with [`'wsfull`](../basics/errors.md#wsfull).
 
-When secondary threads are configured and `.Q.gc[]` is invoked in the main thread it will automatically invoke `.Q.gc[]` in each secondary thread. 
-If the call is instigated in a secondary thread – i.e., not the main thread – it will affect that thread’s local heap only.
+When secondary threads are configured and `.Q.gc[]` is invoked in the main thread, `.Q.gc[]` is automatically invoked in each secondary thread. 
+If the call is instigated in a secondary thread, it affects that thread’s local heap only.
 
 !!! detail "Notes on the allocator"
 

docs/kb/cpu-affinity.md

@@ -1,44 +1,91 @@
 ---
-title: CPU affinity | Knowledge Base | kdb+ and q documentation
-description: kdb+ can be constrained to run on specific cores through the setting of CPU affinity. Typically, you can set the CPU affinity for the shell you are in, and then processes started within that shell will inherit the affinity.
+title: CPU affinity – Knowledge Base – kdb+ and q documentation
+description: Kdb+ can be constrained to run on specific cores through the setting of CPU affinity. Typically, you can set the CPU affinity for the shell you are in, and then processes started within that shell will inherit the affinity.
 keywords: affinity, cpu, kdb+, kernel, linux, numa, q, solaris, unix, windows, zone_reclaim_mode
 ---
 # CPU affinity
 
 
 
 
-kdb+ can be constrained to run on specific cores through the setting of CPU affinity.
+Kdb+ can be constrained to run on specific cores through the setting of CPU affinity.
 
 Typically, you can set the CPU affinity for the shell you are in, and then processes started within that shell will inherit the affinity.
 
 :fontawesome-regular-hand-point-right: 
-[`.Q.w`](../ref/dotq.md#w-memory-stats) (memory stats)  
+[`.Q.w`](../ref/dotq.md#qw-memory-stats) (memory stats)  
 Basics: [Command-line parameter `-w`](../basics/cmdline.md#-w-workspace), 
 [System command `\w`](../basics/syscmds.md#w-workspace)
 
 
-## :fontawesome-brands-linux: Linux
+## Linux
 
-Use the `taskset` command to limit to a certain set of cores, e.g.
+:fontawesome-regular-hand-point-right: 
+[Non-Uniform Access Memory (NUMA)](linux-production.md#non-uniform-access-memory-numa-hardware)
+
+
+### Detecting NUMA
+
+The following commands will show if NUMA is active.
 
 ```bash
-taskset -c 0-2,4 q
+$ grep NUMA=y /boot/config-`uname -r`
+CONFIG_NUMA=y
+CONFIG_AMD_NUMA=y
+CONFIG_X86_64_ACPI_NUMA=y
+CONFIG_ACPI_NUMA=y
 ```
 
-will run q on cores 0, 1, 2 and 4. Or
+Or test for the presence of NUMA maps.
 
 ```bash
-taskset -c 0-2,4 bash
+$ find /proc -name numa_maps
+/proc/12108/numa_maps
+/proc/12109/task/12109/numa_maps
+/proc/12109/numa_maps
+...
 ```
 
-and then all processes started from within that new shell will automatically be restricted to those cores.
 
-You can also use numactl -S to specify the cores, perhaps combined with -l to always allocate on the current node or other policies discussed in the [linux production notes](linux-production.md#non-uniform-access-memory-numa-hardware):
+### Q and NUMA
+
+Until Linux kernels 3.x, q and NUMA did not work well together. 
+
+When activating NUMA, substitute parameter settings according to the [recommendations for different Linux kernels](linux-production.md#non-uniform-access-memory-numa-hardware).
+
+
+### Activating NUMA
+
+When NUMA is 
+
+-   **not active**, use the `taskset` command, e.g.
+
+	```bash
+	taskset -c 0,1,2 q
+	```
+
+	will run q on cores 0, 1 and 2. Or
+
+	```bash
+	taskset -c 0,1,2 bash
+	```
+
+	and then all processes started from within that new shell will automatically be restricted to those cores.
+
+-   **active**, use `numactl` instead of `taskset`
+
+	```bash
+	numactl --interleave=all --physcpubind=0,1,2 q
+	```
+
+	and set
+
+	```bash
+	echo 0 > /proc/sys/vm/zone_reclaim_mode
+	```
+
+You can change `zone_reclaim_mode` without restarting q.
 
-```bash
-numactl --interleave=all --physcpubind=0,1,2 q
-```
 
 ### Other ways to limit resources
 
@@ -68,7 +115,7 @@ psrset -e 2 bash
 ```
 
 
-## :fontawesome-brands-windows: Windows
+## Windows
 
 Start `q.exe` with the OS command `start` with the `/affinity` flag set
 

docs/kb/file-compression.md

@@ -176,7 +176,7 @@ kdb+ allocates enough memory to decompress the whole vector, regardless of how m
 
 This is reservation only, and can be accommodated by increasing the swap space available: even though the swap should never actually be written to, the OS has to be assured that in the worst-case scenario of decompressing the data in full, it could swap it out if needed.
 
-If you experience `wsfull` even with sufficient swap space configured, check whether you have any soft/hard limits imposed with `ulimit -v`.
+If you experience [`wsfull`](../basics/errors.md#wsfull) even with sufficient swap space configured, check whether you have any soft/hard limits imposed with `ulimit -v`.
 
 !!! tip "Memory overcommit settings on Linux"