Merge branch 'pjt-dev' into vite_config_base

.gitignore

@@ -15,4 +15,14 @@ _test.py
 __pycache__
 vite-dist
 dist
-.env
+dist/_mlv
+dist/_avocato
+.env
+*.spec
+python/build
+python/data
+python/mdvtools/static/*
+.ipynb_checkpoints
+
+python/requirements.txt
+python/dist/

README.md

@@ -27,7 +27,7 @@ Multi Dimensional Viewer (MDV) is web based application for analyzing, annotatin
 
 * Data can added and/or modified by the user
 
-* Diverse range of data sources (API calls,static files) can be used by implementing custom data loaders 
+* Diverse range of data sources (API calls, static files) can be used by implementing custom data loaders 
 
 * Runs in a web browser (installation not required for uploading and viewing data)
 
@@ -64,49 +64,44 @@ Then cd to the python directory
 cd path/to/mdv/python
 ```
 
-Install the required python packages
+Install `mdv` (using `editable` flag for development):
+
 ```
-pip install -r requirements.txt
+pip install -e .
 ```
 
 Open a python shell
 ```
 python
 ```
 
-Create an MDV project from the downloaded folder and display it in a browser. Either run the code from the mdv/python folder, add 
-the folder to the PYTHONPATH environment variable or add the path in the code `sys.path.append("/path/to/mdv/python")`
+Create an MDV project from the downloaded folder and display it in a browser:
+
 ```python
-from mdv.mdvproject import MDVProject
+from mdvtools.mdvproject import MDVProject
 p = MDVProject("/path/to/hyp_example_data")
 p.serve()
 ```
 
+This will open a browser window at http://localhost:5000/ but you will need to go to
+[http://127.0.0.1:5000](http://127.0.0.1:5000) to avoid permissions errors.
+
 ## Running on a server
 
 
 The default data storage is an hdf5 file which is a compressed files that allows random read/write access to the data. However, it cannot be accessed directly but requires some kind of wrapper e.g. h5py. Hence access via http calls directly is not possible and backend code is required to display an MDV project in a web page. However, it can be converted to simple continuous compressed blocks of data for each column and a json index . This allows direct access via an http request with a range header:-
 
 ```python
-from mdv.mdvproject import MDVProject
+from mdvtools.mdvproject import MDVProject
 p = MDVProject("/path/to/hyp_example_data")
 p.convert_to_static_page("/path/to/myapp/")
 ```
 
 The function also creates a simple home page for the project (index.html), which can be customized. The folder can then be put on the server and accessed via:-
 ```
-http://myserver.com/path/to/myapp
+https://myserver.com/path/to/myapp
 ```
 
-Images and JavaScript files are obtained from /static/mdv/js and /static/mdv/img , so the python/static/mdv folder needs to be copied to the /server folder of the server. 
-
-
-No back end architecture is required , although the server needs to accept range headers and have the following headers on responses from the project folder and /static/mdv/js folder in order to allow SharedArrayBuffers and Webworkers:-
-
-```
-Cross-Origin-Opener-Policy:same-origin
-Cross-Origin-Embedder-Policy:require-corp
-```
 
 
 ## Development

desktop.config.js

@@ -5,15 +5,14 @@ module.exports = (env)=> {
 	return {	  
 	   	entry: './src/modules/desktop_index.js',
   	   	output: {
-    		   path: path.resolve(__dirname,"python/mdv/static/mdv/js"),
+    		   path: path.resolve(__dirname,"python/mdvtools/static/js"),
     		   filename: 'mdv.js',
 			   assetModuleFilename: '../img/[name][ext]',
-
   	   	},
+		// devtool: 'eval', //for marginally better debugging
 		resolve: {
 			extensions: ['.ts', '.js', '...']
 		},
-		mode:env.mode,  
   	   	module:{ 			
      		rules:[
 				{
@@ -27,7 +26,7 @@ module.exports = (env)=> {
 					test: /\.(png|svg|jpg|gif|eot|ttf|woff|woff2)$/,
 					type: 'asset/resource',
 					generator:{
-						publicPath:"/static/mdv/img/"
+						publicPath:"static/img/"
 					}		
 				},	
 				{

dev.config.js

@@ -0,0 +1,47 @@
+const path = require('path');
+webpack=require("webpack");
+module.exports = (env) => {
+	return {
+        mode:"development",
+
+        devtool: 'inline-source-map',
+		devServer: {
+			headers: {
+				"Cross-Origin-Opener-Policy":"same-origin",
+				"Cross-Origin-Embedder-Policy":"require-corp"
+			},
+			static:env.folder
+		  },
+
+	   entry: "./src/modules/desktop_index.js",
+
+  	   output: {
+    		   filename: 'static/js/mdv.js',
+  	   },
+		 resolve: {
+            extensions: ['.ts', '.js', '...']
+        },
+  	   module:{ 
+
+     		rules:[
+			{
+				test:/\.css$/,
+				use:[
+					"style-loader",
+					"css-loader"
+				]
+			},
+			{
+				test: /\.(png|svg|jpg|gif|eot|ttf|woff|woff2)$/,
+				type: 'asset/resource'
+
+			},
+			{
+				test: /\.tsx?$/,
+				use: 'ts-loader',
+				exclude: '/node_modules'
+			}
+		]
+  	}
+}
+};

docs/extradocs/chartmanager.md

@@ -1,29 +1,33 @@
 
 The ChartManager object links DataSources, loads column data on demand and manages charts/widgets. It is the main interface for interacting with MDV. The ChartManager's constructor requires list of DataStore configs, a dataloader object, a config for construction and optionally a listener. 
-```
-    const cm =  new ChartManager("mydiv",datasources,dataloader,config,listener)
+```js
+    const cm = new ChartManager("mydiv", datasources, dataloader, config, listener)
 ```
 * **mydiv** - the id or the element to house the app
-* **datasources** - list of DataStore [configs]{@tutorial datasource}, these configs should also include a size parameter showing the number of rows in the data set
-* **dataloader** - a dataloader which comprises of three parameters
+* **datasources** - list of `DataStore` [configs]{@tutorial datasource}, these configs should also include a size parameter showing the number of rows in the data set
+* **dataloader** - a dataloader which can have the following parameters
     * **function** - this [function]{@tutorial dataloader} accepts a list of columns and returns a promise (not needed if all data is to be loaded from files)
-    * **viewLoader** - a function that will return the [view]{@tutorial views} given the view name  
-    * **files** - specifies static files (tsv,csv or json), which contain the data to display. Useful for small amounts of data (100 000s rows) and testing. If all the data is to be loaded dynamically then this is not required.
+    * **viewLoader** - a function that will return the [view]{@tutorial views} given the view name
+    * **rowDataLoader** - an optional function that accepts a datasource name and index and returns an an object with data. The
+    function will be called if the datasource's config contains `row_data_loader:true` and `DataHighlighted` is invoked.
+    * **binaryDataLoader** - an optional function that accepts a datasource name and the name of a the binary data to load. the datasource's config should contain `binary_data_laoder: true`
+    * **files** - specifies static files (tsv, csv or json), which contain the data to display. Useful for small amounts of data (100 000s rows) and testing. If all the data is to be loaded dynamically then this is not required.
     ```json
     [
         {
-            "type":"tsv",
-            "dataSource":"cells",
-            "url":"data/cell_all_archr.tsv"
+            "type": "tsv",
+            "dataSource": "cells",
+            "url": "data/cell_all_archr.tsv"
         },
         {
-            "type":"tsv",
-            "dataSource":"genes",
-            "url2":"data/genes.txt"
+            "type": "tsv",
+            "dataSource": "genes",
+            "url2": "data/genes.txt"
         }
     ]
     ```
-* **listener** An optional listener function, although this can be added later with the *addListener* method.
+* **config** - A config 
+* **listener** An optional listener function, can also be added later with the *addListener* method.
 
 
 
@@ -34,7 +38,7 @@ Can be added with the method *addListener(id, function)* and removed with *remov
 The listener should be a function which receives the type of event, the ChartManager object and any data associated with the event. A typical listener would be:-
 
 ```js
-    (type,cm,data)=>{
+    (type, cm, data) => {
         switch(type){
             case "view_loaded":
                 ..do stuff with data
@@ -48,11 +52,66 @@ The listener should be a function which receives the type of event, the ChartMan
 
 The types of listeners are:- 
 
-* **chart_added**  Called When a chart is added with notify=true e.g. when a user adds a chart. The data received is the chart object itself.
-* **chart_removed** Called When a chart is removed with notify=true e.g. when a user removes a chart. The data received is the chart object itself.
+* **chart_added**  Called When a chart is added with `notify=true` e.g. when a user adds a chart. The data received is the chart object itself.
+* **chart_removed** Called when a chart is removed with `notify=true` e.g. when a user removes a chart. The data received is the chart object itself.
 * **state_saved** Called when the user saves the state. The data being the state object
 * **view_loaded** Called when a view has been completely loaded i.e. all data retrieved and all the charts added. The data being passed is the view that was loaded
-* **filtered** Called when a DataStore is filtered, passing the Dimension that has done the filtering
+<!-- todo add links -->
+* **filtered** Called when a `DataStore` is filtered, passing the `Dimension` that has done the filtering
+
+
+
+## state_saved
 
+Called when the user saves the data. The object passed to the listener consists of the following:-
 
+* `view` - A config containing all data for the view
+* `currentView` - 
+* `all_views` -  A list of views 
+* `updatedColumns` -  a dictionary with a entry for each datasource 
+    * `columns` - a list of all columns that have either been updated or added containing:
+        * `metadata`- the columns metadata
+        * `data` an array of the column's raw data
+    * `added` -  a list of all column id(fields) that have been added
+    * `removed` -  a list of all columns that have been deleted
+    * `colors_changed` -a list of columns whose color scheme has changed
+        * `column` - the id of the column
+        * `colors` -the new color scheme (list of hex values)
 
+* `metadata` - a dictionary of all datasources whose metadata has been updated, with each entry being a dictionary of the parameter and value
+
+```json
+{
+    "view": {....},
+    "cuurentView": "default",
+    "all_views": ["default", "myview"],
+    "updatedColumns": {
+        "cells": {
+            "columns": [
+                {
+                    "metadata": {"field": "new annotations", ...},
+                    "data": [2, 1, 2.....]
+                },
+                {
+                    "metadata": {"field": "annotations", ...},
+                    "data": [0, 0, 2.....]
+                }
+            ],
+            "added": ["new annotations"],
+            "removed":[ ],
+            "colors_changed": [
+                {
+                    "column": "annotations",
+                    "colors": ["#4532FF", ....]
+                }
+            ]
+        }
+    },
+    "metadata": {
+        "cells": {
+            "param1": {....},
+            "param2": {....}
+        }
+    }
+}
+```

docs/extradocs/charts.md

@@ -6,11 +6,11 @@
 
 * **param** - the columns the describe a genomic location e.g. chr, start and finish
 
-* **tracks** An array of track objects. The very minimum it should contain a track with id '_base_track' that corresponds to the DataSource's data i.e. a gzipped bed file with a tabix index containing the columns chr, start. end and the index(id) of the row in the DataStore
+* **tracks** An array of track objects. The very minimum it should contain a track with id `'_base_track'` that corresponds to the DataSource's data i.e. a gzipped bed file with a tabix index containing the columns chr, start. end and the index(id) of the row in the DataStore
 
 * **feature_label** - the column to use in the DataStore to label the features (optional)
 
-* **color_wig_tracks_by** - the column to color any tracks by. The track must have an id of <column_field>|<value>
+* **color_wig_tracks_by** - the column to color any tracks by. The track must have an id of `<column_field>`|`<value>`
 
 * **view_margins**
     * type - either percentage (of the feature being viewed) or absolute (in bp)
@@ -22,14 +22,14 @@
 Charts should have a `getImage()` method that takes a function and a type, either 'svg' or 'png'. The callback should pass a serialized svg string in the case of svg or a canvas element in the case of a png.
 
 ```js
-getImage(callback,type){
-    if (type=="svg"){
+getImage(callback, type){
+    if (type === "svg"){
         this.getSerializedSVG().then((svg)=>{
             callback(svg);
         });
     }
-    else if type="png"{
-        callback.this.canvas;
+    else if type === "png"{
+        callback(this.canvas);
     }
 }
 
@@ -42,11 +42,14 @@ For charts which extend `SVGChart`, this function has already been implemented a
 ```js
 
 addToImage(){
-    this.svg.append("text").attr("text","Added Text");
+    this.svg.append("text").attr("text", "Added Text");
 }
 
 removeFromImage(){
-    this,svg.select("text").remove();
+    this.svg.select("text").remove();
 }
 
-```
+```
+
+# Downloading data
+The chart should have a `getChartData` method which returns a text blob e.g. tab delimited table