Allow to get the data from a function instead of neo4j cypher

[thebestnom]

Closes #188 and #185
Also closes #35 I think

[dolleyj]

Does the recent activity on this indicate this will be a new feature in an upcoming release, or is this ticket consolidation?

Neovis is a candidate of mine, but it's a non-starter if neo4j credentials are stored directly in the javascript. I'd prefer flask to serve the graph data to neovis.

[thebestnom]

I want to work on that, but I don't have a lot of free time and a library we dependent 9n (vis-data) have a broken types on newer typescript that I need to fix first

[dolleyj]

@thebestnom , thanks. I appreciate the quick response. Cheers

[thebestnom]

@dolleyj hey, just published 2.1.0-alpha2 with the PR code if you want to test, API not final but it's good enough for now

just add to the config dataFunction with a iterator function that yields Records and it should work

[nielsjansendk]

@thebestnom I would like to test this, but I am not super strong in javascript. As I understand it, instead of passing

const config = {
      neo4j: {
                serverUrl: "neo4j://locahost:7687",
                serverUser: "neo4j",
                serverPassword: "password",
      },
     }
     

 You would instead pass

     const config = {
                dataFunction: myDataFunction(),
     }

And the datafunction should basically do the same as this:

function myDataFunction() {
let recordCount = 0;
			const _query = query || this.#query;
			const session = this.#driver.session(this.#database ? { database: this.#database } : undefined);
			const dataBuildPromises: Promise<unknown>[] = [];
			session.run(_query, parameters)
				.subscribe({
					onNext: (record) => {
						recordCount++;
						dataBuildPromises.push(this.#createSingleRecord(record));
					},
					onCompleted: async () => {
						await Promise.all(dataBuildPromises);
						await session.close();

						this.#completeRun();
						this.#events.generateEvent(NeoVisEvents.CompletionEvent, { recordCount });
					},
					onError: (error) => {
						this.#consoleLog(error, 'error');
						this.#events.generateEvent(NeoVisEvents.ErrorEvent, { error });
					}					
				} as Neo4jCore.ResultObserver);
				}

But replacing all references to the driver with my own instance of neo4j-driver. Is that the gist of it? The dataFunction has run the query and return the result of the run. Or do I just return the record from the dataFunction? There seems to be some sort of housekeeping with the return result of the datafunction, where createSingleRecord is called on the result. So what is the data function expected to provide? Is it possible that you could add an example with a dataFunction that uses neo4j-driver to execute some cypher on the server and return the result. Even a simple example would help a lot.

I am also not sure how the query gets passed to the datafunction. If my configuration has an initialCypher definition, shouldn't I be able to pass that to my datafunction?

[thebestnom]

Close, but first, don't accidentally call in the config, so

const config = {
                dataFunction: myDataFunction //make sure without brackets
     }

And it need to return Iterable or async iterable, that means Array, or generator with yield or async generator with yield
So somthing like

import * as neo4j from "neo4j-driver-core"; // not sure if from "neo4j-driver" or from core but somthing like that
async function *myDataFunction() {
    const data = await (await fetch(url)).json(); //you're server should return original neo4j record 
   // I think that for now youll have to convert it back to record class so somthing like
   for (const re of data) {
       yield neo4j.Record(re));
   }
   // Or somthing like that, that's a test api but it should work
}

[nielsjansendk]

@thebestnom I am going to give it a shot.

[thebestnom]

Make sure to see what I edited in the function 😅

[nielsjansendk]

@thebestnom I think maybe there is a problem. It is trying to connect to localhost:7687. However, in my dataFunction, I have specifically instructed it to connect to: 'http://localhost:7474/db/neo4j/tx/commit'; I think maybe it tries to connect to localhost:7687 because I have removed the neo4j part of my config, and then it falls back to the default. But if there is a dataFunction it really shouldn't attempt to talk to directly to neo4j from neovis, right? If that part of the config is missing, it should look for a datafunction and if that is not there it should throw an error of some sort.It is quite possible I didn't program the datafunction properly, but I still think t should never attempt to connect directly to neo4j if it has not been configured to do so. That can lead to confusing errors.

[thebestnom]

Yeah, guessed somthing was not right by not testing that in #init but it shouldn't make the render to fail, it should just log railing to connect but continuing aniway, no?

[thebestnom]

Ok, published 2.1.0-alpha4 with a working exmaple, tested that it works and should be good
https://github.com/neo4j-contrib/neovis.js/blob/add-function-data-input/examples/simple-dataFunction-example.html
https://github.com/neo4j-contrib/neovis.js/blob/add-function-data-input/examples/server/server.ts

[nielsjansendk]

@thebestnom I got it to work, but I had to remove the group option from the labels config. This did not work:

var config = {
                containerId: "viz",
                labels: {
                    Character: {
                        label: "name",
                        value: "pagerank",
                        group: "community"
                    }
                },
                relationships: {
                    INTERACTS: {
                        value: "weight"
                    }
                },
                dataFunction: test
            };

The error message was:

caught (in promise) Error: updateGroupOptions: group values in options don't match.
    at Function.value (vis-network.js:24254:22)
    at Function.value (vis-network.js:24372:9)
    at t.value (vis-network.js:23753:16)
    at new t (vis-network.js:23717:40)
    at t.value (vis-network.js:24794:28)
    at t.value (vis-network.js:24734:16)
    at t.value (vis-network.js:24719:5)
    at zj.setData (vis-network.js:43130:58)
    at new zj (vis-network.js:42876:10)
    at TB.GB (neovis.ts:747:137)

But when I remove the group option, it works and draws the graph.

[nielsjansendk]

@thebestnom Would if be possible to call the dataFunction with a parameter? Such that the cypher code can be specified by the initialCypher config option? Suppose I want to call it with a node id and return something like 'MATCH (n:Node{id: 12345})<-[r:KNOWS]-(m:Node) return n,r,m' I don't want to write a datafunction for every single possible id. So basically, let the dataFunction take initialCypher as input.

[nielsjansendk]

@thebestnom Sorry, I realise that wouldn't be a good idea since the server handles cypher. However, I do need to be able to specify a parameter for the url. So the function should be something like:

async function *test(node_id) {
	const data = await (await fetch("http://localhost:8080/data/node_id")).json();
	for(const da of data) {
		yield da;
	}
}

I just don't know how you would specify a parameter in the config object, since you are not actually calling the function from the config. It would have to be in another part of the config object, right? So neovis can check for it and call the dataFunction with it.

[nielsjansendk]

@thebestnom Also, for some weird reason using the dataFunction instead of connection directly to neo4j ignores all styling in config->relationships. It uses the correct styling from config->labels.

[thebestnom]

Hmm.. I don't think that's a good idea to try to send parameter, I think a better approach would be to get the IDs you need in the function, somthing like

async function *test() {
        
	const data = await (await fetch("http://localhost:8080/data/all_ids")).json();
	for(const id of data) {
                const single_id_data = await (await fetch(`http://localhost:8080/data/${id}`)).json();
		yield single_id_data;
	}
}

Or somthing like that

[thebestnom]

@nielsjansendk Also looking at the group bug

[nielsjansendk]

@thebestnom Not if you have millions and millions of nodes. The server will crash trying to send them all. So that is definitely a no. There needs to be some way to pass information through to the datafunction.

[nielsjansendk]

@thebestnom The database I am working on currently has a nodeCount of 141,826,440 and a relation count of 217,842,760. So I can't just send it all over and sort it later. It would only work for very small graphs.

[nielsjansendk]

@thebestnom Could this part be changed like this:

async #runFunctionDataGetter() {
		let recordCount = 0;
		try {
			const dataBuildPromises: Promise<unknown>[] = [];
			//change here:
			for await (const record of this.#config.dataFunction!(this.#config.dataFunctionParams)) {
				dataBuildPromises.push(this.#createSingleRecord(record));
				recordCount++;
			}
			await Promise.all(dataBuildPromises);
		} catch (error) {
			this.#events.generateEvent(NeoVisEvents.ErrorEvent, { error });
			return;
		}
		this.#completeRun();
		this.#events.generateEvent(NeoVisEvents.CompletionEvent, { recordCount });
	}

And then datafunctionParams is an empty hash as a default.

[thebestnom]

I can do that or get it from the parameters, but I think that you can also get the parameters from outside 😅

[nielsjansendk]

@thebestnom Which means that the datafunction need to check the parameters. Like I could set config.dataFunctionParams.my_parameter: 3 Then in my dataFunction I could look for the value for my_parameter, check that it is valid and send it to the server. If there are options I don't want to send to the server, I need to handle it. If I build a datafunction to get a specific path for a specific node, I need to accept that it can get that kind of path for all nodes. But if I want my users to be able to draw an graph for all possible nodes that is fine. Otherwise, I will have to restrict it in the datafunction. I can't see how that can be abused, but maybe I am missing something.

[thebestnom]

That noy what I said, what I meant from outside I meant somthing like

let dataParameters = {somthing : 5}
let config = {} // normal config
function dataFunction() {
   //Use global dataParameter
}

neovis.render()
//Wait render complete
dataPrameter.somthing = 6
//rerender

The second thing I suggested was

function dataFunction(parameters) {
}
const config = {}// normal config
neovis.render(undefined, {somthing: 5})// the first is saved for query for now, and {somthing:5 } will be sent to dataFunction

For now global works, and I will probably only add the second option, I don't think adding it to the config makes sense as the config should be mostly static unless somthing bug needs change

[nielsjansendk]

@thebestnom Ah, I get what you are saying. If I define my datafunction as a nested function inside my draw function, it will have access to the variables that are available inside my draw function. Yeah, that totally works. Now it's just the group thing and the missing styling of the relationships, then it is a very useful feature. Maybe you can update the example to have the datafunction as a nested function inside draw()?

[thebestnom]

I'll try later today, also I think Ill do add the parameters, and the nested

Also the group is vis bug, Ill fix that also
And style is probably how I handled deserializing of the neo4j data Ill also fix that

[thebestnom]

Will not work on sptial or temporal data for now as I don't have the energy to desirialize them

[nielsjansendk]

@thebestnom Is is possible to have NeoVis.objectToTitleHtml take arguments? Like this: NeoVis.objectToTitleHtml(node, ['my_id','my_date["year"]["low"]')?

[nielsjansendk]

@thebestnom Also, there does not seem to be a 2.1.0-alpha5 tag in this repository.

[thebestnom]

@thebestnom Is is possible to have NeoVis.objectToTitleHtml take arguments? Like this: NeoVis.objectToTitleHtml(node, ['my_id','my_date["year"]["low"]')?

@nielsjansendk yes, just do (prop) => objectToHtml(prop, config) instead of using it directly

Yeah, I forgot to push the tag, but I did push it to npm

[nielsjansendk]

@thebestnom I'm sorry, I don't get how that works. Suppose this is my configuration:

 [NeoVis.NEOVIS_ADVANCED_CONFIG]: {
                function: {
                    title: (node) => NeoVis.objectToTitleHtml(node, ['my_id',
                                                                    'my_date'])
                    }
                    }

The date now comes out as an object. Would it be something like:

 [NeoVis.NEOVIS_ADVANCED_CONFIG]: {
                function: {
                    title: (node) => NeoVis.objectToTitleHtml(node, ['my_id',
                                                                    objectToHtml('my_date', ['year','month','day'])])
                    }
                    }

Where does objectToHtml come from? What if I wanted to call toStandardDate from the neo4j driver on the property, would that be possible?

[thebestnom]

Yeah, I said temporal are not supported for now as it means funding and desriallzing more objects and I didn't had time for that, so currently you cant call to string as it's dump object, but you can currently edit the object directly before passing it to objectToHtml like

// Import neo4j driver
import * as Neo4j from "neo4j-driver";
(node) => {
    const date = node.properties.my_date;
    node.properties.my_date = new Neo4j.types.Date(date.year, date.month, date.day) //I think, check real api, this is educated 😅
    return NeoVis.objectToHtml(node, ['id', 'my_date'])
}

[nielsjansendk]

@thebestnom I am getting some weird negative integers. The problem is definitely in the driver, not in neovis, so this is just a heads up. These are very large integers, so it might have something to do with that.

[thebestnom]

Hmm... The high, end should what allow the big integers, which means somewhere is broken, the seriallization but might be my deseriallzation

[nielsjansendk]

@thebestnom What I did was get the result directly from the driver. It is definitely broken from there, even before it gets to neovis. The value is an integer, but it's not returned correctly. Neither the high or the low value has anything to do with the integer that it's supposed to be.

[nielsjansendk]

The value is supposed to be 655005720010910 I am getting a value which has {"low": 1732534430,
"high": 152505}. It is consistent, so every time I am expecting 655005720010910 I am getting the same low and high. I can turn them into strings on my nodes so it's solvable. But it's weird.

[thebestnom]

The number is too high for js, so they represent it by 2 number, Im pretty sure the low means how many int does it pass or somthing like that, not sure though, not sure also if that a bug on my end Ill test that later

[thebestnom]

They did it before BigInt was a thing 😅

[thebestnom]

Or making more sense it's the lower and higher part of a double sized int

[thebestnom]

https://stackoverflow.com/questions/42645342/neo4j-return-an-integer-instead-of-high-0-low-10

Now just why it's doesn't deseiallized correctly

[thebestnom]

Ok, found out we where working with integer wrong all this time, fixed that
published 2.1.0-alpha7, added tests and deserialize all neo4j types including the Time I asked you to do workaround for
I want to close this soon as I think it works correcly, but I do want to know if there is anything that I can do better with naming of the API as I didn't think hard enough for that (if you have better name for dataFunction please say so)

[nielsjansendk]

@thebestnom I am still getting errors for dates, but not datetimes. I hope that's an easy fix, since they are just a less complex version of datetime.

I think maybe a name with with hook, or maybe dataConnection? I'm not super good with naming things.

[thebestnom]

Can you send here how the date look in the json? Maybe I didn't know some fields are optional

[nielsjansendk]

@thebestnom sorry, I was wrong, dates actually work, I hadn't reloaded my app with the new version. Everything looks great now.

[thebestnom]

Anything more you think the api needs? I think about publishing it in the following days, I think maybe renderWithFunction

[nielsjansendk]

@thebestnom I'm sure other people will have different problems than I have, but it works for me. Maybe an example where the dataFunction is defined inside the draw function? To show how the datafunction can get access to variables.

[thebestnom]

Have you looked at the new examples? Can you say how can I make it better?

[nielsjansendk]

@thebestnom I would like an example where the draw function is like this, to illustrate how to call the draw function with a parameter and pass it trough to the datafunction.

		function draw(my_variable) {
		     async function *getData() {
                     const data = await (await fetch(`localhost:8000/getMyData/${my_variable}`)).json();
                    for(const da of data) {
                        yield da;
                   }
                }

			var config = {
				containerId: "viz",
				consoleDebug: true,
				labels: {
					Character: {
						label: "name",
						value: "pagerank",
						group: "community",
						[NeoVis.NEOVIS_ADVANCED_CONFIG]: {
							function: {
								title: NeoVis.objectToTitleHtml
							},
						}
					}
				},
				relationships: {
					INTERACTS: {
						value: "weight",
						[NeoVis.NEOVIS_ADVANCED_CONFIG]: {
							function: {
								title: NeoVis.objectToTitleHtml
							},
						}
					}
				},
				dataFunction: getData
			};

			viz = new NeoVis.default(config);
			viz.render();
			console.log(viz);

		}

[nielsjansendk]

@thebestnom Just a heads up: this can't deal with NaN. By mistake I had some calculations that resulted in NaN for a node property, and the way I found out was that the visualisation would not render. It might be a good idea to throw an error message when a number is a NaN.

[thebestnom]

That's a bug 😅 ill fix that it will return NaN string 😅

[thebestnom]

published as 2.1.0, I fix the bug on next patch 😅

[thebestnom]

@nielsjansendk fixed in #349
published in 2.1.1-alpha1 if you want to test before I merge

[PeterPilley]

Just discovered this thread, does this mean we can use the output of neo4j result.records.map and then load it into neovis without using cypher directly in neovis?

if so is there a working example?

[thebestnom]

Yep, @PeterPilley there is an example with example server in examples directory