Further expansion of the relations system

[Didas-git]

As the majority is probably aware now the RedisGraph module EOL was announced.
With this we pretty much lost our ability to use graphs and to have an easy way to create complex relations, thats where this expansion comes in.

This was actually my original idea for relations but i ended up simplifying it to what it is now because i though it was way to overcomplicated for a simple orm library specially when we had RedisGraph that could handle the majority of the work for us, with that said, the system i want to propose is not completely original and its pretty much a copy of how surrealdb handles relations internally.

The current system

This is the currently implemented "primitive" system, it just stores ids into an array and when you use auto-fetch it just loops through it and gets them, pretty simple and pretty straight forward.

// Lets use this simple schema to show how it works internally
const userSchema = client.schema({
    friends: { type: "reference", schema: "self" }
});

const userModel = client.model("User", userSchema);

// We can "create" relations by passing an array of ids
// This is extremely unsafe since it relies on the user to know everything about the ids
await userModel.createAndSave({
    friends: ["Nekdis:V1:User:3"]
})

// That is why the `ReferenceArray` was crated
// Here is how to use it, and you can find its definition at:
// https://github.com/Didas-git/Nekdis/blob/main/src/utils/reference-array.ts
const aUser = userModel.create();

// You can use the `reference` property to help you in this process
// It accepts ids and other documents
// However this is not perfect since it doesn't do any validation on the ids (strings) given to it
aUser.friends.reference("Nekdis:V1:User:3");

await userModel.save(aUser);

The new proposal

The idea of expanding this system as said before would be to create a system just like surreal's and use the search functionality with it.

Where to start?

Well, to start it there is a huge decision that has to be made, this being, do we deprecate the current system and override it with the new one or do we keep the current one and just create a new type (relation) for the new system?

This is something i don't want to decide without getting any feedback since there might be some people who enjoy the simplicity of the current system and would not want to change to a more complex system that also comes with its downsides, which gets us to the next point.

What are the major downsides to this new approach?

As everyone knows by now, redis is a in-memory database which means its not the cheapest thing to run at scale and this system would bring noticeable memory usage, "why?" you mays ask, for that lets direct ourselves to the next point.

How does it work?

I will use some code examples to try and explain this as best as i can and to start we need to understand what really are what im calling here "complex relations".

Complex relations

Complex relations are relations that have and entry point and an exit point (in & out) and can contain metadata about that said relation. They would be represented entirely outside of the main document and would be represented as follow:

{
    "in": "entry relation id",
    "out": "exit relation id",
    // ... any additional metadata
    // lets say this specific relation has an age field on its meta
    // It would be represented just like a normal json field like:
    "age": 18
}

External representation

What do i exactly mean by a relation being represented externally?

This just means that the relation data would not be present in the main document and would be just a reference to an external id.

So lets imagine we have a user in our database with a friends list, this is how it would be represented internally:

// Nekdis:V1:User:1
{
    "name": "DidaS",
    "age": 18,
    "friends": ["Nekdis:V1:User-Relation-friends:someRandomUUID"]
}

// Nekdis:V1:User-Relation-friends:someRandomUUID
{
    "in": "Nekdis:V1:User:1",
    "out": "Nekdis:V1:User:2"
}

This would also allow us to index this relations with RediSearch to have great query functionality without having to do much work ourselves.

The syntax

One of the biggest issues with this proposal is finding a syntax that everyone is happy with. The main idea is to just follow the Builder Pattern just like how we currently do for RediSearch.

However having proper naming for defining this relations in a human readable and interactive way is the hard part.

This are my initial ideas for the syntax and im fully open to discussions about the topic.

// Lets use this simple user schema for the example
const userSchema = client.schema({
    name: { type: "string", required: true },
    age: { type: "number", required: true },
    // For this example lets assume we got rid of the old simple references
    // The `meta` property could be passed to fully type the metadata the document has
    friends: { type: "reference", schema: "self", meta: {
        age: "number"
    } }
});

const userModel = client.model("User", userSchema);

const aUser = userModel.create({
    $id: 1,
    name: "DidaS",
    age: 18
});

// Create a new reference
// The `references` method would accept an id (string) or a document
/*
The huge dilemma here is: "How do we fill the meta properties"
Well this is why the meta types by default are optional.
You can either pass the manually with `where(field).eq(value)` (just like RediSearch)
Or you can pass a boolean as the second argument to `references`
What this will do is fetch the ids and use what they return to create the metadata
Further details on this will be given later on
*/
aUser.friends.references("Nekdis:V1:User:2");

await userModel.save(aUser);

// Finding via relations
// Other aliases for `references` would include `is`, `are`, `relation` and `relates`
// The possible queries are `from` (aliases: `in`, `entry`) and `to` (aliases: `out`, `exit`)
// The `from` query searches for relations where the entry point is the given id
// While the `to` query searches for relations where the exit point is the given id
// To add metadata constrains you use the same syntax as the normal search (`.where().eq()`, etc...)
await userModel.search().references("friends").from("Nekdis:V1:User:1")

Auto Fetching

Auto fetching is a more complicated topic in this case.

There are 2 ways i want to go about this:

• Fetch-All
  • As the name implies you just fetch all the existing relations of that document
• Fetch-Constrain
  • This would only fetch and return the queries where the metadata as the one that you searched for

// Using the same schema from the example before

// Fetch-All
await userModel.get("Nekdis:V1:User:1", true);

// Fetch-Constrain
await userModel.get("Nekdis:V1:User:1", { age: 21 });

[Didas-git]

There is a mix of reference and relation on this post which makes it a little bit confusing.
From now on reference is the currently implemented system that will stay in the library with no plans for removal and relation will be the new system.
I do want to implement this, however since it is a massive change that will take a chunk of my time i want to firts fix some internal problems and add features that have more impact first like #10

[Didas-git]

I have been working on a prototype for this and for now what i can say is that it will most likely use a lua script so it can be Atomic and not cause any issues.
But for that i will include an option to enable it, by default injecting any type of luas into redis will be disabled so to use relations and probably other features to come you will have to allow it.

[Didas-git]

An update on the relation system (0.13)

The "spec"

As of Nekdis 0.13 there will be a new option on the client called inject (or enabledInjections still deciding on that) that will make it so the client can inject (add) lua scripts to your redis instance, this allows us to make relations Atomic & bring new features like atomic update in the future.

As for the functionality of the script itself it is divided into 2 categories each one with its sub categories.

Creating relations

There are 2 commands to create relations JSONCR and HCR for JSON and HASH respectively, they receive exactly the same arguments in the same format:

FACLL JSONCR 3 inId outId omitId field meta

Where:

• inId - The id/key you are creating the relation for
• outId - The id/key you are relating to
• omitId - The id/key where the metadata will be stored
• field - The "field" where this relation is saved, its actually used to create the Set to save the omitted keys at.
• meta - The metadata is a stringified JSON object with the data you will use as metadata

First step

The first step of creating a relation is creating the key where all the metadata will be saved, the type of the key can be either JSON or HASH depending on the command you use.

Second Step

The second step is to append the id to the set that stores all the relations.

Internally all it does is SADD iniId:field omitId

Getting all the relations

The script also provides you a JSONGR and HGR function to fetch all of the relations of said field.

The syntax is:

FCALL JSONGR 1 key field

Where:

• key - The id/key that you want to fetch the relations of
• field - The field name of the relation

This is easier to see if you look at the cli examples.

How it works

The first step is to get all the omitted ids from the set using SMEMBERS so they can be processed.

Then we process them by fetching them, getting the out field and fetching the respective id/key.

Redis-cli Example

Lets setup 2 different users and create a relation between them with some metadata

JSON.SET user:1 $ '{"name": "DidaS"}'
JSON.SET user:2 $ '{"name": "Leibale"}'

FCALL JSONCR 3 user:1 user:2 user:contacts:1 contacts '{"company": "Redis"}'

After running this you will see 2 new keys on your database:

A JSON key called user:contacts:1 which will look like:

{
    "in": "user:1",
    "out": "user:2",
    "company": "Redis"
}

And a set called user:1:contacts which will contain user:contacts:1

To fetch all the relations you can run the following

FCALL JSONGR 1 user:1 contacts

Which for this example will return

[
    {
        "name": "Leibale"
    }
]

You can note that metadata is not appended to the returning object, this is because metadata only exists so you can leverage RediSearch functionality on relations.

An example of this would be:

FT.CREATE exampleIdx ON JSON PREFIX 1 user:contacts: SCHEMA $.in AS in TAG $.out AS out TAG $.company AS company TEXT

And then we want to search all contacts of user:1 that work on redis

FT.SEARCH exampleIdx '@in:{user\:1} @company:redis' RETURN 1 out

Nekdis example

NOTE
This is a work in progress and some things could change

I will translate the cli example into nekdis syntax so they will have exactly the same functionality.

const userSchema = client.schema({
    name: "string",
    contacts: {
        type: "relation",
        index: true,
        // Meta could be another schema just like the properties in an object type
        meta: {
            company: "text"
        }
    }
});

const userModel = client.model("User", userSchema);

// Im not using "createAndSave" so i can have the document without having to create and then fetch
const user1 = userModel.create({
    name: "DidaS"
});

const user2 = userModel.create({
    name: "Leibale"
});

await userModel.save(user1);
await userModel.save(user2);

// Im passing the record id just to exemplify that you can use either the document or an id/key
await userModel.relate(user1).to(user2.$record_id).as("contacts").with({ company: "Redis"}).exec();

await userModel.get(user1.$record_id, { withRelations: true });
/*
JSONDocument {
    name: "DidaS"
    contacts: [
        {
            name: "Leibale"
        }
    ]
}
*/

await userModel.get(user1.$record_id, { 
    withRelations: true,
    relationsConstrain: {
        contacts: (r) => {
            r.where("company").eq("redis");
        }
    }
})

Final notes

This is still a work in progress so if you have any ideas on how to improve please let me know.

[Didas-git]

Here i will leave a screenshot of how the syntax is currently.
I will be working more on this next week to bring relation functionality to other methods and i will expand the search capabilities with this as well