Improve UTxOs Retrieval in Koios

[edridudi]

In order for Koios to also be used in Lucid Library as another Provider, there are some gaps that should be closed:

UTxO Model

UTxO model should have all the relevant fields as follows:

it is also required to add the Bech32 Payment Address in the UTxO model as a field

Endpoints

• UTxOs from payment credentials Endpoint should support the entire UTxO model mentioned above.
• new Endpoint(s) should be created or modified to Receive UTxOs from Credentials type Payment Key Hash or Stake Key Hash or Script Hash.

Lucid's TypeScript model for Credential:

export type Credential = {
  type: "Key" | "Script";
  hash: KeyHash | ScriptHash;
};

/** Hex */
export type KeyHash = string | PaymentKeyHash | StakeKeyHash;

/** Hex */
export type PaymentKeyHash = string;

/** Hex */
export type StakeKeyHash = string;

/** Hex */
export type ScriptHash = string;

[rdlrt]

It's a bit tough one, and needs to be thought through very carefully.

1. I am not sure I am a fan of breaking all the consumer libs for duplication/interpretation between _info and _utxos endpoints.
   By design, _utxo endpoints are meant to provide list of utxos alongwith basic information (to add info to these fields, the destination should be same as account_utxos). Further information for those utxos go via _info endpoints (at the most, we could potentially replicate the model into tx_utxos - but even then it would lose the entire point of having that lightweight endpoint). I do agree - however that consistency for output between address, account and credential has become lagged, and we will ensure to improve and align them [likely towards account_utxos].

2. credential_utxos is a list of utxos, usable to drill down further. Doing a direct link for utxos to actual tx_info will terribly risk the performance of instance provider for that object as well as risk the system to bad user queries (imagine someone initiating this call against a popular nft marketplace). 1 important point of distinguishing between providers is koios allows bulk query of objects via a single call (so a consumer could end up doing 2 query for a set of addresses and fetch details across all objects, as against an alternate which might loop through each and get user to do 50 calls).

3. The only endpoint that could be aligned IMO would be tx_utxos, but even then - it would lose out the advantage of having a seperate endpoint than tx_info in the first place. As it would become predominantly same as tx_info (minus certificates)

[Scitz0]

This is a tough one and as Priyank mentioned has to be thought through carefully.

1. Fully agree that the current inconsistency between utxo endpoints should be addressed.
2. tx_info does indeed give you all information, but tailored for querying for a complete tx, not specific utxo(s). I like our current approach with splitting it into two types, lightweight list of x and then info for x, that can then be queried in bulk like mentioned.
3. We have /credential_utxos for fetching all relevant utxo's connected to a payment credential. What we lack is a utxo_info endpoint like tx_info that only look at those specific utxo's and not the whole transaction like tx_info.
4. I propose that all utxo endpoints revert to only return the very basics like credential_utxos and that we then create a new utxo_info endpoint that return relevant data sort of like suggested. So de-spam account_utxos and tx_utxos. Create address_utxos (seems to be missing) and utxo_info.

[edridudi]

Few points:

1. There are no gaps for tx_utxos except the one we know of from Update txs and utxos endpoints #186
   I really don't see how tx_utxos becomes closer to tx_info as tx_info gives a lot more data. tx_utxos is okay as it is right now from the current model written in docs.
2. We miss an endpoint for getting utxos by script hash. (please correct me if I'm wrong)
3. I don't think it is best to have 2 different kinds of endpoints for utxos (basic and full) as it is the most common object clients and devs are looking to retrieve. it is best to have all data possible (utxo model I showed will suffice) and not lots of calls and focus on clients' cost-efficiency.
   Utxos Are commonly used on client-side to be filtered by assets, scripts, or datum hashes. it will be very slow to drill down each utxo and try to filter out those for hundreds of utxos per address.
4. utxos model have to be the same for all endpoints.
5. I understand that it is a tough one. I'm only here to show the gaps :)

for having utxos by all kind of credential types, BF follows CIP-0005 and uses prefixes and conversions between types (Maybe it can help).

as long as we can have those that would work best both for Koios and for Lucid Users.

Please also take into account the speed and amount of calls for a Library in order to achieve the best results.

[rdlrt]

1. tx_utxos is okay as it is right now from the current model written in docs

Like others it was supposed to be a list of utxos (but with input & outputs), the issue is with specs which reuses tx_info's output field, instead of having one for utxos, a utxo_info sounds more apt to the job (but we can prolly leave tx_utxos as alias for it for compatiblity) - we can add that to next release as desired, since adding more columns shouldnt break existing endpoints expecting specific schema.

2. We miss an endpoint for getting utxos by script hash. (please correct me if I'm wrong)

Agreed, can add it to the list

3. It is best to have all data possible (utxo model I showed will suffice) and not lots of calls and focus on clients' cost-efficiency.

I addressed this specifically in my first reply. Unlike many other providers, Koios supports bulk queries - which means not only can you query all utxos for an address, but also all utxos across multiple addresses sent in bulk to fetch info against. So what you're looking at is perhaps 1 call for list of utxos (across multiple address or accounts or creds) and second call for all utxos involved. As against that you'd end up querying each utxo seperately. The performance impact is exactly why it makes sense to seperate the two - and why _utxos have always been about list of utxos. Duplicating same object across all endpoints will risk an instance provider being badly abused, not a path we'd want to take - especially as chain history grows further.

4. utxos model have to be the same for all endpoints.

agreed - the utxos model would indeed by same, and simply be at tx_info and utxo_info endpoints, while the endpoints containing list of utxos would be aligned too, to represent same information.

Please also take into account the speed and amount of calls for a Library in order to achieve the best results.

If you're leveraging bulk query option, you'd likely be ending up with even lesser calls and better performance when compared (wallets for instance would seldom look at single address for inputs, they'd look at collection of addresses) 🙂

[edridudi]

Will it be possible to add filters to the basic utxos endpoints by different fields?
like fetching specific utxos filtered by a specific asset/datum hash/script hash etc. ?

We don't need them to retrieve the fields themselves. We then can make another query to retrieve the entire model from the info endpoints.

If it will be possible then I think it's good.

If I'm not mistaken, currently we can't use horizontal filtering for POST endpoints hence it would be very much needed to save some iteration time at client side.

[rdlrt]

fetching specific utxos filtered by a specific asset/datum hash/script hash

You'd already be specifying asset/datum/script on the input to list UTxOs by - to then be able to query utxo_info (or tx_utxos), so yes - that should be possible

If I'm not mistaken, currently we can't use horizontal filtering for POST endpoints

You can surely do horizontal as well as verticle filtering for rows on POST endpoints as well, to only return rows (just not sub-array items, for which - if filtered you receive entire row of 1 tx , instead of multiple txs) that match the condition. For example to filter txs by specific address:

curl -X GET "https://api.koios.rest/api/v0/account_utxos?_stake_address=stake1uxpdrerp9wrxunfh6ukyv5267j70fzxgw0fr3z8zeac5vyqhf9jhy&address=eq.addr1qyh6fakqvqfsfs0w9qgchg2hrh77e927vr6wt5su
twnw5huz68jxz2uxdexn04evgeg44a9u7jyvsu7j8zyw9nm3gcgq8guve9&select=tx_hash,tx_index,address,value"  -H "accept: application/json"
# [{"tx_hash":"7668b5d5dfad8dbb6160bb117d7cbcda8d04cd42575c32701236e62093cf710b","tx_index":14,"address":"addr1qyh6fakqvqfsfs0w9qgchg2hrh77e927vr6wt5sutwnw5huz68jxz2uxdexn04evgeg44a9u7jyvsu7j8zyw9nm3gcgq8guve9","value":"1518517"},
# {"tx_hash":"492d07303dff3feca78464b34231f4c5c3b3940bab201c3b05363f96af45a045","tx_index":20,"address":"addr1qyh6fakqvqfsfs0w9qgchg2hrh77e927vr6wt5sutwnw5huz68jxz2uxdexn04evgeg44a9u7jyvsu7j8zyw9nm3gcgq8guve9","value":"1679147"},
# {"tx_hash":"0e0731aa17c9373f6934d65222abc512a1f738894464a3526975987cea8e7697","tx_index":32,"address":"addr1qyh6fakqvqfsfs0w9qgchg2hrh77e927vr6wt5sutwnw5huz68jxz2uxdexn04evgeg44a9u7jyvsu7j8zyw9nm3gcgq8guve9","value":"1636209"}]

[rdlrt]

Can be tracked via #186 [unfortunately, since it becomes a breaking change, will have to align with a wider breaking change upcoming with dbsync which will tremendously help current UTxOs retrieval as it should no longer require a join]