How to get IPFS addresses for resources?

[joaosantos15]

Hey, @diasdavid, I believe this might be a question for you. As we'd discussed on our Nov 6 meeting, one of the next implementation goals was to store all the application dat on IPFS, so that anyone could run the project without the need for a centralised storage server to be running anywhere.
To better explain my question let's look at this scenario:

Alice used Clickbait buster to generate a claim about article A
Bob uses Clickbait to learn about the claims that have been made about article A

Being that the IPFS address of the claim will depend on the claim's content, how can Bob's instance of the application know what IPFS files to retrieve without the existence of a central registry?

[daviddias]

You need to have some kind of indexing system that matches URLs to URI (Content Addrs)

[joaosantos15]

Yes, I searched quite a lot for that this week. I started by looking at IPFS' PubSub but I found two problems with that:

1. When I was running the system with two nodes in different machines (my laptop and a server at IST) it took a long time for the flooded messages to start going from one node to the other, I was using {discovery=true}.
2. I couldn't get the the history of the messages, which I needed to know what was the last message sent about a given topic.

More on problem 2...

The alternative for this to work would be to have put the publisher constantly publishing the same message every x seconds so when the subscriber came online it could learn what the latest message was for a topic. Also, each node is at the same time a Pub and a Sub so that periodic Pub message would have to be constantly updated to take into account the updates made by other nodes. This would lead to time stamping problems.
I'm not saying it couldn't be done, it most likely could, but I just didn't see a way to get it done before the deadline.

Taking that into account I started looking at Orbit, more specifically, Orbit-DB. I made some experiments with it but only got to work when using the ipfs-daemon module. When I switched to the js-ipfs-api it started throwing errors all over the place, even with a super simple use case. Then I saw that Orbit hadn't been updated for a couple months so maybe that's why the errors were coming up.

Then I followed your tip on Y.js and the y-ipfs-connector. I looked at it but it only works with js-ipfs, which doesn't have discovery, so that was a no go as well.

Finally I settled on a centralized registry. I'm running a server that exposes an API to translate news article id's into the corresponding IPFS links with the claims made about those articles.

All the claims are stored on IPFS. The central server is only used to translate addresses, that's why I called it blockchain-api, as it is mimicking the behaviour we will ultimately have from the smart-contract. Here is the PR. 😄
@diasdavid

Also, @nuno-santos , creating a key-value store on top of IPFS could be an interesting research topic ⚡️, and it's inline with the objectives for the Clickbait thing. There's a discussion about this on the IPFS forum.