Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add IPFS Utilities application #1187

Merged
merged 6 commits into from
Nov 1, 2022

Conversation

dobschal
Copy link
Contributor

Project Abstract

This application is about the utilities of Substrate within IPFS.
The existing offchain::ipfs will be extended and adjusted to have a proper handling of IPFS CIDs and a redirect feature to retrieve files from IPFS.

This is a follow up to an already closed application PR: #1127

Grant level

  • Level 1: Up to $10,000, 2 approvals
  • Level 2: Up to $30,000, 3 approvals
  • Level 3: Unlimited, 5 approvals (for >$100k: Web3 Foundation Council approval)

Application Checklist

  • The application template has been copied and aptly renamed (project_name.md).
  • I have read the application guidelines.
  • A BTC, Ethereum (USDT/USDC/DAI) or Polkadot/Kusama (aUSD) address for the payment of the milestones is provided inside the application.
  • The software delivered for this grant will be released under an open-source license specified in the application.
  • [] The initial PR contains only one commit (squash and force-push if needed).
  • The grant will only be announced once the first milestone has been accepted (see the announcement guidelines).
  • I prefer the discussion of this application to take place in a private Element/Matrix channel. My username is: @_______:matrix.org (change the homeserver if you use a different one)

@dobschal dobschal mentioned this pull request Sep 27, 2022
10 tasks
Copy link
Collaborator

@Noc2 Noc2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updated grant application. I’m still willing to go ahead with it and will share it again with the rest of the committee.

@Noc2 Noc2 self-assigned this Sep 28, 2022
@Noc2 Noc2 added the ready for review The project is ready to be reviewed by the committee members. label Sep 28, 2022
@SBalaguer
Copy link

Hi @dobschal thanks for the updated proposal! It does look very interesting to me.

Would it be possible for you though to briefly explain how this proposal is improving current implementations like place_storage_order from Crust?

@semuelle
Copy link
Member

Hi @dobschal. Thank you for the application. Are you building this out of an in-house need? Since we now have multiple unmaintained forks of offchain_ipfs, it would be good to know that it won't be dropped again right away.

@semuelle
Copy link
Member

ping @dobschal

@bbros-dev
Copy link

bbros-dev commented Oct 19, 2022

@Noc2 this also seems to be relevant prior art/grant:

@bbros-dev
Copy link

bbros-dev commented Oct 19, 2022

@dobschal @SBalaguer @semuelle @laboon @Noc2 : Our understanding of the state of play is in the comment after this one.

With that background..... We wonder if something like the following isn't needed?

Objective:

including the Rust implementation IPFS in the native Substrate runtime

Milestones:

  1. An example ...

pallet, which is essentially a Rust module that complies with the requirements to be included within a substrate runtime.

  1. Merge into the native Substrate runtime Rust IPFS crate scaffolding sufficient for native runtime support for:

Or are all these substrate forks un-necessary, and these teams have missed a trick?

Or the Rust implementation of IPFS won't land in the native Substrate runtime for the foreseeable future/ever, and the Substrate forking can be expected to continue?

If the former, can someone say how these use cases should be implemented without using Substrate forks? can some detail be added to the IPFS section here

If the later, can that fork-your-own-substrate detail be added to the IPFS section here

@bbros-dev
Copy link

bbros-dev commented Oct 19, 2022

lacked points to post this here: https://substrate.stackexchange.com/questions/5406/storing-data-offchain-to-ipfs-pallet-vs-chain-extension

Subtrate runtime with native IPFS support

Objective:

including the Rust implementation IPFS in the native Substrate runtime

Benefit:

An example ...

pallet, which is essentially a Rust module that complies with the requirements to be included within a substrate runtime.

Status: 19 Oct 2022

Activity in last 30 days:

Status Summary

  • Upstream changes to Substrate are required for native Substrate runtime support of IPFS (things have been tried).
  • Upstream is not resistant
  • Upstream changes won't be easy/quick
  • Web3 Foundation have made two grants related/using IPFS, but not focused on getting IPFS in the native Substrate runtime
  • The Iris grant project completed
  • The offchain_ipfs project review/merge of contributions stalled due to lack of resources
  • Substrate forks proliferate....

Status Conclusion

Until this resolves, however that is, you'll need to use a substrate fork closest to your use case:

Or start and maintain your own fork....

@bbros-dev
Copy link

Hmmmm.... rs-ipfs is in a state of flux, possibly being archived, with iroh being a new candidate. Performance is impressive (presentation) but it is early days.

Substrate forking seems unavoidable for the foreseeable future, say 12+ months

@dobschal
Copy link
Contributor Author

dobschal commented Oct 21, 2022

Hi @dobschal. Thank you for the application. Are you building this out of an in-house need? Since we now have multiple unmaintained forks of offchain_ipfs, it would be good to know that it won't be dropped again right away.

Hey @semuelle
We faced this problem in our NiftyMarket project. We created a workaround but with this grant we want to solve it also for others. Once this implementation is available we plan to integrate this in our NiftyMarket blockchain implementation. Updates on this pallet during our development would be then contributed back to the community.

@Noc2 Noc2 requested a review from semuelle October 24, 2022 09:49
Copy link
Collaborator

@takahser takahser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks interesting and could be useful to the community. However, I do think that the price is currently too high with 16.5k/month/FTE. In your past grant the rate was a bit more than 1/3 of that price: 5.5k/month/FTE. The other teams that worked on ipfs-related grants that you quoted also demanded much less for similar scopes.

Provide a pallet implementation that allows clients to add files to IPFS and store the related CID on chain with one RPC. For instance when minting an NFT with a given artwork file.

In addition to that, the Substrate blockchain can offer an RPC API to read the file from IPFS too. Therefore a redirect feature needs to be implemented based on the PRG mechanism.
As it takes a few moments until an added file is available on the IPFS network thru public gateways, the local IPFS node can used to retrieve the file.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
As it takes a few moments until an added file is available on the IPFS network thru public gateways, the local IPFS node can used to retrieve the file.
As it takes a few moments until an added file is available on the IPFS network thru public gateways, the local IPFS node can be used to retrieve the file.

### Ecosystem Fit

The implementation helps people using IPFS with Substrate, this is a common scenario.
Especially for NFT products this solution is helpful to avoid external middleware implemetations taking action. Having the whole logic for minting NFTs with related content inside Substrate makes it easier to maintain and more secure.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Especially for NFT products this solution is helpful to avoid external middleware implemetations taking action. Having the whole logic for minting NFTs with related content inside Substrate makes it easier to maintain and more secure.
Especially for NFT products, this solution is helpful to avoid external middleware implementations taking action. Having the whole logic for minting NFTs with related content inside Substrate makes it easier to maintain and more secure.

Copy link
Member

@semuelle semuelle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Happy to proceed.

@Noc2 Noc2 requested a review from takahser October 25, 2022 15:25
Copy link
Contributor

@alxs alxs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bbros-dev thanks for the in-depth summary. Regarding the wiki, feel free to open an issue in its repo.

@dobschal thank you for the updated application. I agree with @takahser regarding the rate, would be great if you could address this. But more importantly, as @bbros-dev pointed out, rust-upfs has now been archived, so it probably doesn't make a lot of sense to build a new integration with Substrate on top of it. Even with iroh being in its early days, I'd be more willing to support a project based on it. Such a project may be likely to break, but given the current state of things, relying on rust-ipfs would make it certain to break. Curious to hear your thoughts.

Copy link
Collaborator

@takahser takahser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still think it's too expensive, as already indicated and I haven't heard any feedback on that yet.

@dobschal
Copy link
Contributor Author

Hey, thanks for the feedback.

As there are too many unsolved doubts we are considering other options.

Thanks!

@dobschal dobschal closed this Oct 27, 2022
@dobschal
Copy link
Contributor Author

dobschal commented Nov 1, 2022

We would like to reopen this grant. There was a misunderstanding on our side and we would be very happy to continue with this grant. If you have any questions, please feel free to ask.

@dobschal dobschal reopened this Nov 1, 2022
@Noc2
Copy link
Collaborator

Noc2 commented Nov 1, 2022

I shared your application again with the team.

@Noc2 Noc2 merged commit d3beeb9 into w3f:master Nov 1, 2022
@github-actions
Copy link
Contributor

github-actions bot commented Nov 1, 2022

Congratulations and welcome to the Web3 Foundation Grants Program! Please refer to our Milestone Delivery repository for instructions on how to submit milestones and invoices, our FAQ for frequently asked questions and the support section of our README for more ways to find answers to your questions.

Before you start, take a moment to read through our announcement guidelines for all communications related to the grant or make them known to the right person in your organisation. In particular, please don't announce the grant publicly before at least the first milestone of your project has been approved. At that point or shortly before, you can get in touch with us at [email protected] and we'll be happy to collaborate on an announcement about the work you’re doing.

Lastly, please remember to let us know in case you run into any delays or deviate from the deliverables in your application. You can either leave a comment here or directly request to amend your application via PR. We wish you luck with your project! 🚀

@dobschal
Copy link
Contributor Author

dobschal commented Nov 3, 2022

Thanks for the approvals 🥳 Will check the next steps and are super happy to jump on that!

@dominikdem
Copy link

Hello W3F Team; we started the implementation phase.

@keeganquigley
Copy link
Contributor

Thanks @dominikdem how is M1 coming along?

@dominikdem
Copy link

Hi @keeganquigley, thanks for reaching out. After some challenges, we have made good progress and are about to finish M1 by the end of next week.

@dobschal
Copy link
Contributor Author

Hey 👋
we created the PR for the milestone 1 delivery.
w3f/Grant-Milestone-Delivery#743

@keeganquigley
Copy link
Contributor

@dobschal awesome thanks for the submission! Someone will look at it shortly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready for review The project is ready to be reviewed by the committee members.
Projects
None yet
Development

Successfully merging this pull request may close these issues.