Datagen_Project

[Lord-Nymphis]

Project Abstract

B-Datagray’s Datagen project concerns the development of a decentralized infrastructure for CPU/GPU cloud computing, in chain, through different blockchain components.

The Datagen infrastructure requires the creation of a Substrate-based blockchain with some key features: low latency time, high efficiency (compatibly with the decentralized nature of the network), high degree of decentralization (higher than a traditional PoS would allow) of the physical hardware providing the cloud computing (therefore requiring a customized consensus protocol) and in-chain (or near-in-chain) computation (no off-chain based solutions).

The Datagen infrastructure primarily want to serve: privacy friendly search engines and browsers, decentralized Web3 games and other ones (for example decentralizing the hardware layer of PoS blockchains, typically hosted on AWS and similar, etc…).

We will implement only a PoC with this grant.

The goal is to achive a fully functional mechanism for the random selection of the nodes in the fast blockchian and smooth communication between the two blockchains.

For which grant level are you applying?

• Level 1: Up to $10,000, 2 approvals
• Level 2: Up to $50,000, 3 approvals
• Level 3: Unlimited, 5 approvals (for > $100k Web3 Foundation Council approval)

Application Checklist

• The application template has been copied, renamed ( project_name.md) and updated.
• I have read and understood the FAQs, application guidelines and announcement guidelines.
• A BTC, Ethereum (USDT/USDC/DAI) or Polkadot/Kusama (aUSD) address for the payment of the milestones is provided inside the application.
• I have read and acknowledge the terms and conditions.
• The software delivered for this grant will be released under an open-source license specified in the application.
• The initial PR contains only one commit (squash and force-push if needed).
• The grant will only be announced once the first milestone has been accepted.
• I prefer the discussion of this application to be in a private Element/Matrix channel. My username is: @_______

How Did You Hear About our grants program?

• Social Media
• Hackathon
• Personal Recommendation
• Substrate Builders Program
• Investor/VC
• Online Search
• Other: _______

[CLAassistant]

All committers have signed the CLA.

[Noc2]

Thanks for the application. Milestone 1 and 2 currently seem relatively expensive to me. I guess you are simply reusing the babe implementation for this one and not implementing your own randomness implementation. Is this correct? At the moment it looks like you want to have 10k per function that you are going to implement. Could you add additional functionality or reduce the price? Regarding the Web Dapp, we usually ask teams to provide some initial designs or mock-ups for front-end focused deliveries. Could you add this? Unless you are simply reusing the substrate front-end template for this.

[Lord-Nymphis]

Well'll revert back ASAP in the next few days.
We'll have an answer also for milestone 1 and 2 (clarifications about the implementation of randomness traits and further implementations for milestone 2 VS reducing the milestone grant amount)
Regarding the mockup we'll speak with our designers. You mean a simple FIGMA or similar, no front end code, right?

[viac92]

Hi @Noc2 Datagen developer here!

For the Milestone 1 we want to develop a pallet to manage the random selection of the nodes that have to check if the computational work (from other random selected nodes) has been done correctly.

So I think we need a custom pallet to manage the logic for this mechanism, maybe we could start with the babe pallet and modify it.

Let me know if it makes sense to you.

[Noc2]

Thanks for the quick reply. Yes, you need to develop your own pallet for this. You should probably take a look at
https://docs.substrate.io/how-to-guides/v3/pallet-design/randomness/
https://github.com/paritytech/substrate/blob/master/frame/babe/src/randomness.rs

I was mostly referring to the fact that the pallet at the moment seems to have only a single function according to the milestone, correct? And therefore it seems relatively expensive, so I would recommend either adding additional details to the delivery or reduce the price a little bit.

[Noc2]

I’m closing the application due to inactivity. Let me know if I should reopen it.

[Lord-Nymphis]

@Noc2 , you asked us to create a mock-up. It required some days. We were also updating the Milestones as requested. We were waiting to have the mock-up to update all in one shot. OF COURSE you should reopen it.

[Noc2]

Got it. I wasn’t sure you are working on it, since you didn’t reply for 18 days. I will reopen the PR.

[Lord-Nymphis]

Thank you very much @Noc2 , yea, sorry for not answering, we directly went back to work ... for us was implied that we were going to answer once we had updated the grant proposal, taking into account of all your suggestions. Sorry for not notifying that to you. :)

[viac92]

Hi @Noc2!

For milestones 1 and 2 we add more functionalities and explain them more in depth.
For milestone 3 we add a mock-up of the dapp.

If there are more things to change let us know :)

[Noc2]

Thanks for the update. @viac92 could you also sign the CLA? All contributors need to sign it. Apart from this, I’m happy to support the project.

[viac92]

Oh sure, I sign it now btw thank you @Noc2 for the approval!

[Lord-Nymphis]

Hi @Noc2, one question, when we submitted the grant, it was a lev.2 (as you can see in the Project Abstract above). I think that rules were changed while our proposal was pending. I asked Monday for some clarification from @SBalaguer and he seems to confirm that you changed the rules in the meantime.

Is it possible to have some clarification even here on the grant proposal?
Because seems to me that the original proposal (according to the rules of the time in which we submitted it) is divergent from the final result (deriving from the new rules).

Many Thanks.

[Noc2]

Yes, correct. We changed the levels, but everyone who applied before the github commit, still needs the same number of approvals as before. So in your case 3 approvals.

[Lord-Nymphis]

Thank you for the clarification @Noc2 , glad to hear it.

So we don't have to worry about the fact that in the right-upper part of the screen I see: "At least 5 approving reviews are required to merge this pull request." ?

As so, once we have three approvals the proposal is passed?

Do you want us to remember you to do that manually if we have the three approvals (since it is different from the general status of new applications)?

[Noc2]

Yes, you don’t need to worry about the 5 approvals. This is just because the github branch protection works like this. We will merge the application after three approvals.

[Lord-Nymphis]

Grazie mille.

[semuelle]

Thanks for the application, @Lord-Nymphis. Could you please replace the screenshots in the specifications under M3 with a list of functions or functionalities, so it's less ambiguous what is being delivered?

[Lord-Nymphis]

Hi @semuelle . I think that would possible to add a brief description / comment of the functionalities below each screenshot.
( The screenshots represent the mock-up requested in the comments above by @Noc2 ).

[Lord-Nymphis]

@semuelle in a nutshell is not so different from a scan. Once you have installed the local blockchains (both heavy and fast) and once they are both producing blocks, will be possible to search for specific blocks or nodes and see (in the specific case of the fast blockchain), accordingly to the consensus protocol, how many nodes have been checked, when, and by what other node. Some info is visible in the home, with sheets that are scrolling while new blocks are generated, some other info (about any specific block, at any height) can be searched for example simply selecting the hight of the block (after inputting the filter for the blockchain "fast" or "heavy").

We'll write clear comment in the milestone M3 sheet.

[keeganquigley]

Hi @Lord-Nymphis are you still planning on submitting a delivery soon?

[Lord-Nymphis]

Hi @keeganquigley , we are planning to deliver soon, by mid December. The bridge setup is completed, we are finishing to complete the RPC methods.

[keeganquigley]

Hi @Lord-Nymphis any updates you can provide?

[Lord-Nymphis]

Ciao @keeganquigley , someone in the Parity team had the wonderful idea of discontinuing ( paritytech/parity-bridges-common#2663 (review) ) the Rialto Millau bridge repo, over which we had built the core functionalities of our solochain to parachain bridge. Thanks to that, we throw away a couple hundred hours of development and we had to refactor the entirety of our core functionalities (shifting to Westend - Rococo), which we are finishing to do.
If we didn't had to refactor we would have delivered the overall milestone by mid to second half of December.

Most of the refactoring had been done, but we are still getting some error when running the setup. We are working to fix everything.

[keeganquigley]

Thanks for the update @Lord-Nymphis I appreciate you informing us on the situation. I will take note of this and consult with the team, perhaps we should create a new RFP to find a team who is willing to take over and maintain these bridges. LMK if that is something your team would be interested in.

[Lord-Nymphis]

Thanks. BTW our dev heard one of Parity devs that worked on the Rialto repo and told us it would have been outdated with the new SDK. Now we already refactored almost everything.

[keeganquigley]

Hi @Lord-Nymphis how did the refactoring go? Can you provide any updates?

[Lord-Nymphis]

Hi @keeganquigley , thank you for asking!

The part concerning the heavy chain's refactoring is done and the blockchain is compiling, the part concerning the fast chain is still missing some features for some dependencies concerning the runtime, but they could potentially be fixed by the end of this weekend.

After that the refactoring is done, with of course the bridge implementation to be done after that.

[keeganquigley]

Hi @Lord-Nymphis any updates? I see that the two grant repos mentioned above haven't had any commits in around 6 months.

[Lord-Nymphis]

Hi @keeganquigley . Yep.

I think you have been looking into some old branches. The last update is two days ago. Here the most updated code branch https://github.com/Datagen-Project/Datagen-Substrate-Grant/tree/ln-update-testing-bridge

[keeganquigley]

Thanks for the update @Lord-Nymphis does this mean a delivery will be submitted soon?

[keeganquigley]

pinging @Lord-Nymphis

[Lord-Nymphis]

Hi @keeganquigley

Yes. Sorry for not answering, the message slipped through.
Yes, the answer is yes. As Soon As Possible.

[keeganquigley]

Hi @Lord-Nymphis seeing as the delivery date keeps getting pushed, and there has been no delivery for a year and half, it's probably best to go ahead and close it at this point, and it can potentially be re-opened in the future should you wish to submit a delivery. This helps us for administrative & budgeting purposes. Let me know if this works, otherwise we'll close it in a few days. Thanks!

[Lord-Nymphis]

Hi @keeganquigley , as previously explained, the company had budget constraints during the bear market and we had to change a few devs. There had been also the matter of Parity discontinuing the support for the Rialto-Millau bridge repositories in November 2023 for the new SDK, meaning that we had to do everything from scratch starting from that date using the Rococo-Westend bridges.

Happy to help you with your administrative budgeting as long as this doesn't require going again through the approval procedure from the grant commission.
We are actively working on the milestone and we have also already committed the payment to the dev working on it.

Let me know the best way to go ahead with this.

[keeganquigley]

Thanks @Lord-Nymphis for the update. Is this dev already listed on the application as part of the team? Usually for core contributors we ask for any personnel changes to be amended on the original document.

[Lord-Nymphis]

@keeganquigley No, he is not. The dev is the one that made the most recent contributions in this repo's branch https://github.com/Datagen-Project/Datagen-Substrate-Grant/tree/ln-update-testing-bridge

Now he is on leave for personal reasons. He should come back in 8 days.

What do you think, could we directly update the application document right before pushing the code for the milestone?

We also had another dev making some small contributions in December and we don't exclude to have others helping the current one with other contributions; to avoid updating the document multiple times and to have something more complete in terms of contributors, we could update right before submitting the Milestone code.

[keeganquigley]

Sounds good, thanks @Lord-Nymphis let me consult with the team and will get back to you!

[keeganquigley]

Hi @Lord-Nymphis thanks for your patience while we discussed internally; unfortunately the committee has unanimously decided to cancel the grant, mainly for the following reasons:

• No deliveries in 1.5 years
• M3 would likely be further delayed
• Many delays with not much progress other than minor fixes/updates
• Undocumented changes in scope
• Many personnel changes over the course of the project
• Changes would require another amendment which is not likely to be approved at this point

That being said, we thank you for all the time & effort you've put into this project and we wish you the best of luck in finding funding moving forward.

[Lord-Nymphis]

Hi @keeganquigley

We would like to know the procedure to reapply for the milestone delivery once delivered. Thanks

[keeganquigley]

Sure thing @Lord-Nymphis yes it can always potentially be re-opened in the future should you have a delivery ready for submission. The procedure would essentially be the same as the regular application process, but in your case you would submit a PR to amend the contract to remove the terminated status. I hope that helps!

[Lord-Nymphis]

@keeganquigley so basically it's like the milestone it's in standby?

[Lord-Nymphis]

We would need to submit only the delivery or the application?

[keeganquigley]

Yes, in this scenario you would need to submit two PRs, one for the application and one for the delivery. However it probably wouldn't be necessary to submit another application entirely, as the current one will stay in our files, but with the added status. So you would just submit a PR amendment to change the status.

[Lord-Nymphis]

So I will basically submit a pull request where I submit the Milestone (as per deliverable of the closed grant) and the request to reopen the Grant and approve the Milestone, all in the sam PR?

[keeganquigley]

@Lord-Nymphis I'm not sure what you mean here, since deliveries are submitted to the milestone delivery repo, while applications are submitted to the grants repo. But seeing as the grant is now closed, we aren't currently able to accept any future deliveries. Therefore you would need to first submit a PR to request that it be re-opened. Please note, however, that the committee will likely take this termination into account as part of any future decisions.

[Lord-Nymphis]

@keeganquigley Yes, it's clear we don't have to submit the delivery in the milestone repo, but that the right procedure is to submit the application to the grant repo.

What I meant (ofc I have clear that the procedure requires the voting of the commission) is that, exactly because there had been delays with the delivery of the second milestone of the closed grant, it's pretty obvious that if we submit the application proposal without any viable code it would be rejected (I can't read in the committee's members' minds, ofc, but I presume it would be only logical), as so we would like to submit the application to the grant repo (requesting to re-open) together with the complete delivery of the second milestone of the previously closed grant, in good faith (understanding that then 1. the committee would have to update the grant status 2. only subsequently to the eventual acceptance of the status the milestone delivery could be approved).

Please let me know if this logic resonates

[Lord-Nymphis]

@keeganquigley

Regarding #980 (comment)

While we acknowledge that you are 100% right about us being late and we take full responsibility on this specific matter, and I understand that this point alone is sufficient for you to change the grant status, I would also like to underline that there has been NO undocumented changes in scope in the grant deliveries. This is a factual error, which I would like to challenge (if I am mistaken, I would like to know in what regard).

[Lord-Nymphis]

Also regarding this specific comment in the post #980 (comment)

• Many delays with not much progress other than minor fixes/updates

I would like to underline that this is another factual error, given that our most recent code branch https://github.com/Datagen-Project/Datagen-Substrate-Grant/tree/ln-update-testing-bridge/heavy_blockchain/node/src
has 184 code commits and ≈5000 lines of code, EXCLUDING dependencies.

It's clear that, unfortunately, we had deeply underestimated the task when submitting the grant proposal.
We also underlined above in the conversation that we thought that the original proposal was deeply below the budget, but continued to implement it nonetheless, considering that the original mistake was ours (but, as anyone who does development knows, sometimes you don't know how much work development will require until you really start doing it, especially for something as experimental as we are doing).
The market cost of a senior substrate developer is typically between 50 and 100 USD$. We had someone woking on this part time (20h week) for at least 1 year = 1000h of development, which means that the real cost to develop ONLY Milestone2 is between 50K and 100K USD.
This is math, it's not my opinion.
We never asked for the full amount and we took the expense on ourselves, yet apparently the Web3 Foundation thinks that it's possible to develop under budget AND in time AND without changing any personnel AND that 184 commits + 5000 lines of code are "minor-fixes / updates".
About 50% of our code commits had been unnecessary, given that we spent 6 months working with the Rialto-Millau Repo's, just to have the support for those suddenly cancelled by Parity in November (event totally outside of our control), and 0 help and 0 clarifications from Parity when asked how projects already working with those repos' had to behave. Meaning that we had to refactor everything and that of the 13 months of delay at least 5 or 6 are because of this issue.

I understand that you have a variety of problems with the restructuring of Parity and the W3F, but, at the same time, you can't expect small project to do the impossible and take the scorn.

I understand that you had already voted on the matter and that clearly for the W3F moving 12K from one row of the balance sheet to another is more important that retaining projects and having development for Substrate, so I don't expect you to reconsider.

I also understand that this isn't your fault, but that this is the policy of the W3F.

I only hope that these considerations will be taken into account when we will submit the PR (together with the code) in good faith.

Have a good day