dangerousfreedom - wallet development 2

Apart from @UkoeHB and @jberman right now he is one of only 3 people who work on the Seraphis wallet with a big time budget, and who already know their way around the Seraphis library code, so he is a major asset for the project.

I've been doing little to no coordination since the 27th but will resume on the 9th so apologies for being left in the dark here. From my end, your regular updates during various work group meetings (no wallet left behind / MRL) are great to see.

I think it would now be advantageous/speed things up if we can get feedback from koe with how he felt the previous ccs went / the specific tasks/goals of this one (seeing as he is the creator of Seraphis / one of the main benefactors of your work). and again feedback from those involved in the 'Seraphis migration'.

I have appreciated dangerousfreedom's effort toward learning about Monero, and enthusiasm about the protocol. Below I assess his Monero work up to this point.

Work record

Here is a summary of dangerousfreedom's work record:

CCS

Inflation checker (480 hr @ 18EUR/hr): resulted in the very interesting Python-based monero inflation site. Here is the repo for that project.
Seraphis development (450 hr @ 40EUR/hr): resulted in A) this PR to the seraphis library based on this research issue (with a writeup here), B) 3k lines of new code in this branch.

Personal projects

monero inflation website
investigation of serialization issue in Borromean signatures
Seraphis knowledge proofs writeup

Seraphis contributions

Migration workgroup issues: 4
Seraphis lib PRs: 1

Core contributions

Seraphis dev CCS assessment

Here is an assessment of dangerousfreedom's work done for his recent Seraphis dev CCS.

Seraphis knowledge proofs

The final solution is currently implemented in the seraphis_lib branch in the src/seraphis_main/sp_knowledge_proof_* files. To reach that state, dangerousfreedom and I went through two rounds of significant design revision here, two rounds of private code review that led to major re-writes (the second rewrite followed the second round of design revision), and then the final PR was followed up by substantial edits from me to get the code up to what I consider production quality.

Overall, dangerousfreedom's main contributions here were a nice design for enote unspent proofs, his PR which can be considered a draft for the final implementation, and his writeup which may be useful for future documentation of the proofs.

Seraphis wallet investigations

The result of those investigations was a 3k line set of changes to the current seraphis_lib branch. The changes include:

A Base32 implementation lifted with edits from here, including 90 lines of unit tests. I personally feel it would be better to use a submodule for this.
A wallet3 implementation which is essentially a trimmed-down wallet2 with modifications to support some seraphis behavior. I strongly feel that pursuing a design similar to wallet2 is the wrong direction. Writing a wallet3 just means another round of accumulating technical debt that is very hard to maintain and incredibly unrealistic to extend beyond a certain point.
Added serialization to the jamtis destination, although I want to discourage that as it's not a good way to handle serialization.
The remaining lines contain the two most useful contributions from my point of view. One is a key_container class with various useful methods (I am very skeptical this needs to be a class since keys are not stateful, and the only invariant to enforce is immutability which can be satisfied in other ways), and the other is a prototype implementation of jamtis checksums with unit tests.

Wow! Thank you for taking the time and writing a detailed review as usually :D

Regarding your comments about the wallet:

I don't think we should go for a base32 submodule for two reasons: 1) I don't think that such a tailored base32 submodule exists. 2) I think it is better to freeze it in one file like it is done for the base58 for stability and customization.
I agree that we should not mimic wallet2 design but coming up with the perfect wallet3 design is above my capacities now. I dont even know if it is possible as such a monocratic decision would imply lack of discussion in many of aspects of the wallet. All I can do for the moment is write functions that could be useful to be implemented in the wallet considering it to be modular. I hope our efforts and group discussions will reach a consensus about the shape of the wallet and gluing all functions/modules together will be then easier.
I don't know the best way for serialization and if there is already a standard defined for Seraphis, if I missed it and there is a defined way to do it then I would rewrite my code and use this new standard from now on.

Finally, I believe that these tools I created (based on wallet2) helped me (and hopefully others) to understand what kind of problems we are facing and I think we are steadily solving them. They can also be used as a basic framework to implement new modules that could be easily transposed to other schemes. I think it will be useful (at least for me :p) in the development of the wallet.

I'm not qualified to look at the technical aspects of your work, i can only judge from a superficial/logistics stand point. After reading koe's comment closer it does not sit right with me. Ideally, when the community sponsors a contributor - the work load of everyone else should be reduced (naturally).

It seems that for some aspects of your work this is not the case (specifically the PR merged into the Seraphis library) which seems to have required koe to 'understand someone elses work entirely' and 'make changes to get it up to standard' - this requires paid time from not only yourself but from koe (who also has an active CCS they need to be working on).

I am also seeing that pay is rising - which would appear as though you went through a stage of learning and have now became mostly independent in your work and are now gaining bonus' (which appears to not be the case) as corrections where needed.

I also feel that koes wishes ( see "wrong direction" .. "would be better to use" ) are important. And the final point is hinting at 'more work for koe' to explain/correct the "best way for serialization"

With these none-technical issues in mind, for me, im not excited about this proposal

I'm not qualified to look at the technical aspects of your work, i can only judge from a superficial/logistics stand point. After reading koe's comment closer it does not sit right with me. Ideally, when the community sponsors a contributor - the work load of everyone else should be reduced (naturally).

It seems that for some aspects of your work this is not the case (specifically the PR merged into the Seraphis library) which seems to have required koe to 'understand someone elses work entirely' and 'make changes to get it up to standard' - this requires paid time from not only yourself but from koe (who also has an active CCS they need to be working on).

I have the impression in your question that you imagine things to be black and white when developing new ideas. The reality though is that when there is research involved things are not always like that. My proposal was to develop the theoretical and practical framework for Seraphis with the same features for the knowledge proofs as we have today in Monero. Neither me nor Koe knew how it should look like in the end. I won't go into all the details of the development process but to specifically address your points. 1) Yes, he had to 'understand someone elses work entirely' and that took him time. He did two reviews on previous designs but also our discussions led into modifying Jamtis/Seraphis structure (https://gist.github.com/tevador/50160d160d24cfc6c52ae02eb3d17024?permalink_comment_id=4432591#gistcomment-4432591) and I was also reviewing his implementation of grootle proofs while learning and using it and pointed some serialization issues I was having. In the end things were improved and I believe it is what matters. 2) To be realistic, he is always 'making changes to get it up to standard'. I was weekly rebasing the code I was developing and it was rare when it didn't break. Unfortunately in the PR I did he didn't comment publicly but decided to merge and do the changes he wanted later. As he holds the truth and has the last word about Seraphis, there was nothing I could do. You can see by yourself the changes he made and judge by yourself what he considered 'substantial' before and after. I agree that in general is better as he corrected some tests I forgot to make (it would appear anyway during the wallet integration) and he added more comments but he also removed all the failure messages, which I believe it is worse. But there was no discussion about it. Anyway, I would rather not spend time with these small details at this point of the project. The knowledge proof framework that we have now (theoretical and implemented) is superior than the current protocol as we have more proofs and more ways of privately proving things.

My take away from your and Koe's comments: I have to avoid discussing issues privately. Instead of asking Koe directly maybe I should have created many issues (as apparently people also uses this metric to measure your efforts). If he allows me, I can also make public all our private conversations about this issue if you are interested in the development process.

I am also seeing that pay is rising - which would appear as though you went through a stage of learning and have now became mostly independent in your work and are now gaining bonus' (which appears to not be the case) as corrections where needed.

Yes, I raised from 40 to 48 EUR/h as I believe I built a solid knowledge of the theoretical and also practical use of the Seraphis lib and I believe I can contribute much more and much faster. In your answer you suppose that someone is independent if his works doesnt require any correction. This is a bit weird for me for two reasons: 1) There is no consensus in first place about the Seraphis lib. Koe has the final word and if he thinks that a comma in a comment is wrong then he will make the corrections as he is a perfectionist, which I really admire it on him. But the point is that there is a big degree of subjectivity. 2) Corrections are part of life. I don't imagine a world where people would be doing always perfect works. I indeed expect to take less and less time from others and contribute more.

I also feel that koes wishes ( see "wrong direction" .. "would be better to use" ) are important. And the final point is hinting at 'more work for koe' to explain/correct the "best way for serialization"

Usually Koe's comments/feelings are very important, nobody denies it. As he mentioned these points in his comment, I felt like commenting here too. I'm curious to see how they will be solved in the end but I think the discussions should not continue here. So, no doubt Koe is a fundamental piece and that his excellent work should be paid even more in my opinion. I just dont understand your point 'more work for koe'? He is the maintainer of the library, should we freeze it as it is (no more weekly clean-ups from his side) and achieve consensus for all the following PRs on it? Should he have a less active part in the decisions? I don't know where you are heading here. If you mean that my work should not have required him a lot of extra work, then we go again into the subjectivity field. But I cannot deny that I got really a lot of help from him and I'm very thankful for that. It is beautiful to see his dedication and care towards maintaining his library.

Finally, I slightly updated the proposal as 1) I believe I can do the transaction_history_component faster than I thought. 2) I want to let it a bit more open so I can engage more in the wallet/dev concrete tasks and better report my activities.

P.S. I'm citing @koe a lot here, so feel free to give your opinion about inaccuracies here.

Thank you for the detailed reply, there are indeed some inaccuracies/subjectivity in my comment e.g. submitting a PR would increase the workload of others, requiring it to be understood from top to bottom and changed if necessary. But overall, i feel I've expressed my concerns correctly.

Following on from @koe comment, would it not be wise to separate this CCS into 5 milestones. With a note in your proposal stating that each of your sizeable tasks (i see 2 that qualify for following note) "be at production quality (meaning: clean, correct, comprehensive) with unit tests that are concise and exercise the full range of component interfaces/behavior".

changed the description

@dangerousfreedom, after considering your past work and acknowledging your increasing understanding of the codebase and continued efforts to improve, I think your CCS could be strengthened with the following changes.

Set completion expectations:
- at least 5 sizable tasks completed (sizable means ~20-40hr of work, e.g. seraphis knowledge proofs, transaction history component) at production quality (meaning: clean, correct, comprehensive) with unit tests that are concise and exercise the full range of component interfaces/behavior; complete doesn't mean 'immune to change', it just means stable and usable
- participation in main-repo code reviews for PRs related to wallet development/consensus and seraphis
- all work hours should be logged
Revise hourly rate: based on the rates and productivity levels of other Monero devs, I think you should stick with 40eur/hr for this CCS. For context: [jeffro: 30usd/hr], [tobtoht: 45eur/hr], [selsta: 45eur/hr], [vtnerd: 55usd/hr], [xiphon: 55usd/hr], [moo: 30 usd + .2xmr / hr], [jberman: 48usd + .16xmr / hr], [koe: 50usd + .2xmr / hr].

Very interesting and important discussion.

First I want to mention something that did not yet feature much in the comments so far: What we try here in the Seraphis wallet workgroup with @dangerousfreedom as an important member is very big, and very hard. I don't think we had something like that at Monero at least since RingCT was implemented back in 2017. And because it's hard we have to expect problems. In fact, it would be strange and suspicious if no problems occured.

I had a look at the proofs PR as originally submitted and how @koe modified that code. Without wanting to blame anybody or judge anything, solely looking from the outside at how much code "survived" and how much was changed, I have to say that this worked out decidedly less than ideal.

I see that as a problem that we should try to solve, e.g. by having a calm look at what happened here with the people involved later on, in an attempt to avoid similar things in the future, but I am pretty sure already that improved project management will be part of a solution. I will try to write down some of my thoughts in this regard tomorrow as an issue on the workgroup GitHub.

What we also should try to improve IMHO is "visibility" of work planned, work in progress, and work done. Again, I see the fact that so much work was done without being visible to anybody except the two people directly involved as a project management problem at least in part, and also here I intend to write down some thoughts about how to improve tomorrow as an issue.

About this CCS proposal I had the following thought: @dangerousfreedom, what would you say to stay at EUR 40 for now, but shorten duration to 3 months, with the idea to limit the time you still work for the original rate, and to show to donators how things improve over these 3 months, to clear the way for a rate of EUR 48 for the follow-up CCS?

(This just crossed with @koe 's new comment, he posted while I composed this.)

changed the description

Thank you for the comments and directions. I will be updating it during the week.

added 1 commit

6b2509ca - Update df-wallet-development-2.md

Compare with previous version

changed the description

With the proposal as it is today, quite some change, with a shorter timespan, significantly lower hourly rate, and a much sharper focus on what to work on.

I definitely see this as a much improved proposal. The discussion that already started about the component to implement also looks good to me.

I hope that after a somewhat bumpy start people will come to see this as worthy to move to the funding stage.

The proposal looks more reasonable, thanks.

mentioned in commit 6e17c7bb

merged

I like these mini crossword suggestions, it's very helpful. I think it makes sense to change now

mentioned in merge request !409 (merged)

The full discussion about this module can be found here.

In this work similar features to the wallet2 correspondents: get_transfers, get_payments, get_payments_out, get_unconfirmed_payments_out, get_unconfirmed_payments, export_outputs, import_outputs, import_outputs_from_str, export_payments, import_payments, import_payments_out, get_num_transfer_details, transfer_details, get_tx_proof, check_tx_proof, get_spend_proof, check_spend_proof, get_reserve_proof, check_reserve_proof were created as proposed.

This module contains:

Creation of Transaction History Component:

It allows to efficiently look at transaction records by searching the txid of sent txs.
The transaction records contains:
- the key_images of sent enotes (so the enote_records can be easily recovered from the enote_store).
- the JamtisPaymentProposals so the enote_ephemeral_private_key and destination addresses can be recovered.
Created (de)serialization functions to the components of the class so they can be imported/exported.

Creation of Legacy knowledge proofs from the wallet side:

Removed the node connection/wallet_keys and cache dependence using jeffro's lib (wallet2_basic)
Created a demonstrator for the legacy_spend_proof and in_proof using jberman's scanner.
Unit_tests to fully test those functions may need to be implemented.

Creation of Seraphis knowledge proofs from the wallet side:

All the seraphis knowledge proofs (except enote_unspent_proof as the practical side needs to be more explored (nodes and bc connection)) with unit tests
Serialization is using base58 as do the legacy.

Creation of the show enotes functions to provide useful visualization of enotes and to be able to create the knowledge proofs:

The idea is to filter the enotes by its spend/origin status and sort them by block/timestamp
The filters and sorting can be easily fine tuned allowing many combinations to show/get the wanted enotes.
Wrote a demonstrator for that

I foresee some rework on this module as:

the node connection is still under work so the functions depending on it will have to be updated.
due to the probable change on the view_tags the knowledge proofs will change.
still gathering consensus on how to show enotes and which libraries to use.

New tasks and issues that originated from this work:

Implement knowledge proofs for hardware wallet devices (e.g., trezor)
Implement mutex for thread safe access to enote_store, tx_store and keys (should be done one level above these functions, whenever the wallet call them)
Decide how to show messages to user (same library as wallet2?)
Decide which legacy knowledge scheme we want to move forward with
Find a way to fully test the knowledge proofs (both legacy and seraphis) due to issues to the node connection.

The pace of development in the cryptocurrency and blockchain space is truly remarkable. It's fascinating to see ongoing ICOs shaping the future of finance and technology. To stay updated on the latest projects and innovations, I often visit https://icoholder.com/en/icos/ongoing. It's a valuable resource for anyone interested in this rapidly evolving industry.

I got the impression from reading along a bit while yesterday's community meeting ran that a comment from me about the work done in the frame of this CCS may be useful:

Almost everything regarding the new Seraphis wallet that we build is still very much in flux; only a few days ago we could finally merge the very first PRs to the wallet code repository. It's a bit unfortunate for judging the work result of this CCS, but not surprising because of this, that nothing here already reached conclusion.

Anyway, I did not yet look closely at the main code-wise results, but a first look told me that quite some amount of code was written here, and code that also demonstrates that @dangerousfreedom 's knowledge about the Monero codebase and the Seraphis library improved.

I am optimistic that the code in this PR will reach "production quality" soon and then can be merged as an important contribution to the Seraphis wallet.

The CCS had a time budget of 160 work hours. It looks plausible to me from the outside that those were spent here with on-topic work.

dangerousfreedom - wallet development 2

What and Why ?

Who?

Activity

Work record

Seraphis dev CCS assessment

Creation of Transaction History Component:

Creation of Legacy knowledge proofs from the wallet side:

Creation of Seraphis knowledge proofs from the wallet side:

Creation of the show enotes functions to provide useful visualization of enotes and to be able to create the knowledge proofs:

New tasks and issues that originated from this work:

dangerousfreedom - wallet development 2

What and Why ?

Who?

Merge request reports

Activity

Work record

Seraphis dev CCS assessment

Creation of Transaction History Component:

Creation of Legacy knowledge proofs from the wallet side:

Creation of Seraphis knowledge proofs from the wallet side:

Creation of the show enotes functions to provide useful visualization of enotes and to be able to create the knowledge proofs:

New tasks and issues that originated from this work: