j-berman full-time 3 months part 4

What

Continue full-time another 3 months in a similar capacity:

• Prioritize debugging any significant issues that arise, such as daemon connection issues, zmq missing publishing txs submitted to the daemon, recovering multisig wallets from seed, etc.
• Complete work stress testing how daemons handle a transaction pool under heavy load to both gauge daemon performance, and to help get PR 8076 - reduce wallet round trips to the daemon across the finish line after gauging its impact on pool processing performance.
  • The day leading up to the hard fork this summer, the transaction pool experienced heavier load than usual. Daemons that saw heavy traffic reported issues serving RPC requests. I would like to dig deeper into this.
• Start working through Seraphis and Jamtis wallet tasks piecemeal, picking up collaboration with @koe, @rbrunner7, and @dangerousfreedom. The first task I've gotten a handle on and have a decently fleshed out view of what work is needed is input selection. I intend to get @koe's naive input selection implementation to production-ready, ensuring it functions as close to the wallet does today. From @koe here: "a real implementation needs many more mechanisms and heuristics (e.g. filter for dust threshold to ignore, filter for origin statuses to permit, randomization and spread to mitigate timing analysis)"
  • The general plan with Seraphis/Jamtis is to start fresh with new wallet code that avoids the complexity and pitfalls of wallet2.cpp, which at the moment is a source of growing technical debt (discussed a bit here). Considering Seraphis and Jamtis would bring a number of changes to core wallet features (a new address scheme, new balance recovery algorithm, new fee calculation, etc.), the proposed plan currently on the table is to start fresh rather than factor the existing wallet2.cpp. Many of the features that users already expect (such as sweeps, tx key recovery, watch-only wallets, etc.) would be maintained, or improved where possible. The implementation of some existing features is still up for discussion, such as accounts.
  • I'd like to reiterate this point: Seraphis and Jamtis still need to undergo stringent review both from the community and by independent 3rd parties. At this time, @koe is still working on the technical paper describing the protocol in its current iteration, which will need to undergo review in the future. Further, feedback on the core feature changes is still strongly encouraged and desired.
• Review PR's.

Who

I've identified and patched several privacy issues with varying severity in the Monero ecosystem:

1. The reference wallet's decoy selection algorithm didn't select very recent spendable outputs in some cases. (source)
2. The reference wallet truncated integers in the decoy selection algorithm, which would have borked the decoy selection algorithm entirely if tx volume were to increase; in the normal case, it marginally weakened the algorithm. (source)
3. openmonero was still using the old proven weak decoy selection algorithm, also leaving a fingerprintable trail by decoy selection algo. (source)
4. MyMonero didn't use the updated CLSAG fee calculation which fingerprinted MyMonero txs on chain by tx fee. (source)
5. MyMonero's fee calculation->input selection logic differed ever-so-slightly from the reference wallet, resulting in a fingerprintable tx fee. (source)
6. monero-lws fee masking on the server caused ever-so-slightly different fee calculations from the reference wallet, resulting in a fee fingerprintable to monero-lws (a fingerprint distinct from MyMonero). (source)
7. In PR review on the latest hard fork's changes to the tx fee, identified the introduction of slightly different fee calculation logic that would have caused tx fees to be fingerprintable to either old or new version until the hard fork.

Other contributions:

• Implemented view tags to speed up wallet scanning.
• Implemented background sync initially proposed by @hyc so that wallets can scan for transactions in the background when the user is not active, without the spend key loaded in memory.
• Identified and patched daemon reliability issues (moved forward solving issues 8520, 6631, 6929, and 6938):
  • Tor/i2p daemons periodically send a peer list that gets itself dropped.
  • The daemon periodically drops outgoing tor/i2p connections.
  • Tx re-relay math was off.
• Implemented the changes needed for Ledgers to function for the latest hard fork:
  • monero repo
  • ledger repo

Prior CCS proposals:

• https://ccs.getmonero.org/proposals/j-berman-3months-full-time-3.html
• https://ccs.getmonero.org/proposals/j-berman-3months-full-time-2.html
• https://ccs.getmonero.org/proposals/j-berman-3-months-full-time.html

Proposal

253 XMR. 480 hours, 0.16 XMR/hr + $48/hr, $131/XMR from coingecko.

I'm requesting a raise from my prior CCS because I feel I have continued to demonstrate my work is worth a competitive market rate.

I think j-berman is a valuable asset to keep in our wallet :)
Definitely he will properly do what he proposes and much more by helping others and providing useful comments in the Seraphis wallet discussions.

Pretty much a no-brainer for me. We can be glad to have a competent dev like him at hand, and ready to work on Seraphis and Jamtis.

Expert C++ coders that know cryptography and Monero are impossible to find. When one finally comes along like j-berman proving himself over and over with the work he produces, he has to be given our full financial support. His CCS proposal should be pushed immediately to the funding stage and the community donate what he asks. Words cannot express how worthwhile j-berman is to the Monero project.

mentioned in commit e914486b

merged

Update 1: ~200 hours

• Completed my investigation stress testing the daemon RPC server when the pool has lots of txs in it. During this investigation, I noticed a few things:
  • PR 8076 (a PR to reduce round trips a wallet needs to make to a daemon to load) is a significant improvement.
  • There is significant room to optimize how long the daemon takes to process requests for pool txs (including when serving wallets not running PR 8076) on the order of ~60-90% (the speedup is more significant the more txs there are in the pool).
  • PR 8076 was not enforcing a limit on the number of pool txs a restricted daemon serves to a client, but enforcing a restricted limit is behavior in the current daemon version v0.18.1.2.
  • I could optimize how the daemon processes pool txs, while also updating PR 8076 to enforce a restricted limit.
  • I noticed a few other improvements for PR 8076 too.
  • Here are the changes I made to PR 8076.
• Implemented a basic "block scanner" using @koe's Seraphis library.
  • Successfully recovered spent and received outputs from a chunk of blocks served by a local testnet daemon running v0.18.1.2.
  • My code is here.
  • I also discussed this experience in the context of a migration to "wallet3" here. It was a positive, enjoyable experience for me.

Instead of working on Seraphis input selection as I initially mentioned in my CCS, I'm planning to continue working on Seraphis balance recovery instead, which would also obviously be a critical component of an upgrade. My immediate goal is:

Use the Seraphis lib to implement a legacy full wallet scanner that makes network requests for chain data to the daemon.

The scanner should handle reorgs correctly and have performance on par with (or better than) wallet2. The code should be re-usable in the CLI wallet, GUI, and RPC wallet. I also want to make it clean and easy for wallet developers to do the same.

• Bonus: demonstrate how an external program can handle threading easily.
• Bonus: implement a legacy offline wallet that points to a legacy watch-only wallet scanner.
• Bonus: the scanner can efficiently handle scanning >1 wallets at once.

Update 2: 382 hours

• Used the Seraphis lib to implement a multithreaded scanner that is clocking in ~5-10% faster than wallet2 on my machine (when compiled in the CLI wallet).
  • Used the scanner to successfully recover the balance of a mainnet wallet that has over 50 RingCT transactions, pointing the scanner to a local daemon.
  • Continuing to harden the implementation.
  • Source.
• Continued discussion and ideation on the migration to update wallets, using my work on the scanner above to help guide discussion.
  • Source.
• Investigated an issue causing some self-compiled daemons to hog the CPU.
  • Proposed a patch that prevents this from occurring.
  • Source.

Final CCS Update

Total hours worked: way over 480 (I stopped keeping track at 480).

Since last update, I completed a mock implementation of an optimized wallet scanner that points to a live daemon, using an asynchronous architecture that @koe came up with + wallet utilities from the Seraphis library. When scanning via a remote daemon that @selsta is running, I observed a speedup of ~35-45%. When scanning via a local daemon, I observed a speedup of ~5-10% on my machine (same as in prior updates above). To be clear, this is legacy wallet scanning meaning this scanning code should be usable to see these kinds of speedups today and does not require any changes to Monero's consensus nor have any impact on privacy (though it's not production ready yet).

My code is here: https://github.com/j-berman/monero/tree/1180c8262e405708411d166ef3cb36bd5f98cb92

I wrote a utility program called monero-blockchain-scanner that anyone can use to benchmark this scanner and compare it to wallet2 (wallet2 contains the scanner most wallets use today [GUI, CLI, RPC, Feather, Monerujo, Cake + I'm sure others]). To run the program, build the code linked above, then from your CLI, cd to the bin directory and run the following command:

./monero-blockchain-scanner --daemon-address 88.99.173.38:18099 --start-height 2840000

@selsta is running a modified daemon at that address and will likely take it down at some point (thank you @selsta!!!). To use your own node, you can either build the code above and run monerod from the bin directory, or you can apply this patch on top of v0.18.2.2.

I'm planning to open a new CCS soon to get this scanner to production ready and continue with Seraphis wallet work.

Additional relevant comment

I made the following deal with @kayabaNerve: if he progresses research into trustless full chain membership proofs for Monero to the point that this research can be handed off and run with, I will do a comparable amount of work on Serai, the cross-chain DEX. He held up his end of the deal and as such, I will do a comparable amount of work on Serai. My CCS work on Monero is still my number 1 priority and will continue to dominate my focus and attention. I figure it's also worth noting that my first task I'm working on for Serai is ensuring that monero-serai calculates the same fees that wallet2 calculates when constructing a tx, and uses the correct decoy selection algorithm. This is a step toward making monero-serai a usable library for production wallets today.

mentioned in merge request !401 (merged)

mentioned in merge request !574 (merged)