j-berman full-time 3 months (part 5)

What

Work full-time 3 months on:

• Continuing Seraphis wallet library work.
  • My next task is to complete an async wallet scanner implementation using the Seraphis wallet lib that points to monerod.
    • I have a working implementation that can be tested as described here.
    • When I point this async scanner on my local machine to a remote monerod instance @selsta spun up, I observe a ~40% speed-up in scanning over the current wallet2 scanner.
    • The bulk of the remaining work to get it to PR-ready includes:
      • Explore replacing the existing epee http client lib with a widely used http client lib like libcurl. This avoids needing to rewrite the epee http client lib to safely handle concurrent network requests.
      • Code cleaning/factoring.
      • Unit tests.
  • Once I complete the async scanner, I plan to collaborate with @dangerousfreedom @rbrunner7 and the no-wallet-left-behind working group to work on whatever is next highest priority to get the Seraphis wallet lib ready for production.
• Integrating full chain membership proofs into the Seraphis library (and therefore demonstrate how to integrate them into the core Monero repo). The goal is to construct and verify Seraphis transactions using full chain membership proofs, including a suite of unit tests and benchmarks.
  • In my view, before Monero could actually consider accepting a PR for full chain membership proofs, there is a large amount of work still needed to prove its security and to land on a ready-for-production design. That work is currently proceeding in parallel. I would personally like to proceed with the above implementation work to maximize certainty that the Seraphis upgrade path is compatible with reasonably efficient full chain membership proofs.
• Miscellaneous Monero work, debug issues, review PR's.

Who

j-berman on github / jberman on matrix

Past CCS's:

• https://ccs.getmonero.org/proposals/j-berman-3months-full-time-4.html
• https://ccs.getmonero.org/proposals/j-berman-3months-full-time-3.html
• https://ccs.getmonero.org/proposals/j-berman-3months-full-time-2.html
• https://ccs.getmonero.org/proposals/j-berman-3-months-full-time.html

Proposal

216 XMR

480 hours, 0.16 XMR/hr + $48/hr, $166/XMR from coingecko

Yes please.

No-brainer, I don't see any downsides of this proposal for obvious reasons and the shitload of great work being done in the past by Justin.

mentioned in merge request !403 (merged)

mentioned in commit 4a5c93dc

merged

Update 1

Hours worked: ~250

After a solid review from @rbrunner7 on my wallet background sync PR triggered further discussion on the feature, I ended up making significant changes to the PR which ended up dominating my time. I implemented the core changes discussed and submitted them for review yesterday.

I also moved forwards with the async scanner in the Seraphis wallet lib as explained below.

I expect continuing work on background sync will take up a slice of my time going forward, but I still expect I will be able to continue making progress on the Seraphis wallet lib async scanner, and make initial progress on full chain membership proof integration before my CCS is complete.

Background Sync

Following discussion with @rbrunner7, @valldrac and @sgp, I made significant changes to the background sync PR in order to support devs who want to offer optional auto-background sync without a user needing to manually open their wallet first. With this option, a user's system could have perpetual access to the user's view key and not spend key. Mobile wallets for example could keep a separate randomly generated password stored in the device's secure key store, which can be used to auto-background sync without loading the spend key into memory at any time.

Before this change, the feature did not enable users to grant their system perpetual access to just the user's view key and not spend key, and required the user to open their wallet first (and load their spend key into memory) before background sync could trigger (wiping the spend key from memory at that point). This method of background syncing is still supported (require user open their wallet first -> background sync when user goes inactive); wallet devs have also indicated interest in this form of syncing (@r4v3r23 and @tobtoht here).

See the PR description for a more detailed explanation of how it works.

Seraphis wallet lib Async Scanner

• Explored using the libcurl http client to support concurrent network requests. (https://github.com/seraphis-migration/wallet3/issues/58)
• Implemented a basic client connection pool using the epee http client. (https://github.com/j-berman/monero/commit/0562fa8590389472539a5fb20054d70fe5c33565)
  • You initiate an RPC using the connection pool like this: conn_pool.rpc_command<cryptonote::COMMAND_RPC_GET_VERSION>(sp::mocks::ClientConnectionPool::invoke_http_mode::JON_RPC, "get_version", req_t, resp_t);
  • The pool will then allocate and initiate a new http connection if all already created connections in the pool are processing a request, or reuse an existing connection.
• In my scanner implementation, I abstracted the request to fetch blocks from the daemon.
  • This way a dev using the scanner can easily "bring their own http client" to the scanner, and whatever http client/network gadget is used has no impact on the scanner code.
  • https://github.com/j-berman/monero/commit/0562fa8590389472539a5fb20054d70fe5c33565#diff-a88c71bdc77be82cc82b6c49d39ab41b7d2a0d5bb66309baf5d57df6a28a49adR143-R144
• Next I plan to continue exploring swapping in usage of libcurl as an http client. Considering it offers a connection pool out of the box, it's still an intriguing option. I also plan to benchmark libcurl versus epee.

Full chain membership proofs

Nothing significant to report on this end from me yet.

Update 2

Hours worked: 342

• Completed all TODO's for the overhaul to the background sync PR to the core monero repo.
  • Source: https://github.com/monero-project/monero/pull/8619
  • The GUI PR is still a WIP.
• Got an initial setup working to call @kayabaNerve 's full chain membership proof code written in Rust from a Monero repo C++ performance test.
  • Source: https://github.com/j-berman/monero/commit/722f266edd75e3ad521b01a9b6b92dae72c1ffa9
  • To test: clone this repo at this commit, run make, then run the test_curve_trees performance test (<build dir>/tests/performance_tests/performance_tests --filter "test_curve_trees").
  • I used the CXX crate to call Rust code from C++ for its flexibility (it can also be used to pass C++ code into Rust), and for the author's argument that it is safer than the alternative (cbindgen/bindgen) since CXX offers stronger static analysis guarantees at build time (source).
  • Still TODO: construct a fcmp in a tx using the Seraphis wallet lib.
• Submitted PR for a utility to convert legacy tx's outputs into Seraphis wallet lib compatible enotes.
  • Source: https://github.com/seraphis-migration/monero/pull/11
  • I use this utility in the WIP Seraphis wallet lib async scanner that scans legacy txs via the /getblocks.bin RPC endpoint.
  • Still TODO: complete PR for the async scanner.

mentioned in merge request !385 (merged)

Update 3

Hours worked: 500+

• Completed changes to the GUI PR for background sync and made corresponding changes to the core repo PR.
  • GUI source: https://github.com/monero-project/monero-gui/pull/4050
  • Core repo source: https://github.com/monero-project/monero/pull/8619
  • Implemented review comments (and fixed bugs) on the background sync PR.
  • Background sync took up a significant portion of my time since last update again.
• Pushed a draft PR with clear remaining TODO's for the Seraphis lib async scanner.
  • Source: https://github.com/UkoeHB/monero/pull/23
  • This draft PR explains more fully how it works and highlights the scanner's key features (beyond its significant speedup).
  • I estimate roughly ~3 weeks of work to complete it.
• Explored swapping epee's http client with libcurl in the async scanner. I found that libcurl's easy interface benchmarked roughly equivalent to epee. I've decided to continue with epee as a result instead of introducing a new dependency.
  • Source: https://github.com/j-berman/monero/commit/d814a3c077ebdacf24caf2913e96224abccb7160
• Implemented review comments on the PR for a utility to convert legacy tx's outputs into Seraphis wallet lib compatible enotes.
  • Source: https://github.com/UkoeHB/monero/pull/22
• I did not make further progress on full chain membership proof integration since update 2.