Progress is going as expected, currently on day 39.
cuprate_database
's main "base" component is complete.
Around 4 databases were tested with during this time:
In the end, LMDB and redb
were chosen.
These are some extremely unrealistic synthetic benchmarks of the 2 database backends that are currently implemented, LMDB
(via heed
) and redb
.
safe
= sync to disk per transactionfast
= never forcefully sync to disk (until the end)function | LMDB (safe) | redb (safe) | LMDB (fast) | redb (fast) |
---|---|---|---|---|
tx_ro | 0.208 | 0.294 | 0.216 | 0.295 |
tx_rw | 0.153 | 16.08 | 0.153 | 2.635 |
open_db_ro | 0.148 | 0.773 | 0.150 | 0.757 |
open_db_rw | 0.149 | 1.319 | 0.151 | 1.328 |
put | 2.714 | 6.636 | 2.722 | 6.603 |
get | 1.936 | 2.892 | 1.906 | 2.924 |
get (random) | 2.477 | 3.285 | 2.472 | 3.259 |
get_range | 0.053 | 0.268 | 0.053 | 0.264 |
delete | 2.814 | 11.18 | 2.838 | 11.11 |
put | 2.698 | 7.016 | 2.669 | 8.176 |
delete (random) | 3.772 | 12.77 | 3.747 | 13.62 |
This data should not be taken too seriously as the tests are very unrealistic and nothing has been optimized; it's just to get a sense of things. Testing code here.
The internal documentation (cargo doc cuprate_database
) is complete, and the main README.md
documentation is waiting on other parts to be implemented.
The other major components left to be implemented for external usage are:
Block
, get_block()
)and of course their documentation and tests.
Cuprate (database):
Cuprate (misc):
Upstream:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Key: https://github.com/Cuprate/cuprate/blob/main/misc/gpg_keys/hinto-janai.asc
Asking for milestone 1's payout for:
https://repo.getmonero.org/monero-project/ccs-proposals/-/merge_requests/422
to the following address:
44hintoFpuo3ugKfcqJvh5BmrsTRpnTasJmetKC4VXCt6QDtbHVuixdTtsm6Ptp7Y8haXnJ6j8Gj2dra8CKy5ewz7Vi9CYW
-----BEGIN PGP SIGNATURE-----
iHUEARYIAB0WIQQxxRRar6Wo3xwdsqbUfOBfoXWkmQUCZfJN7QAKCRDUfOBfoXWk
mbu7AP4ivj+txl119+ZAPTDe2g17kPRJa2PqcFOmktzLG+FXBwEAz+WRkWG/84iK
Ge+G+lQLq4duiBNg23Tii5txRab2wAk=
=wvF1
-----END PGP SIGNATURE-----
hinto (1cbfee7c) at 02 Feb 13:31
hinto (1cbfee7c) at 21 Jan 16:07
Update hinto-2023-nov.md
I've added milestones and made it more clear what the intention/end-goal of this CCS is (database).
hinto (896b0a8f) at 15 Jan 21:34
Update hinto-2023-nov.md
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
It's been 1 year since release - requesting payout for the last milestone.
44hintoFpuo3ugKfcqJvh5BmrsTRpnTasJmetKC4VXCt6QDtbHVuixdTtsm6Ptp7Y8haXnJ6j8Gj2dra8CKy5ewz7Vi9CYW
https://github.com/hinto-janai/gupax/releases
https://github.com/hinto-janai/gupax/blob/main/pgp/hinto-janai.asc
-----BEGIN PGP SIGNATURE-----
iHUEARYIAB0WIQQxxRRar6Wo3xwdsqbUfOBfoXWkmQUCZY2IoQAKCRDUfOBfoXWk
mT+TAP4sovoSAbt/pIfLyLGtzA+wI/jF5IOEVp5alIwRY0eKUwEA8MWu6TrEybZ0
GIO6QmbL//tvFHnKxMzI8iRBriV+pQQ=
=8VRk
-----END PGP SIGNATURE-----
Updated description to reflect the past ~1.5 months of discussion with Boog.
hinto (4598b128) at 19 Dec 17:23
Update hinto-2023-nov.md
hinto (39108092) at 19 Dec 17:18
Update hinto-2023-nov.md
hinto (8f31100d) at 19 Dec 17:16
Update hinto-2023-nov.md
hinto (7f5fb28b) at 19 Dec 17:11
Update hinto-2023-nov.md
Cuprate is an alternative Monero node implementation, currently solely worked on by Boog900.
I've been in contact with Boog and there are many sections within Cuprate that can be worked on in parallel, currently the most pressing section is the database. This work was left unfinished by SyntheticBird45 before they left the project. Boog is currently working on block downloading/verification/consensus with cobbled together database-like functions - this means an actual database layer can be worked on in parallel without any merge conflicts, and can be cleanly slotted in once complete.
This CCS is to support me working on full-time on Cuprate for the next 3 months, specifically on the database implementation.
This CCS will fund part of a larger effort to create an alternative Monero node implementation (among other things).
As such, things will initially be slow-going as groundwork is laid. This CCS specifically may not directly benefit monero-core
immediately, yet it's work that must be completed such that Cuprate can benefit monero-core
(and Monero) in the future.
I'm hinto-janai.
Past CCS: https://ccs.getmonero.org/proposals/gupax.html.
The design document and active PR is here: https://github.com/Cuprate/cuprate/pull/35 and will be updated as things change.
The main goal is to create a layer that separates the underlying database and the higher-level functions that are called by the other portions of Cuprate, e.g, get_block()
. The aim here is to allow for easy hot-swapping of databases to test for the various operations Cuprate will be performing. In the end, a single database will be selected after testing, although the layer will continue to exist.
The database implementations in mind (for now) include:
Both of these will be forked and maintained by Cuprate as there are monerod
-specific things that must be added (e.g output lookup using sub-keys).
The overall design is mostly from Boog900 - I am simply implementing it.
There's plenty more work to be done after the database. Boog plans on handling P2P code after his current CCS, which means I could possibly work on these things in the future (in parallel with Boog's work):
get_info
)./cuprated --option
)Most likely, the RPC server would be the next thing I would be working on.
And there are things that are planned, but for the very distant future:
C/C++ <-> Rust
bindings but as RandomX is in active use (and will most likely be for the foreseeable future) another implementation or more realistically, a maintained "Rust"-like shim on-top would be nicemonerod
)These points are laid out to showcase what else must/could be done for Cuprate to reach some level of parity with monerod
.
Cuprate has an advantage in that it has an 8+ year old codebase as reference - although we are starting with a blank canvas, we have a finished painting next to us, displaying all the very good, okay, and not-so-good parts.
As such, we can focus on things other than making it work, notably: documentation, maintenance, and correctness.
Other than documenting the Monero protocol itself, we plan to write documentation for internal subsystems such that future Cuprate contributors/maintainers will be able to pick up easily where we left off. What this means for me and this CCS is that I'll be documenting how this database "layer" fits in with the rest of Cuprate, and answer all the "what does this do?", "how does this work?", "why was it done like this?" questions. If I were to continue working on Cuprate past this CCS, i.e. the RPC server, all RPC types would also be documented (and the internals as well) - the same goes for any future subsystems.
Other notable projects following this pattern: Tor, Bitcoin, Ethereum, Zcash.
I propose to work for 44 hours/week for 3 months.
By the end of this proposal, Cuprate will have a working database. I believe 3 months is enough time to create and refine an implementation enough for usage.
It is hard to separate large single implementations like this cleanly into goals/timeframes as usually many things get worked on at any given moment - although, there are 3 bigger milestones this CCS has:
Milestone 3 is only to be paid out upon:
This means if the database takes 428~ hours to complete, I will continue to work for 100~ more hours (most likely on the things listed in Future
) to complete milestone 3.
hinto (44a43649) at 06 Nov 22:38
Update hinto-2023-nov.md
hinto (c155d658) at 06 Nov 01:09
Add new file
hinto (d8d70bec) at 06 Nov 00:53
hinto (de3af254) at 31 May 16:27
Add new file
hinto (95d4bc42) at 31 May 13:03