Modularize Monero

Modularize Monero

About

This proposal aims to modularize the Monero codebase, transforming it from it's current monolithic structure into a more modular codebase which can be cherry-picked and taken apart for different use cases.

Objectives:

(Ordered by importance)

1. 100% Backward compatibility with the exiting code case. The source base will be transformed that a complete compile will yield still the same outcome and functionality. Where necessary there will be created wrapper code to stay compatible.
2. Separations of concerns. All functionalities should be separated modules to cherry pick what to use and/or compile. E.g. if you build an hardware wallet, you don't need a huge part of the code, but at the moment it is a very difficult part to separate it. Maybe there even possibilities to run on micro controller an hardware wallet, what at the moment looks already out of scope. XmrSigner on a pi zero armv6@800MHz and 512MB RAM, needs >3 minutes to load or restore a wallet. But there is hope with the findings on separating all in modules, to find better ways to do signing and handling of outputs and key images.
3. Minimize external dependencies, all cryptographic code should be included in the source code. External dependencies often lead to breaking things later, maybe the domain vanished, or the author wants to make a statement and takes it offline, etc. Cryptographic code anyway almost never changes after established once, and normally the license allows to include the that source code into your own. To make sure that changes because of vulnerabilities/fixes get updated asap the source should be monitored if possible. Other dependencies like boost should be minimized as far as possible, and if not possible to get rid of the dependency at least it should be possible to compile without if the provided functionality is not needed. All code that can be substituted to Qt6/C++ will be substituted with the hope of eliminate things like epee.
4. Improved readability and understandability. How almost all touching real world value, a new developer should have it easier to understand what is going on, then browsing through a source file with 15K LOC and then jump through external helper functions or complicated constructs. Source should be as simple as possible but not simpler.
5. Flexible compilation. The source should be restructured in a way that you can cherry-pick which code to import into your project, have it easy to update modules when modules change. Also should it be easy to generate a library where you cherry-pick the needed modules/functionality.
6. Modular architecture. Redesigning the architecture so that it is possible to exchange modules easily, like want to test a new blockchain back-end, simply inherit from the blockchain interface and write the new module using another database or protocol to retrieve the new blocks....
7. Comprehensive documentation. Every developer loves to have up to date and complete documentation, and every developer hates to write it. In the transformation the current monolithic source and the new modular source should get a top notch documentation, because before transforming the source one anyway first needs to understand, and so 70% of the documentation work is almost done, only missing to write it down and phrase it in the best possible way. After transforming also the differences and what the code does is pretty clear and therefore is the best moment in time to write. This effort should benefit the current like the new implementation equal.
8. Enhanced auditability. Through a more broken down and easier to read code base and improved structure and documentation, both current and new source should be easier to audit by entities not familiar with the source yet. (If you audit source you already know, you get also blind to some issues, because they are simply familiar).
9. Facilitate future translations. The modularized structure with clear interfaces and comprehensive documentation should it end also lead to potential future translations into other languages like Rust, Kotlin, Erlang or Python (the current monero-python library is e.g. almost only a wrapper for the wallet rpc).
10. Cleaner Code. Somehow a bit redundant to point 2, 3 and 4, only addition to adhere to modern C++ practices and clean code principles.

Non-objectives:

1. Performance. Of course the code should be as performed as possible, but no objective will be sacrificed for performance, clarity above performance.
2. Substitute the work of the original authors or diminish the worth of their work. Their work is highly respected and cooperation is searched to tame the monster. It is only natural that the monster grows as long you feed it. And it is logical that you feed the monster if you love it and if it lives in your house. That this source has become a so monolithic beast is almost unavoidable. Of course it would be cool if one day this effort will lead to the core development on the more modularized code base, and for that I will search also current developers opinions while walking the path, but the transformation is not done for that purpose. The current developers have light-years more experience and knowledge in that source - but like normal (I assume) no time to refactor the whole monster - even less while worked on that very source actively.
3. Another language, introducing new libraries or tool-kits. The very purpose is less of all with the same functionality. And sticking with Qt6/C++ is very intentional, first needs to be a modularization and comprehensive documentation, only after that it can be thought to go a step further IMO.

Challenges

1. Transforming source which is under current development. Somehow like making a surgery on a person out for a walk, while walking... But I hope I can mitigate this challenge in working on modules, one at the time and have like a recipe from A to B, so if there are not huge changes it should not derail me.
2. Qt6/C++ newest C++ standards and best practices how I'm a novice on C++. I develop for more then 20 years now, but touched during that 20 years only C and once for a hack I modified C++ code and only recently I started with Qt/C++. So as usual I need to get up to speed while I'm doing. It is always easier (in the beginning) to modify source in a language you are not fit yet then write from ground up until it flips and writing from ground up will be easier. But I still in the phase where I will have to read some books in parallel.
3. The time effort is impossible to estimate as long the the source is not completely understood, and even then it is hard - but I assume as soon the complete source is understood, almost all the work is also done.

Who

Me, Thor a.k.a vThor a.k.a DiosDelRayo, I'm about to finishing the XmrSigner Monero Signer Resurrection and encountered various difficulties on getting XmrSigner production ready, like main points monero-python is almost only a mere wrapper for wallet RPC, the monero source is (almost) take all and digest or take nothing. So finishing this proposal will be directly beneficial for XmrSigner and probably even lead to a XmrSigner NG dropping a lot of ballast, going directly bare metal and not using an interpreted language like Python, but Qt/C++, Rust or Zig. Since more then 20 years I'm in the IT field, from developing (where I started) over network administration, to secure communication systems. Wrote code in a lot of different languages and prefer always to learn what is needed on the way instead of only making that what I know fitting for the operation. The hammer I know well, but it is not always the best tool to archive anything.

Why it is important for community

• It would make developing new wallets or even projects not heard about easier for new developer and so attracting more developers creating products for the Monero Ecosystem.
• It would improve audits of the monero source.
• Me getting familiar with the monero source would create also the opportunity to help one day on that very source.
• Possibility of finding hidden bugs, how I will need to read and understand the source code intensively.
• Maybe the possibility to get parts of monero working on even more restricted/limited hardware like micro controller for hardware wallets or similar products
• Getting XmrSigner well done, at the moment the state is unsatisfying at best.

Milestones

How it is impossible to predict or estimate anything as long not familiar with the whole source, I prefer to propose to work at least 130h/month for 45 XMR (don't care about the exchange rate because XMR should be the anchor) for 3 months, and very probably writing a new proposal to continue. All what I work above 130h in one month shall go into the next month/milestone, I will give a biweekly report below the proposal in GitLab and commit and push the changes each day (normally I push only after a big junk is finished, but here it makes sense to push daily for accountability reasons).

Results in 3 Milestones of 130 hours for 45 XMR each, in total 390 hours for 135 XMR.

Project Timeline

Project timeline is open how time cannot really be estimated (at least for now, maybe it becomes easier after the third milestone). So it is simply to process as much as is possible per day.