Compilation time reduction and housekeeping
Merge request reports
Activity
Hi koe,
Yes, I have read about the deterministic builds for Monero.
I have absolutely no prior development engagements with Monero. I just like the project very much, been just a full node operator and miner, and now wanted to contribute my collective commercial and open source development experience. The part, that I currently proposed, is universal across all C++ projects, so my contribution doesn't require deep knowledge of the Monero system.
I can assure everybody, that this will reduce the compilation, thus developers', turnaround time. The question is, whether the community and developers want this to happen, before I touch the keyboard and risk, that my changes are rejected. Therefore I would love to reach consent and see just 1 XMR funding for a start, so that I can prove my point - wrapping the boost serialization headers and serializing the wallet2.h in its .cpp file, not in the header, in order to reduce the compilation time.Edited by mjA common theme in discussions around CCS proposals is that people with prior contributions are much more favorable (and trustworthy). As a show of good faith, it might go a long way toward getting this CCS approved if you did the initial milestone as a volunteer.
If your concern is proposed changes might be rejected on a conceptual level, you could ask in the #monero-dev IRC channel, or via Github issue, if those changes are dead ends.
Reducing compilation time sounds like a worthy project, so I hope it gets completed in the end :)
-
I totally understand the trust concern, especially since it comes to money.
Thank you for your tip. Asking in the proposed channels sounds like a good start. After the initial contribution we may always return to this PR discussion. Until then, may the created .md file serve as a presentation for now.Edited by mj I believe this would help with the problems Endogenic highlighted at Konferenco. Hopefully this gets approved and funded!
Has moneromooo provided any feedback on this? I agree with comments from @koe -- would be much better if you had some volunteer PR experience first.
Generally speaking, it's great to see this proposal. Optimization / cleanup is good as long as it doesn't hurt readability and improves maintainability.
-
Thanks for sharing your support.
Yes, he did provide feedback, after I had specifically asked him.
In meantime I was able to bring ccache to the project. The amount of code committed is not large, but the effect size is. The Travis CI compilation time went from 22 minutes down to 2 minutes for each build (excluding exception, where ccache wasn't usable).
Afterwards, selsta picked it up and brought ccache to the "workflows", achieving similar results.
Upon a request by vtnerd, I made the ccache optional.
Optimization / cleanup is good as long as it doesn't hurt readability and improves maintainability.
That's basically my motto. There's a lot of good work, that can be done in this project in such direction without hurting anything. Obviously, there will come moments, where some functionality might get broken accidentally, but that's what you've got the tests for.
I was also struggling with https://github.com/monero-project/monero/pull/6455 , where some disagreements occurred, as well as even some communication gap, for what I reckon. I can continue going this way, but my motivation for doing this for free got somewhat crumbled, since such communication style is no different than at my job. Thanks for understanding.
Edited by mj Hi @mj, I'm honestly at a loss about how to help this proposal. It's unfortunate...
In any case, thank you for taking the time and effort to push forward your project as much as you did!
@mj Are you still interested in this CCS? In general there seems to be community support for it.
-
Dear @koe and @selsta, First of all, thank you again for your support. I haven't given up at all. I simply turned silent for a couple of reasons:
- I was being hyperactive up to May, because I was in a middle of switching jobs and I always like to take a break in between to do something real, like I can do here.
- I was renovating a rental apartment
- I wanted to give the team some time and observe how fast my branches are merged, and if not, then why.
- I was waiting for a delivery for a long time (because of Corona) of a new hardware and then preparing it. It has enough RAM, to be able to compile Monero on multiple cores.
Ad. 3) I noticed, that I was way too fast at the beginning. The branches are merged with a lot of hesitation and carefulness. This was bad for me back then, but OTOH it fits perfectly to a more realistic scenario of me being able to code either only during weekends, or during some short occasional breaks from my daily job. The self reflection part here is, that I have to make smaller diffs, learn git better and test locally, whenever possible (in this order of importance).
Ad. 4) The RAM upgrade was necessary for the reason, that was unknown to me at the beginning, since I'm used to something else in my project. In Monero (and other large projects) the compilation units are too large to fit into memory, when compiling on multiple cores. You need about 6 GB per core for Monero. This will sound very biased now, but guess what is able to reduce the RAM per core needed when compiling? Splitting the unrelated compilation parts, which happens to be the same work that needs to be done in order to reduce the compilation time (still on a single core). Obviosly, if you can compile with less GB per core, you can use more cores, additionally speeding up the compilation time linearly. Also smaller compilation units make it more probable for ccache to do its thing.
By "Splitting the unrelated compilation parts" I mean moving for instance the boost-serialization includes to a file containing only a class or a set of functions dealing specifically with serialization, so that any changes in the original file, where the headers were mistakenly included at the beginning, will decrease the compilation time of this file by not having to compile the serialization headers. The probability of a need of a change of a given functionality in a file plays a role as well - often changed files should be reduced first.
I think we should slowly start negotiating. I do realize, that writing a simple configuration entry in CMakeLists.txt (like I did) is a bit too little, but if anybody can tell me authoritatively or democratically, that you want to make a deal with me, then I'm as ready as I was. I'd then continue analyzing the source code using objective tools, identify the bottlenecks and present the results on (this time) relatively small diff branches. Once we're done, I'd express my request for an hourly pay of about 35-40$ per hour (I have western costs unfortunately).
Edited by mj -
@luigi1111 These XMR prices were set some time ago and were only representative at that time and with my limited overview where the bottlenecks were. I'm pretty flexible, honest and negotiable. All the experience in coding that I have also taught me, that it's hard to talk about fixed prices ahead of time, other than an hourly basis (in dollars). Let's try to implement something that brings value (in other words - accepted, reviewed and merged), I will then tell you how long it lasted, you are allowed to tell me that you'd do it in, say 80% of that time, and thus we'd find the fair number of hours, and in the end multiply by the hourly wage. That dollar sum would be sent via XMR, valued accordingly at the time of making the transfer.
Does this sound OK?
Edited by mj 
@mj hourly is not the best way to go on a proposal like this. Rather, make your best estimate on deliverable milestones, which may just be updating the existing milestones here, if required. You may get overpaid/underpaid on a particular milestone depending on prices and how much work it took, but that's the nature the CCS for the most part. Other than that, please update the XMR prices to reflect current values and squash your commits and I'll merge this one. Sorry for the delay in responding.
-
@luigi1111 No worries about the delay. The most important thing for me, is that the changes I wish to do, are generally accepted. Whatever the course of the project may be, it will be achieved somewhat faster while I proceed.
OK, then let’s try to give a best estimate and update the prices to the current USD price and squash the commits.
I’m now more aware, what needs to be done within the project (cleanup wise), but I need to put a lot of focus now on getting a solid plan, so that we’re always even. I need a few days on this alone.
-
@mj thanks for the update. Let us know what you come up with.
-
@luigi1111 I'm done for now. The XMR amount has been even increased, because I added additional milestones regarding tests. I have also done quite some analysis of the source code in the week, and taking into account the current compilation time (which I want to reduce), the time that I need for the changes is indeed realistic.
I have set the PR to automatically squash the commits once it's merged. Is this enough?
added 1 commit
- 9f25bcf4 - Update mj-compil-time-reduction.md prices and milestones
