layout: cp
title: Compilation time reduction and housekeeping
author: mj
date: April 15, 2020
amount: 52.9
milestones:
- name: ccache for CMake (demo)
funds: 0 XMR
done: 31 December 2020
status: finished
- name: Dynamic linkage
funds: 2% (0.9 XMR)
done: 31 December 2020
status: finished
- name: Automated reports of ClangBuildAnalyser and iwyy
funds: 4% (2 XMR)
done: 31 December 2020
status: finished
- name: Automated reports of Valgrind (test bottlenecks)
funds: 2% (1 XMR)
done: 20 May 2021
status: finished
- name: Optional precompiled headers for CMake 3.16
funds: 4% (2 XMR)
done: 20 May 2021
status: unfinished
- name: Forward declarations of own classes + interfaces
funds: 15% (8 XMR)
done: moot
status: unfinished
- name: One class per header
funds: 4% (2 XMR)
done: moot
status: unfinished
- name: Parallel tests (ctest -jN)
funds: 9% (5 XMR)
done: 20 May 2021
status: finished
- name: Static methods of the wallet2
funds: 8% (4 XMR)
done: moot
status: unfinished
- name: Proper ordering of headers (general last)
funds: 6% (3 XMR)
done: moot
status: unfinished
- name: Miscellaneous hourly work @ $40/hr (23.4 XMR remaining)
funds: 47% (25 XMR)
done: moot
status: unfinished
- name: All remaining -> General fund
funds: All remaining (40.4 XMR)
done: 26 February 2022
status: finished
payouts:
- date: 4 January 2021
amount: 2.9
- date: 17 June 2021
amount: 9.6
- date: 26 February 2022
amount: 40.4Update
This proposal has been marked completed and the remaining funds forwarded to the Monero General Fund at the request of the proposer, after seeking community input. The 12.5 XMR was paid out to mj, leaving 40.4 XMR.
What
The proposal is about minimizing the compilation time of the project.
Who
I have 12 years of object oriented programming experience, mostly in C++. I'm a passionate programmer, not only somebody who does this for money. I hold a M.Sc. degree in Computer Science. Being able to see the code in a hierarchical order, both projects allowed me to create an extensive library, ready to be reused in a project like Monero, with serialization being my first low hanging fruit. My contributions to Monero so far are the following:
- I was able to bring ccache to the project. The amount of code committed is not large, but the effect size is. The Travis CI compilation time went from 22 minutes down to 2 minutes for each build.
- afterwards, selsta picked it up and brought ccache to the CI "workflows", achieving similar results.
- upon a request by vtnerd, I made the ccache optional.
Why
During all these years I have noticed how important it is to have a quickly compilable code base, which would otherwise put a negative psychological pressure on developers, making them refrain from changing anything for the better, not to mention the obvious reduction of required computational resources. For Monero specifically, I have set up the following experiment: I have calculated the sizes of header files, summing up the sub headers included by each of them (column 3). Then I have calculated how many times a given header is included by .cpp files (column 4), thus indicating both the approximate compilation time of the header and how many .cpp files would be affected by the change if the header (column 2) and sorted ascending by this value. Below is the list of the greatest 90% of the headers, using this convention:
11% = 10495 = 2099 * 5: cryptonote_boost_serialization.h
12% = 11907 = 1323 * 9: wallet2_api.h
13% = 12393 = 4131 * 3: cryptonote_core.h
13% = 12924 = 718 * 18: crypto.h
16% = 14856 = 4952 * 3: core_rpc_server.h
17% = 15990 = 1599 * 10: rctTypes.h
17% = 16500 = 3300 * 5: blockchain_db.h
26% = 24225 = 8075 * 3: blockchain.h
30% = 27979 = 3997 * 7: core_rpc_server_commands_defs.h
61% = 56616 = 2696 * 21: cryptonote_basic.h
100% = 92620 = 9262 * 10: wallet2.hIt becomes obvious, that the wallet2.h is the largest "hot spot" of the whole project. While compilation of the project took 30 minutes, touching the wallet2.h and recompiling took entire 6 minutes = one fifth.
Milestones
What can be done with this is creating as many wrappers of the boost library as possible and putting as much as implementation code into .cpp files, instead of insisting on writing them inline, when these spots aren't bottlenecks. Putting a trivial method as inline may help, but only when it's called very very frequently, and only if that improvement is a large percentage of other parts of the software, which it usually isn't. Inlining has to be proven by profiling the software, and not being a default policy, since it brings nothing, while costing a lot, not only because multiple recompiles of the code in .cpp files in one session, but recompiles upon changes of the inlined implementation.
I'd like to earn something like 40/h. It's hard now to assess how much time it will take, so I'm not strict on the concrete values. If I happen to finish ahead of time, thus becoming overpaid, I will admit it. I will be writing the time of work in each of my PRs. By assessing the payments, I will now assume a price of XMR of 125.
Milestone 1: ccache for CMake (demo)
Done with quite a success. The Travis CI was relieved and you, as a developer benefit each time you switch a branch. Previous text: "I'd like the CMake script to automatically pick ccache and clcache, when it can find them in the PATH. This piece of software helps in reducing the compilation time of compilation units (.cpp files and all their included headers), when their content hasn't changed. This means, that the more forward declarations and fewer included headers your headers have, the more the ccache will be able to leverage your discipline. This is especially useful when switching between branches."
Milestone 2: Dynamic linkage
Static libraries tend to grow in sizes exponentially and slow down the generation of the final binaries. I would like to enable (opt-in) dynamic linkage in CMake for development purposes. Also whenever you are done writing a new test and would like to just modify the production code and just execute the test, the test binary can be made so, that it doesn't have to relink upon change of the production code's resulting shared library. This is quite a low hanging fruit. There are 70 CMakeLists.txt, so multiplying each one by 2 minutes gives 2.33h plus 0.30h for creating some framework for further such changes gives 2.83h, and that equals to 0.9 XMR.