layout: fr
title: Compilation time reduction and housekeeping
author: mj
date: April 15, 2020
amount: 52.9
milestones:
- name: ccache for CMake (demo)
funds: 0 XMR
done:
status: finished
- name: Dynamic linkage
funds: 2% (0.9 XMR)
done:
status: unfinished
- name: Automated reports of ClangBuildAnalyser and iwyy
funds: 4% (2 XMR)
done:
status: unfinished
- name: Automated reports of Valgrind (test bottlenecks)
funds: 2% (1 XMR)
done:
status: unfinished
- name: Optional precompiled headers for CMake 3.16
funds: 4% (2 XMR)
done:
status: unfinished
- name: Moving boost headers out of own headers 1/3
funds: 9% (5 XMR)
done:
status: unfinished
- name: Moving boost headers out of own headers 2/3
funds: 19% (10 XMR)
done:
status: unfinished
- name: Moving boost headers out of own headers 3/3
funds: 19% (10 XMR)
done:
status: unfinished
- name: Forward declarations of own classes + interfaces
funds: 15% (8 XMR)
done:
status: unfinished
- name: One class per header
funds: 4% (2 XMR)
done:
status: unfinished
- name: Parallel tests (ctest -jN)
funds: 9% (5 XMR)
done:
status: unfinished
- name: Static methods of the wallet2
funds: 8% (4 XMR)
done:
status: unfinished
- name: Proper ordering of headers (general last)
funds: 6% (3 XMR)
done:
status: unfinished
payouts:
- date:
amount:
- date:
amount:
- date:
amount:
- date:
amount:
- date:
amount:
- date:
amount:What
The proposal is about minimizing the compilation time of the project.
Who
I have 12 years of object oriented programming experience, mostly in C++. I'm a passionate programmer, not only somebody who does this for money. I hold a M.Sc. degree in Computer Science. Although I'd prefer staying anonymous, what I can tell about myself, is that I used to actively develop a couple of space flight navigation instruments in C++ for a realistic freeware space flight simulator. Nowadays I work on an automated trading and backtesting platform, also in C++, where speed matters. Being able to see the code in a hierarchical order, both projects allowed me to create an extensive library, ready to be reused in a project like Monero, with serialization being my first low hanging fruit. To pay the rent, I've worked in various fields, from automatic control, GIS apps, navigation systems, visual driving assistance and recently in autonomous driving. My contributions to Monero so far are the following:
- I was able to bring ccache to the project. The amount of code committed is not large, but the effect size is. The Travis CI compilation time went from 22 minutes down to 2 minutes for each build.
- afterwards, selsta picked it up and brought ccache to the CI "workflows", achieving similar results.
- upon a request by vtnerd, I made the ccache optional.
Why
During all these years I have noticed how important it is to have a quickly compilable code base, which would otherwise put a negative psychological pressure on developers, making them refrain from changing anything for the better, not to mention the obvious reduction of required computational resources. For Monero specifically, I have set up the following experiment: I have calculated the sizes of header files, summing up the sub headers included by each of them (column 3). Then I have calculated how many times a given header is included by .cpp files (column 4), thus indicating both the approximate compilation time of the header and how many .cpp files would be affected by the change if the header (column 2) and sorted ascending by this value. Below is the list of the greatest 90% of the headers, using this convention:
11% = 10495 = 2099 * 5: cryptonote_boost_serialization.h
12% = 11907 = 1323 * 9: wallet2_api.h
13% = 12393 = 4131 * 3: cryptonote_core.h
13% = 12924 = 718 * 18: crypto.h
16% = 14856 = 4952 * 3: core_rpc_server.h
17% = 15990 = 1599 * 10: rctTypes.h
17% = 16500 = 3300 * 5: blockchain_db.h
26% = 24225 = 8075 * 3: blockchain.h
30% = 27979 = 3997 * 7: core_rpc_server_commands_defs.h
61% = 56616 = 2696 * 21: cryptonote_basic.h
100% = 92620 = 9262 * 10: wallet2.hIt becomes obvious, that the wallet2.h is the largest "hot spot" of the whole project. While compilation of the project took 30 minutes, touching the wallet2.h and recompiling took entire 6 minutes = one fifth.
Milestones
What can be done with this is creating as many wrappers of the boost library as possible and putting as much as implementation code into .cpp files, instead of insisting on writing them inline, when these spots aren't bottlenecks. Putting a trivial method as inline may help, but only when it's called very very frequently, and only if that improvement is a large percentage of other parts of the software, which it usually isn't. Inlining has to be proven by profiling the software, and not being a default policy, since it brings nothing, while costing a lot, not only because multiple recompiles of the code in .cpp files in one session, but recompiles upon changes of the inlined implementation.
I'd like to earn something like 40/h. It's hard now to assess how much time it will take, so I'm not strict on the concrete values. If I happen to finish ahead of time, thus becoming overpaid, I will admit it. I will be writing the time of work in each of my PRs. By assessing the payments, I will now assume a price of XMR of 125.
Milestone 1: ccache for CMake (demo)
Done with quite a success. The Travis CI was relieved and you, as a developer benefit each time you switch a branch. Previous text: "I'd like the CMake script to automatically pick ccache and clcache, when it can find them in the PATH. This piece of software helps in reducing the compilation time of compilation units (.cpp files and all their included headers), when their content hasn't changed. This means, that the more forward declarations and fewer included headers your headers have, the more the ccache will be able to leverage your discipline. This is especially useful when switching between branches."
Milestone 2: Dynamic linkage
Static libraries tend to grow in sizes exponentially and slow down the generation of the final binaries. I would like to enable (opt-in) dynamic linkage in CMake for development purposes. Also whenever you are done writing a new test and would like to just modify the production code and just execute the test, the test binary can be made so, that it doesn't have to relink upon change of the production code's resulting shared library. This is quite a low hanging fruit. There are 70 CMakeLists.txt, so multiplying each one by 2 minutes gives 2.33h plus 0.30h for creating some framework for further such changes gives 2.83h, and that equals to 0.9 XMR.
Milestone 3: Automated reports of ClangBuildAnalyser and iwyy
Some advanced tools can be employed, that help in dynamic assessment of the quality of the code from many perspectives. For my purpose, I could use (ClangBuildAnalyzer)[https://github.com/aras-p/ClangBuildAnalyzer], which gives an objective truth about which parts of the code take longest time to compile. There's also CLang-based (Include What You Use)[https://include-what-you-use.org/] tool, which not only gives advice how to optimize the bottlenecks, but also tries to do it automatically (however it's better to use it just as a hint). And last but not least, there exist tools, which help finding dangerous constructs in the code, which could lead to segmentation faults at runtime. I'd like to use my low powered PCs to generate a daily report of the CLang tools and publish them to a dedicated webpage, that I'd manage. I will of course contribute the scripts, that generate the reports into the Monero source tree. Setting up the tools will take some time.