layout: wip
title: Compilation time reduction and housekeeping
author: mj
date: April 15, 2020
amount: 52.9
milestones:
- name: ccache for CMake (demo)
funds: 0 XMR
done: 31 December 2020
status: finished
- name: Dynamic linkage
funds: 2% (0.9 XMR)
done: 31 December 2020
status: finished
- name: Automated reports of ClangBuildAnalyser and iwyy
funds: 4% (2 XMR)
done: 31 December 2020
status: finished
- name: Automated reports of Valgrind (test bottlenecks)
funds: 2% (1 XMR)
done: 20 May 2021
status: finished
- name: Optional precompiled headers for CMake 3.16
funds: 4% (2 XMR)
done: 20 May 2021
status: unfinished
- name: Forward declarations of own classes + interfaces
funds: 15% (8 XMR)
done:
status: unfinished
- name: One class per header
funds: 4% (2 XMR)
done:
status: unfinished
- name: Parallel tests (ctest -jN)
funds: 9% (5 XMR)
done: 20 May 2021
status: finished
- name: Static methods of the wallet2
funds: 8% (4 XMR)
done:
status: unfinished
- name: Proper ordering of headers (general last)
funds: 6% (3 XMR)
done:
status: unfinished
- name: Miscellaneous hourly work @ $40/hr (23.4 XMR remaining)
funds: 47% (25 XMR)
done:
status: unfinished
payouts:
- date: 4 January 2021
amount: 2.9
- date: 17 June 2021
amount: 9.6
- date:
amount:
- date:
amount:
- date:
amount:
- date:
amount:What
The proposal is about minimizing the compilation time of the project.
Who
I have 12 years of object oriented programming experience, mostly in C++. I'm a passionate programmer, not only somebody who does this for money. I hold a M.Sc. degree in Computer Science. Although I'd prefer staying anonymous, what I can tell about myself, is that I used to actively develop a couple of space flight navigation instruments in C++ for a realistic freeware space flight simulator. Nowadays I work on an automated trading and backtesting platform, also in C++, where speed matters. Being able to see the code in a hierarchical order, both projects allowed me to create an extensive library, ready to be reused in a project like Monero, with serialization being my first low hanging fruit. To pay the rent, I've worked in various fields, from automatic control, GIS apps, navigation systems, visual driving assistance and recently in autonomous driving. My contributions to Monero so far are the following:
- I was able to bring ccache to the project. The amount of code committed is not large, but the effect size is. The Travis CI compilation time went from 22 minutes down to 2 minutes for each build.
- afterwards, selsta picked it up and brought ccache to the CI "workflows", achieving similar results.
- upon a request by vtnerd, I made the ccache optional.
Why
During all these years I have noticed how important it is to have a quickly compilable code base, which would otherwise put a negative psychological pressure on developers, making them refrain from changing anything for the better, not to mention the obvious reduction of required computational resources. For Monero specifically, I have set up the following experiment: I have calculated the sizes of header files, summing up the sub headers included by each of them (column 3). Then I have calculated how many times a given header is included by .cpp files (column 4), thus indicating both the approximate compilation time of the header and how many .cpp files would be affected by the change if the header (column 2) and sorted ascending by this value. Below is the list of the greatest 90% of the headers, using this convention:
11% = 10495 = 2099 * 5: cryptonote_boost_serialization.h
12% = 11907 = 1323 * 9: wallet2_api.h
13% = 12393 = 4131 * 3: cryptonote_core.h
13% = 12924 = 718 * 18: crypto.h
16% = 14856 = 4952 * 3: core_rpc_server.h
17% = 15990 = 1599 * 10: rctTypes.h
17% = 16500 = 3300 * 5: blockchain_db.h
26% = 24225 = 8075 * 3: blockchain.h
30% = 27979 = 3997 * 7: core_rpc_server_commands_defs.h
61% = 56616 = 2696 * 21: cryptonote_basic.h
100% = 92620 = 9262 * 10: wallet2.hIt becomes obvious, that the wallet2.h is the largest "hot spot" of the whole project. While compilation of the project took 30 minutes, touching the wallet2.h and recompiling took entire 6 minutes = one fifth.