mj part time 2021 q4

What

In the same way as previously, I propose to work for 3 months, spending 30 hours a week on Monero Core, specifically on topics such as (in this order):

• statistical simulation (see Proposal)
• addressing user issues (whenever I can help)
• enabling and helping new developers
• code reviews
• CI fixes
• extending my Monero health report
• adding Monero-GUI to the health report
• general firefighting, whatever problems we face in near future

Why

I need to raise the stakes unfortunately, since I'm strongly considering ditching my job, which became a pain in my back. This means, that I will be more dependent on the XMR money. Also, since I live in Western Europe, I pay my bills in EUR, which is seemingly going down against USD in a long term trend. I normally hate hearing from people, who I have to hire to do things, which they can do better than me, that "they Have to earn more". I prefer saying first what additional stuff I'll deliver. Since my job will not get into the way as it did before, I will have more mental capacity to learn and read about the internals of Monero (outside of the proposed 30h/w). So far I've been doing (and gaining experience in) a lot of support tasks and I'm happy with the results as far as they can be reviewed and merged by the Maintainers. However there are now also some new challenges in Monero, which deal with statistics. I have some self-taught knowledge the field and I could provide some time series simulations, that would help in verifying if a given statistician's (or my) solution is able to yield the expected results in multiple alternative scenarios, as opposed to relying on a fixed history, which always leads to overfitting. Please refer to the wonderful work of Nassim Nicholas Taleb, especially Fooled by Randomness for more background. I already have a working software for such simulations, but it has to be adapted to fit Monero. I'd gladly reuse it for you guys, while you'd only pay for the adaptation itself. I think it's a good deal. I've been working on this software for many years for the purpose of statistical analysis of financial data, and a huge amount of them.

Some more details and summaries of other work packages are below:

Statistical simulator

Here are the most relevant features of the simulator, that I'd adapt for Monero:

• Parallel Monte-Carlo simulation of alternative scenarios. The result of such a simulation are median and standard deviation of all experiments (in other words: the expected value as well as best and worst case scenario)
• The resulting value(s) are of a unit, defined by whatever objective function that is being fed into the simulator. In my current case of the financial simulation, it's the trading profitability.
• Smooth interfacing with Python (either via Boost or TCP), as Python is the (reasonable) language of choice for Statisticians. This means, that the Statisticians will not have to immediately rewrite their code to C++, if they want to have them checked by a very fast C++ simulator, compared to an analogous task, performed by Python code.
• Very fast loading of serialized data

The current visuals of my simulator can be seen below:

Visualization

The visualization allows to take a closer look at what happens on the lowest level:

Configuration

The current configuration UI, which produces serializable configuration files:

Evaluation

A console program, which performs the evaluation on larger data in parallel and joins them, plotting the result in the console:

Reporting

An HTML report is being generated after each evaluation. Here's how it currently looks:

A compound report, like in the console evaluation:

An individual report for each component:

CI

Me and the team were able to fix the prevailing FreeBSD ccache issue. I also separated the caches based on how often they have to be recreated, which in turn saved space, so that GitHub doesn't have to purge them as often as before. All this together lead to a lot quicker reactions. I don't see a potential for improvements of the CI anymore, which makes me happy. We can work relatively efficiently now, offsetting a lot of testing onto the CI without having to wait for a long time, nor having a bad feeling about abusing the service. However, as soon as an unpredictable problem appears on the CI, I'm ready to address it.

Enabling new devs

I helped in adaptation of Monero for RPi4 in the last month. I found it encouraging to work with people, who know what they want and are able to react lively. I spent some time on introducing new devs into the project, but somehow they gave up. OTOH, many new Monero devs, who didn't give up, to say the least, often message me privately with C++ questions, that I'm always happy to answer. I'd bet a lot of XMR, that it saves them a lot of frustration and time of own research. I'll happily keep doing it all.

Health report

• I extended my Monero health report with the Memory consumption (for compilation) report lately uncovering a huge RAM consumption of source files like wallet2.cpp. See here.
• The report will be extended to cover Monero-GUI
• I will keep adding next tools into Monero codebase itself, whenever I decide, that they will have reached a production-ready state

Who

mj, I have been contributing to Monero-core since 2020 with 73 merged commits. Here is a list of my previous work:

CLI contributions: https://github.com/pulls?q=is%3Apr+author%3Amj-xmr+archived%3Afalse+is%3Amerged+

Previous reports

Here is a list of the previous reports, that describe my completed or started tasks in more detail:

• Report 01
• Report 02
• Report 03
• Report 05
• Report 06
• Report 07

Previous CCS: https://ccs.getmonero.org/proposals/mj-part-time-2021-q3.html

Proposal

In Q4, I am able to realistically spend 30 hours a week on Monero. I arranged everything so, that the Q4 and January will be easy on me, so I won't have to prolong the work (and payment) like I had to do it in Q3. I'd like to start in November.

I propose a wage of 45 €/h for 3 months. As of 20.10.2021 the XMR/EUR is at around 220 €. This would make a total of: 45 €/h * 30 h/week * 4 weeks * 3 months / 220 XMR/EUR = 73.6 XMR, rounded down to be divisible = 72 XMR

Cheers!

Expiration date

31 Jan, 2022

Report posted on MO: https://www.monero.observer/mjxmr-ccs-proposal-q4-2021/

@mj thanks for your work.

Thanks for summarizing.

I have just had a first discussion with @Rucknium about our potential cooperation, and a very fruitful one.

A short summary of the discussion:

1. Rucknium is happy about having a new testing environment
2. Rucknium writes in R, not Python, but we quickly agreed, that we can interface the R to C++ gap via HTTP/TCP or Rcpp
3. The simulator will not be limited to cooperation with Rucknium only, but rather stay flexible to enable other Researchers as well
4. The details of Rucknium's and other Statisticians' research don't need to be disclosed to me, in order to perform their researches. The tool shall be agnostic towards the input received (my personal requirement)

Ad. 4) This will be achieved in two ways:

• by providing a plugin interface, where each Collaborator can plugin their own, non-disclosed code, which the Simulator will simply agnostically call through the interface
• by writing a script, being an input to a module, that derives from the above interface as well, and delivers at least basic statistical functions. This method would not require a recompilation of the module.

The yield of point 4) will be:

• the secrets remain secret for as long as they have to
• a guarantee that the simulator is flexible enough, while still delivering the speed it advertises

Since this question would surely appear: the simulation part shall be open sourced. I have no problem with this, if I get the pay, that I declared. Especially since I can make it so, that the trading part can still remain private, while the simulator is self-sufficient without it. This is because the trading part is just a plugin.

@jberman also expressed, that he'd use my tools.

Adding to this: it does seem like the tools are well suited for certain aspects of @Rucknium 's OSPEAD from my perspective

Also very much so appreciate @mj's deep knowledge of C++ :)

Let me elaborate on why the time series simulation tool is useful for improving the Monero protocol. Time series is an inherent part of Monero. The blockchain never stops, after all. A recent piece of research used time series techniques to analyze an apparent transaction flood incident on the Monero blockchain.

So, why is simulation needed? The issue boils down to the fact that time series data is typically not independent. Most statistical theory works with the assumption that observations are independent. The goal of that statistical theory is to establish closed-form expressions for the variance, bias, convergence properties, etc. of statistical estimators.

When there is a lack of statistical independence (as is typically the case in time series settings), often a closed-form expression is out of reach. At this point, Monte Carlo simulations are a common technique to overcome the issue. @mj 's tool is basically a way to handle Monte Carlo simulations.

In @Rucknium 's proposal, he wrote:

"On September 16, I submitted a 28-page encrypted document to Monero's Vulnerability Response Process (VRP). It contained a practical statistical attack on user privacy as well as an outline of a plan to overhaul the MSA to (1) reduce the potency of the attack in the medium term through a novel technique I have named Optimal Static Parametric Estimation of Arbitrary Distributions (OSPEAD); and to (2) eventually render the attack completely inert through a nonparametric and possibly dynamic approach."

Re: point 2), this is where another independent subsystem of my simulator will come in handy: a Walk Forward Optimization. This subsystem is able to show to what extent a given model is able to adapt to a changing situation, by simulating a complete lack of future data at a given (and advancing) point of time. The adaptation is exactly the act of dynamically choosing the parameters on the fly, based only on what has been seen up to the given point of time.

I have also had a chat with @gingeropolous , who said, that we could use his network of computers for parallel simulations. The computers have all Ubuntu installed, and this is all I need to deploy my calculation nodes. The parallel computations already work very well in my system. No "adaptations for Monero" are even needed in this case.

FYI, there is a meeting in less than 24 hours which might be a good opportunity for a Q&A on your proposal: https://github.com/monero-project/meta/issues/625

Thanks for the heads up. I'll be there.

The requested Reddit article is here.

mentioned in merge request !255 (merged)

mentioned in commit 2e20b4f1

merged

Here's my monthly report on Reddit. Enjoy!

Happy New Year! Here's my December report on Reddit.

Hi

My January report is* available on Reddit.

(*) Or rather will be available after it passes the moderation.

mentioned in merge request !283 (closed)

mentioned in merge request !287 (merged)

Great article I appreciate the time and work you put into writing this insightful and fascinating essay. basketball stars

So impressive when i read your post. please, more information in the future in order to update posts like that. Idle mining empire

It's great to read this information, I have read all your posts, please share more.https://idleminingempire.com/