Research Computing Upgrade

Gingeropolous CCS Request

Hello all! This CCS proposal is to fund me to work part time on monero in various capacities, fund some monero research computing infrastructure, and to support my official training in Computer Science. I am a long time member of the monero community, and I've always wanted to dedicate more time to Monero. I am in a place professionally where I can potentially pivot to become "full time monero" if support is available. I have a lot of projects that have interested me over the years. This first CCS is to increase the storage capacity of a particular machine, as Rucknium is embarking on a project that needs a lot of fast storage. In addition, I also requests funds for my time. In the future, I plan on submitting additional CCS requests to further improve computational and storage infrastructure for the monero research and development community, perform various monero R&D, and begin my official computer science education / training.

Who:

I am Gingeropolous, long time community member and tinkerer. Over the years I've worked on various things in Monero. Notable things include webmin for xmrchain.net, testing fluffyblocks on mainnet (https://www.reddit.com/r/Monero/comments/6dords/help_test_fluffyblocks_compact_blocks_whatever/), and being sometimes the only mining pool for testing PoW variants during the ASIC war (remember killallasics.moneroworld.com ?). For some background, I do hold a PhD in a field of biomedical science, and I am/was a Research Assistant Professor. I came into monero years ago with no cryptocurrency, linux, or software development experience - but I've been building and working with computers for decades. I've had a lot of help from many monero contributors, and I feel i've reached a point of competance that I hope is valuable to the project. This CCS will hopefully be my first of many. During the dedicated and protected time (that I will request in the future), I hope to become even more valuable to the project through various training, infrastructure, and research activities. I have already begun my path to getting a Masters in Computer Science by working on the pre-requisite courses, as I have nothing on paper that indicates that I will perform well in a CS degree.

What:

Infrastructure: high end servers and tools for monero development and research

Specific to this CCS: Funds to increase storage on existing hardware to fascilitate current research

I would like to build, maintain and administrate high-end servers for monero development and research in a cost effective manner. From my own experience in monero, it's ideal if you can just do things faster. Build something, test it / run it, tweak something, do it again. Time spent waiting for the computer to do something means you lose time and momentum - and this momentum, and time in general, is scarce for open source contributors that mostly contribute in their free time. Thus, I plan on providing high end servers to the monero developer and researcher community. Lots of ram, virtually unlimitted storage, and threads everywhere.

I've already done this at a small scale with my own funds, providing access to my existing mining hardware (with some additional memory and storage upgrades just for monero R&D) to researchers like Sarang, Rucknium, and ack-j; members of the Noncesence Research Lab, and developers like TheCharlatan, mj-xmr, jberman. For example, ack-j recently told me "Hey so that the multiprocessing worked great yesterday and saved an estimated 100 days compared to running it on my VM with 1 thread." Renting this kind of hardware from a hosting provider can get expensive. Furthermore, CCS investments in this infrastructure represesent one-time investments. Once it is built, I will provide access to the monero R&D community as long as I can keep the lights on (with or without community funding). The end goal of this effort (built over the years) will be a veritable research computing cluster with job scheduling etc (if people want that - sometimes direct shell is just easier). For the time being though, it will be much simpler - custom VM provisioning (if needed) and dedicated metal. Additionally, I will provide i2p and/or onion connections to allow any member of the community to use the resources in a private manner.

Granted, the building of this infrastructure creates an obviously centralized thing, and I'm not a huge fan of that. For the time being, I think it will be OK, but part of my future effort will include a way by which other monero folks rich in computer resources (e.g., those with big monero mining ops) can add their computers to a global monero research computational resource. To this end, a future project will include scripts that allow for anyone to launch their own Monero Research and Development cluster. For instance, a way for someone to build their own databases etc.

Existing infrastructure in the Monero Research Computing garage:

HPE Proliant DL325 Gen10 AMD EPYC 7402P 48 thread, 64 GB RAM ASUS Zenith 2 Alpha with 64 thread Threadripper 3970x , 256 GB ram, 2TiB NVME, 24TiB HDD. (I maxed out the RAM on this one) some sad old opterons with 96 GB ram ~6 3900x AMD systems with 16-64gb ram and various storage abilities. 1 1920x threadripper with 32gb ram old laptops so we can run performance tests on scrap

UPS:

N1C L-series 3000VA

Network: 1 gbps up/down with unlimitted bandwidth 1 gbps 48 port network switch

Funds requested in this CCS:

A. 5 * 4096 Gb NVME SSDs to max out the high-speed storage capability of the Zenith 2 Alpha. Market prices have these around $1k. Currently considering Kingston KC3000 M.2

B. 3 months of part time salary support and partial costs: this will allow me to spend more time with the hardware (for instance, that new UPS isn't installed yet), focus on computer science pre-reqs, and assist with electrical costs. During this time I will also assist with whatever Monero things I can assist with - usually these involve testing software and trying to find bugs. I will also put more time into helping folks on Reddit, IRC, and Matrix. I am requesting $1500 a month.

Timeline / Milestones:

Storage: I will purchase the drives upon full funding of the request, and request a payout once I provide a screenshot of the drives mounted. Monthly: I will provide an update (roughly a month) when I feel I have completed enough to warrant a payout.

I have calculated the XMR based on the USD fiat price of $165. $185 (roughly using the daily 21 ema as of commit)

I am open to any suggestions for this proposal.

Expiration 2022-12-01 (YYYY-MM-DD). If not all claimed, released to the general fund.

51 xmr total, 27 for SSDs, 8x3 for monthly support

I have calculated the XMR based on the USD fiat price of $165.

Have you added a total XMR amount that needs to be raised? I can't see one, unless I'm overlooking something.

the reddit thread

https://old.reddit.com/r/Monero/comments/un0wqi/my_first_ccs_request_more_storage_for_monero/

Just need to leave a space after the "#" then the titles will display fine e.g. "# Who:" "## Infrastructure"

added 1 commit

• 98c4c7af - Update gingeropolous_zenith_storage.md for markdown and include xmr in text

Compare with previous version

I think Gingeropolous is a valuable member and his proposal seems reasonable and needed. I would like to ask if it would be possible also for me to use these resources? At the moment I'm using my old laptop to make a bunch of elliptic curve point multiplications and would be great if the Monero Community assisted the guys who are doing some kind of work at the MRL. Definitely I think it could improve the wellness of the MRL researchers and boost the developing environment :)

@dangerousfreedom , yeah u can get an account. U have a CCS funded, so you sorta passed the bar of entry.

Gingeropolous's existing server has supported many Monero research and development projects:

• The "Fingerprinting a Flood" investigation that was the first documentation of a flood incident "in the wild" on the Monero blockchain, which was theorized as a way to weaken user privacy all the way back in Monero Research Lab Bulletin 1 (September 2014).

• Measurement of the performance of Seraphis signature size and CPU time.

• Visualizations of Monero ring member age throughout its history.

• Investigation of a possible bug in selection of duplicate ring members across rings within the same transcation.

• Ongoing work to estimate the effect of minexmr dot com's fee increase from 1% to 1.1% on minexmr's share of the total Monero hashrate.

• Other work I'm forgetting at the moment, has been mentioned in the proposal already, or I'm unaware of.

The current resources are good, but we need more. A glance at the job openings at several chain analysis companies shows that they are hiring many "data engineers" and most likely have the hardware to match.

The immediate motivation for this request for 20 Terabytes of SSD storage space is a need to compare the inter-temporal stability of the distributions of the age of spent outputs within and across blockchains. This is important because it sheds light on any risk to Monero user privacy of having a static rather than dynamic decoy selection algorithm.

Right now we are looking at some of the top transactional cryptocurrencies: BTC, BCH, LTC, DOGE, and DASH. I have a working prototype for LTC and DOGE, but the bitcoin blockchain is much larger and requires more storage space.

Blockchain data is designed to be compact. To analyze it statistically, larger data structures are needed. To take one example, Blockstream's electrs software required 610GB of storage when the BTC blockchain itself was 280GB.

My working prototype linked above requires 400GB for LTC, whose blockchain itself is 70GB. Extrapolating, BTC's 400GB blockchain would require 2.4 Terabytes of space.

This is of course a single specific project, but as we get further into empirically analyzing user behavior to ensure that Monero users do not "stick out" on the Monero blockchain, I expect that large storage resources will become increasingly necessary.

My questions to @gingeropolous are:

1. Would it be possible to use one of the SSDs for RAM swap space when there is particular need? My understanding is that using SSDs in this way can cause excessive wear on the hardware, but I wanted to check.

2. Would it be possible to clarify criteria for allowing researchers and developers access to the resources? On the other hand, maybe it is hard to codify exact criteria since it is sort of trust-based.

3. Would this CCS affect the statement that the MAGIC Monero Fund currently has on its Request for Research Proposals, which refers to your server: "Computing and data storage resources can be made available to applicants depending on research requirements, resource availability, and vetting by a third party."?

Thanks @Rucknium for the references of the prior work.

regarding your questions:

1. Yeah, we could use the SSD space for swap space. Excessive wear isn't a concern for me. These are meant to be used. (though it was a factor in deciding to do the 4TB models over the 8TB, because the 8TB don't last as long). Though ideally we'll eventually get a real research server with 2+TB ram and actual enterprise SSDs, but I wanted to take small steps because that kind of machine is like $50k and I feel like I would need full time support to get that kind of thing going. Right now i'm still contributing in my off time, which is why, for instance, my new UPS is sitting there next to the rack.

2. I'm not sure re: clarifying criteria for access. Some obvious ones have been A) have you received CCS funding and / or B) have you had multiple PRs accepted into the monero codebase. Both of those kind of overlap with "time in monero". But as you say, it is sort of trust based.

3. Re: MAGIC Monero Fund computing resource availability - I don't see how the CCS would affect that statement, unless you think there's something that might pop up?

Thanks for your answers :)

Re: MAGIC Monero Fund computing resource availability - I don't see how the CCS would affect that statement, unless you think there's something that might pop up?

No, I don't think that something will pop up. I just wanted to double-check.

changed the description

@gingeropolous please fix the amounts. They shouldn't have "xmr" in them.

I'm not a tech person but if it means Monero can be strengthened or improved via research then I support it and will donate.

Also, should expiration date be 2023 and not 2022?

added 1 commit

• 8526f293 - Update gingeropolous_zenith_storage.md, removed xmr from front matter, also...

Compare with previous version

added 1 commit

• 982e5831 - Update gingeropolous_zenith_storage.md, fixed expiration date format

Compare with previous version

changed the description

Supporting this. Already said it in a community meeting but chainalysis companies have access to supercomputers so our researchers should too.

mentioned in commit 24a801e9

merged

I wasn't looking here for some time and missed this proposal.

I fully support it, can recommend working with @gingeropolous , and am sure that this is money well spent.

First Month Update, Milestones Completed:

I have installed the 5 NVME 4TB drives into "Junior" that were funded by this CCS. During the acquisition of these drives, I found it optimal to also upgrade the HDD storage of the server. So the device has 5x4TB NVME drives in an array for scratch space, and 7x12TB HDDs + 1x4TB NVME SSD (L2ARC) + 2x256GB (SLOG) drives in a RAIDZ2 ZFS array to create a usable 71TB (hooray compression). (if anyone wants to suggest a better ZFS setup, please do).

I am thus requesting the payout for milestone "Expand data storage on existing server".

Additionally, I am requesting the "Month 1 Support" milestone payout (roughly Mid June to Mid July). During the past month, I worked on the server cluster, including racking up the UPS, getting all the new storage into "Junior", tried to debug a stalled monerod, some network troubleshooting (ended up with new 1 Gbps switch due to this), updating explorers and seed nodes, trying to rescue moneroworld.com, and python classes. Monthly support also goes towards the power bill and all the other stuff I bought for the server. I ended up needing a second SATA controller and a NVME expansion card.

some network troubleshooting (ended up with new 1 Gbps switch due to this),

I was the reporter of this issue. @gingeropolous installed the right measurement software, which allows us to independently test the network speed. I confirm that between the computers, that I have access to, the transfer rate is now at 1 Gb/s (rounded), while before his hardware modification and investigation, it was at 100 Mb/s.

my_user@zenith2alpha:~$ iperf3 -c 192.168.1.XX
Connecting to host 192.168.1.XX, port 5201
[  5] local 192.168.1.YY port 39332 connected to 192.168.1.XX port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   114 MBytes   959 Mbits/sec    0    758 KBytes       
[  5]   1.00-2.00   sec   112 MBytes   944 Mbits/sec    0    793 KBytes       
[  5]   2.00-3.00   sec   112 MBytes   944 Mbits/sec    0    834 KBytes