layout: fr title: Monero Kubernetes Operator author: Ciro S. Costa (utxobr) date: May 3, 2021 amount: 9.89 milestones: - name: Proof of concept funds: 0 done: 02 May 2021 status: finished - name: Prototype refactoring, installation improvements and docs funds: 2.47 done: status: unfinished - name: Support annonimity networks funds: 3.71 done: status: unfinished - name: Improve observability of nodes funds: 3.71 done: status: unfinished payouts: - date: amount: - date: amount: - date: amount:
My name is Ciro S. Costa (https://github.com/cirocosta, https://twitter.com/utxobr), I'm currently a staff engineer, having previously being a core contributor to https://concourse-ci.org.
Monero-wise, I've been mostly focused on the networking side of it, having implemented the basics of Levin's handshake in Go (https://github.com/cirocosta/go-monero) with full support for the Portablestorage format, which lets me create some interesting reports on node distribution (see https://twitter.com/utxobr/status/1386458317405540360) by crawling the P2P network.
tl;dr: there's no good solution for running a large number of monero nodes
For those with more than a machine or two to run Monero nodes (or even miners), there's not a good solution out there for having those up and running in an easy to upgrade fashion.
It's great that folks like Seth provide wonderful guides on how to run Monero nodes (see https://sethsimmons.me/guides/run-a-monero-node-advanced/), and that within the functional tests in the codebase we can tell how to run regtest, but none of that helps with running a larger-scale setup.
tl;dr: extend the Kubernetes API via its common extension system to provide semantics that make deploying clusters of monero nodes or miners with ease. See proof of concept at https://github.com/cirocosta/monero-operator
Kubernetes (see what is kubernetes) provides us with this vendor-neutral API for expressing what the desired state should be, and then behind the scenes, having that state achieved (and maintained) through the use of small programs whose whole job is to deal with going from current state to desired state.
Aside from being offered by pretty much every cloud provider (and many VPS offerings out there too) and still remaining not vendor-specific, its API is open for extension, which we can leverage to provide extra functionality that it didn't have before.
By extending the Kubernetes API via the use of Custom Resources, we're able to provide a new semantics for the users of those clusters so that we simplify a lot running, say a few Monero nodes all configured the same across different machines
kind: MoneroNodeSet apiVersion: utxo.com.br/v1alpha1 metadata: name: nodes spec: replicas: 3 hardAntiAffinity: true monerod: image: utxobr/monerod:v0.17.2.0 # if testing a release candidate, then args: # just bump the image and the operator - --public # will take care of rolling out, preserving - --enable-dns-blocklist # the data already synced. - --enforce-dns-checkpointing - --out-peers=1024 - --in-peers=1024 - --limit-rate=128000
which could be very useful for businesses like CakeWallet that run sets of full nodes (or literally anyone wanting to run highly-available monerod deployments), but it can be also useful for folks doing research like me, wanting to roll out a regtest network with many peers:
kind: MoneroNetwork apiVersion: utxo.com.br/v1alpha1 metadata: name: regtest spec: replicas: 20 template: spec: monerod: args: # each replica has these args - --regtest # plus `--add-exclusive-node` - --fixed-difficulty=1 # pointing just at the other # peers, forming a closed net
(^ which under the hood gets materialized in the form of
pointing one at each other, with volumes attached and everything you'd want for
a real setup.)
Naturally, we can do the same for miners, for instance, we can get to run 10
xmrig against a pool like so:
kind: MoneroMiningNodeSet apiVersion: utxo.com.br/v1alpha1 metadata: name: miners spec: replicas: 10 hardAntiAffinity: true xmrig: args: - -o - cryptonote.social:5556 - -u - 891B5keCnwXN14hA9FoAzGFtaWmcuLjTDT5aRTp65juBLkbNpEhLNfgcBn6aWdGuBqBnSThqMPsGRjWVQadCrhoAT6CnSL3.node-$(id) - --tls
and then, if we regret chosing that pool, all it takes is patching the object and under the hood, our extension to Kubernetes takes care of rolling the updates out.
(aside: couple this with horizontal pod autoscaler (HPA) and you don't even need to pre-provision any underlying machines - if your provider supports HPA - as by making use of proper resource reservation, asking for extra replicas would trigger the creation of new machines).
I currently have a working proof of concept
(https://github.com/cirocosta/monero-operator) that implements those three
custom resources mentioned above (
This CCS would cover:
boosting the confidence in the codebase by providing more tests to cover edge cases glanced over while building the prototype, as well as improving installation and documentation as a whole
adding support for Tor and I2P so that nodes and networks can be deployed on annonimity networks with a line or two in the yaml while still running the services with high availability
improving the observability of the deployed
monerodinstances introducing a sidecar to expose
monerodmetrics for any OpenMetrics consumer (like Prometheus)
As a result, the community will end up with:
a Kubernetes extension that lets anyone deploy highly-available
monerod(and miners) on any Kubernetes-enabled platform
a Go package that they can rely on for interacting with
The structure, milestones, and price.
Working on this during my personal hours, I plan to do the work a few hours a day on the side (with a few healthy periods of break) until completion.
The proposal is structured to be paid along with the delivery of the three points above:
- confidence in the codebase + installation/doc guides: ~10Hr
- support for Tor and I2P for full nodes and whole networks: ~15Hr
- observability of
Assuming a rate of 61.75$/hr and a current rate of 250 USD/xmr (June 1st, 2021):
|1||10||$ 617.5||XMR 2.47|
|2||15||$ 927.5||XMR 3.71|
|3||15||$ 927.5||XMR 3.71|