Skip to content
Snippets Groups Projects

add proposal for monero kubernetes operator

Merged add proposal for monero kubernetes operator
cirocosta/ccs-proposals:monero-k8s-operator into master
1 unresolved thread
Merged utxobr requested to merge cirocosta/ccs-proposals:monero-k8s-operator into master
1 unresolved thread
Viewing commit 78f19d4a
Show latest version
1 file
+ 216
− 0
Compare changes
  • Side-by-side
  • Inline
utxobr-monero-k8s-operator.md 0 → 100644
+ 216
− 0
 
---
 
layout: fr
 
title: Monero Kubernetes Operator
 
author: Ciro S. Costa (utxobr)
 
date: May 3, 2021
 
amount: 22.86
 
milestones:
 
- name: Proof of concept
 
funds: 0
 
done: 02 May 2021
 
status: finished
 
- name: Prototype refactoring, installation improvements and docs
 
funds: 2.47
 
done:
 
status: unfinished
 
- name: Support annonimity networks
 
funds: 3.71
 
done:
 
status: unfinished
 
- name: Improve observability of nodes
 
funds: 3.71
 
done:
 
status: unfinished
 
payouts:
 
- date:
 
amount:
 
- date:
 
amount:
 
- date:
 
amount:
 
---
 
 
 
## Brief Intro
 
 
My name is Ciro S. Costa (https://github.com/cirocosta,
 
https://twitter.com/utxobr), I'm currently a staff engineer, having previously
 
being a core contributor to https://concourse-ci.org.
 
 
Monero-wise, I've been mostly focused on the networking side of it, having
 
implemented the basics of Levin's handshake in Go
 
(https://github.com/cirocosta/go-monero) with full support for the
 
Portablestorage format, which lets me create some interesting reports on node
 
distribution (see https://twitter.com/utxobr/status/1386458317405540360) by
 
crawling the P2P network.
 
 
 
## Problem
 
 
_**tl;dr**: there's no good solution for running a large number of monero
 
nodes_
 
 
For those with more than a machine or two to run Monero nodes (or even miners),
 
there's not a good solution out there for having those up and running in an
 
easy to upgrade fashion.
 
 
It's great that folks like Seth provide wonderful guides on how to run Monero
 
nodes (see https://sethsimmons.me/guides/run-a-monero-node-advanced/), and that
 
within the functional tests in the codebase we can tell how to run regtest, but
 
none of that helps with running a larger-scale setup.
 
 
 
## Proposal
 
 
_**tl;dr**: extend the Kubernetes API via its common extension system to provide
 
semantics that make deploying clusters of monero nodes or miners with ease. See
 
proof of concept at https://github.com/cirocosta/monero-operator_
 
 
Kubernetes (see [what is kubernetes]) provides us with this vendor-neutral API
 
for expressing what the desired state should be, and then behind the scenes,
 
having that state achieved (and maintained) through the use of small
 
programs whose whole job is to deal with going from current state to desired state.
 
 
Aside from being offered by pretty much every cloud provider (and many VPS
 
offerings out there too) and still remaining not vendor-specific, its API is
 
open for extension, which we can leverage to provide extra functionality that
 
it didn't have before.
 
 
By extending the Kubernetes API via the use of [Custom Resources], we're able
 
to provide a new semantics for the users of those clusters so that we simplify
 
*a lot* running, say a few Monero nodes all configured the same across
 
different machines
 
 
 
```yaml
 
kind: MoneroNodeSet
 
apiVersion: utxo.com.br/v1alpha1
 
metadata:
 
name: nodes
 
spec:
 
replicas: 3
 
hardAntiAffinity: true
 
monerod:
 
image: utxobr/monerod:v0.17.2.0 # if testing a release candidate, then
 
args: # just bump the image and the operator
 
- --public # will take care of rolling out, preserving
 
- --enable-dns-blocklist # the data already synced.
 
- --enforce-dns-checkpointing
 
- --out-peers=1024
 
- --in-peers=1024
 
- --limit-rate=128000
 
```
 
 
which could be very useful for businesses like CakeWallet that run sets of full
 
nodes (or literally anyone wanting to run highly-available monerod
 
deployments), but it can be also useful for folks doing research like me,
 
wanting to roll out a regtest network with many peers:
 
 
```yaml
 
kind: MoneroNetwork
 
apiVersion: utxo.com.br/v1alpha1
 
metadata:
 
name: regtest
 
spec:
 
replicas: 20
 
 
template:
 
spec:
 
monerod:
 
args: # each replica has these args
 
- --regtest # plus `--add-exclusive-node`
 
- --fixed-difficulty=1 # pointing just at the other
 
# peers, forming a closed net
 
```
 
 
_(^ which under the hood gets materialized in the form of `monerod` instances
 
pointing one at each other, with volumes attached and everything you'd want for
 
a real setup.)_
 
 
Naturally, we can do the same for miners, for instance, we can get to run 10
 
replicas of `xmrig` against a pool like so:
 
 
```yaml
 
kind: MoneroMiningNodeSet
 
apiVersion: utxo.com.br/v1alpha1
 
metadata:
 
name: miners
 
spec:
 
replicas: 10
 
hardAntiAffinity: true
 
 
xmrig:
 
args:
 
- -o
 
- cryptonote.social:5556
 
- -u
 
- 891B5keCnwXN14hA9FoAzGFtaWmcuLjTDT5aRTp65juBLkbNpEhLNfgcBn6aWdGuBqBnSThqMPsGRjWVQadCrhoAT6CnSL3.node-$(id)
 
- --tls
 
```
 
 
and then, if we regret chosing that pool, all it takes is patching the object
 
and under the hood, our extension to Kubernetes takes care of rolling the
 
updates out.
 
 
_(aside: couple this with [horizontal pod autoscaler (HPA)] and you don't even
 
need to pre-provision any underlying machines - if your provider supports HPA -
 
as by making use of proper resource reservation, asking for extra replicas
 
would trigger the creation of new machines)._
 
 
 
[what is kubernetes]: https://kubernetes.io/docs/concepts/overview/what-is-kubernetes
 
[Custom Resources]: https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources
 
[horizontal pod autoscaler (HPA)]: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
 
[OpenMetrics]: https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md
 
[Prometheus]: https://prometheus.io/
 
 
 
## The scope
 
 
I currently have a working proof of concept
 
(https://github.com/cirocosta/monero-operator) that implements those three
 
custom resources mentioned above (`MoneroMiningNodeSet`, `MoneroNodeSet`, and
 
`MoneroNetwork`).
 
 
This CCS would cover:
 
 
1. boosting the confidence in the codebase by providing more tests to cover
 
edge cases glanced over while building the prototype, as well as improving
 
installation and documentation as a whole
 
 
2. adding support for Tor and I2P so that nodes and networks can be deployed on
 
annonimity networks with a line or two in the yaml while still running the
 
services with high availability
 
 
3. improving the observability of the deployed `monerod` instances introducing a
 
sidecar to expose `monerod` metrics for any [OpenMetrics] consumer (like
 
[Prometheus])
 
 
 
As a result, the community will end up with:
 
 
- a Kubernetes extension that lets anyone deploy highly-available `monerod`
 
(and miners) on any Kubernetes-enabled platform
 
 
- a Go package that they can rely on for interacting with `monerod`
 
 
 
## The structure, milestones, and price.
 
 
Working on this during my personal hours, I plan to do the work a few hours a
 
day on the side (with a few healthy periods of break) until completion.
 
 
The proposal is structured to be paid along with the delivery of the three points above:
 
 
1. confidence in the codebase + installation/doc guides: ~10Hr
 
2. support for Tor and I2P for full nodes and whole networks: ~15Hr
 
3. observability of `monerod`: ~15Hr
 
 
Assuming a rate of 100$/hr and a current rate of 404 USD/xmr (May 3rd, 2021):
 
 
| deliverable | hours | usd | xmr |
 
|-----|------|-----|-----|
 
| 1 | 10 | $ 1000 | XMR 2.47 |
 
| 2 | 15 | $ 1500 | XMR 3.71 |
 
| 3 | 15 | $ 1500 | XMR 3.71 |