From 78f19d4ab9d29ffc5a66b367046514bd91952462 Mon Sep 17 00:00:00 2001 From: "Ciro S. Costa" <utxobr@protonmail.com> Date: Mon, 3 May 2021 10:19:12 -0400 Subject: [PATCH] add proposal for monero kubernetes operator Signed-off-by: Ciro S. Costa <utxobr@protonmail.com> --- utxobr-monero-k8s-operator.md | 216 ++++++++++++++++++++++++++++++++++ 1 file changed, 216 insertions(+) create mode 100644 utxobr-monero-k8s-operator.md diff --git a/utxobr-monero-k8s-operator.md b/utxobr-monero-k8s-operator.md new file mode 100644 index 00000000..b17a1368 --- /dev/null +++ b/utxobr-monero-k8s-operator.md @@ -0,0 +1,216 @@ +--- +layout: fr +title: Monero Kubernetes Operator +author: Ciro S. Costa (utxobr) +date: May 3, 2021 +amount: 22.86 +milestones: + - name: Proof of concept + funds: 0 + done: 02 May 2021 + status: finished + - name: Prototype refactoring, installation improvements and docs + funds: 2.47 + done: + status: unfinished + - name: Support annonimity networks + funds: 3.71 + done: + status: unfinished + - name: Improve observability of nodes + funds: 3.71 + done: + status: unfinished +payouts: + - date: + amount: + - date: + amount: + - date: + amount: +--- + + +## Brief Intro + +My name is Ciro S. Costa (https://github.com/cirocosta, +https://twitter.com/utxobr), I'm currently a staff engineer, having previously +being a core contributor to https://concourse-ci.org. + +Monero-wise, I've been mostly focused on the networking side of it, having +implemented the basics of Levin's handshake in Go +(https://github.com/cirocosta/go-monero) with full support for the +Portablestorage format, which lets me create some interesting reports on node +distribution (see https://twitter.com/utxobr/status/1386458317405540360) by +crawling the P2P network. + + +## Problem + +_**tl;dr**: there's no good solution for running a large number of monero +nodes_ + +For those with more than a machine or two to run Monero nodes (or even miners), +there's not a good solution out there for having those up and running in an +easy to upgrade fashion. + +It's great that folks like Seth provide wonderful guides on how to run Monero +nodes (see https://sethsimmons.me/guides/run-a-monero-node-advanced/), and that +within the functional tests in the codebase we can tell how to run regtest, but +none of that helps with running a larger-scale setup. + + +## Proposal + +_**tl;dr**: extend the Kubernetes API via its common extension system to provide +semantics that make deploying clusters of monero nodes or miners with ease. See +proof of concept at https://github.com/cirocosta/monero-operator_ + +Kubernetes (see [what is kubernetes]) provides us with this vendor-neutral API +for expressing what the desired state should be, and then behind the scenes, +having that state achieved (and maintained) through the use of small +programs whose whole job is to deal with going from current state to desired state. + +Aside from being offered by pretty much every cloud provider (and many VPS +offerings out there too) and still remaining not vendor-specific, its API is +open for extension, which we can leverage to provide extra functionality that +it didn't have before. + +By extending the Kubernetes API via the use of [Custom Resources], we're able +to provide a new semantics for the users of those clusters so that we simplify +*a lot* running, say a few Monero nodes all configured the same across +different machines + + +```yaml +kind: MoneroNodeSet +apiVersion: utxo.com.br/v1alpha1 +metadata: + name: nodes +spec: + replicas: 3 + hardAntiAffinity: true + monerod: + image: utxobr/monerod:v0.17.2.0 # if testing a release candidate, then + args: # just bump the image and the operator + - --public # will take care of rolling out, preserving + - --enable-dns-blocklist # the data already synced. + - --enforce-dns-checkpointing + - --out-peers=1024 + - --in-peers=1024 + - --limit-rate=128000 +``` + +which could be very useful for businesses like CakeWallet that run sets of full +nodes (or literally anyone wanting to run highly-available monerod +deployments), but it can be also useful for folks doing research like me, +wanting to roll out a regtest network with many peers: + +```yaml +kind: MoneroNetwork +apiVersion: utxo.com.br/v1alpha1 +metadata: + name: regtest +spec: + replicas: 20 + + template: + spec: + monerod: + args: # each replica has these args + - --regtest # plus `--add-exclusive-node` + - --fixed-difficulty=1 # pointing just at the other + # peers, forming a closed net +``` + +_(^ which under the hood gets materialized in the form of `monerod` instances +pointing one at each other, with volumes attached and everything you'd want for +a real setup.)_ + +Naturally, we can do the same for miners, for instance, we can get to run 10 +replicas of `xmrig` against a pool like so: + +```yaml +kind: MoneroMiningNodeSet +apiVersion: utxo.com.br/v1alpha1 +metadata: + name: miners +spec: + replicas: 10 + hardAntiAffinity: true + + xmrig: + args: + - -o + - cryptonote.social:5556 + - -u + - 891B5keCnwXN14hA9FoAzGFtaWmcuLjTDT5aRTp65juBLkbNpEhLNfgcBn6aWdGuBqBnSThqMPsGRjWVQadCrhoAT6CnSL3.node-$(id) + - --tls +``` + +and then, if we regret chosing that pool, all it takes is patching the object +and under the hood, our extension to Kubernetes takes care of rolling the +updates out. + +_(aside: couple this with [horizontal pod autoscaler (HPA)] and you don't even +need to pre-provision any underlying machines - if your provider supports HPA - +as by making use of proper resource reservation, asking for extra replicas +would trigger the creation of new machines)._ + + +[what is kubernetes]: https://kubernetes.io/docs/concepts/overview/what-is-kubernetes +[Custom Resources]: https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources +[horizontal pod autoscaler (HPA)]: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ +[OpenMetrics]: https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md +[Prometheus]: https://prometheus.io/ + + +## The scope + +I currently have a working proof of concept +(https://github.com/cirocosta/monero-operator) that implements those three +custom resources mentioned above (`MoneroMiningNodeSet`, `MoneroNodeSet`, and +`MoneroNetwork`). + +This CCS would cover: + +1. boosting the confidence in the codebase by providing more tests to cover + edge cases glanced over while building the prototype, as well as improving + installation and documentation as a whole + +2. adding support for Tor and I2P so that nodes and networks can be deployed on + annonimity networks with a line or two in the yaml while still running the + services with high availability + +3. improving the observability of the deployed `monerod` instances introducing a + sidecar to expose `monerod` metrics for any [OpenMetrics] consumer (like + [Prometheus]) + + +As a result, the community will end up with: + +- a Kubernetes extension that lets anyone deploy highly-available `monerod` + (and miners) on any Kubernetes-enabled platform + +- a Go package that they can rely on for interacting with `monerod` + + +## The structure, milestones, and price. + +Working on this during my personal hours, I plan to do the work a few hours a +day on the side (with a few healthy periods of break) until completion. + +The proposal is structured to be paid along with the delivery of the three points above: + +1. confidence in the codebase + installation/doc guides: ~10Hr +2. support for Tor and I2P for full nodes and whole networks: ~15Hr +3. observability of `monerod`: ~15Hr + +Assuming a rate of 100$/hr and a current rate of 404 USD/xmr (May 3rd, 2021): + +| deliverable | hours | usd | xmr | +|-----|------|-----|-----| +| 1 | 10 | $ 1000 | XMR 2.47 | +| 2 | 15 | $ 1500 | XMR 3.71 | +| 3 | 15 | $ 1500 | XMR 3.71 | + -- GitLab