SolOptXMR - Solar Optimal mining of XMR
What
The goal is to create open-source software that aids people mining Monero with excess solar power in the most profitable way. It will accommodate for issues such as:
- Time of day
- Weather fluctuations
- Avoiding depleting batteries below a threshold, that would damage them
- Avoiding overheating the mining rig
- Leaving enough power for your daily use
Intro
I promised I'd help out in the Monero Recruitment Matrix Room. Although there was just one direct request to me so far, recently I've also pro-actively proposed cooperation to the chat room members. I presented there 2 projects, that are on my mind, and don't require extensive crypto knowledge, typically not available to students. Yet the projects are still useful for Monero specifically. One of the projects that I have on mind is optimization of mining performance on a solar farm, depending on varying inputs. The member Endor (endorxmr on GitHub and endor00 on Matrix) signed up for being the main contributor to the project, while I would serve as a mentor: namely coordinator, reviewer and designer. Such a setup will guarantee, that I won't get dragged away too much from the maintenance of Monero, that I signed up for, while at the same time, Endor will be able to continuously have access to my years of experience in automatic control and object oriented design. Endor on the other hand is heavily involved in everything, that’s connected to mining profitability.
Should Endor not make it until the deadline, I guarantee, that I will finalize the project by myself until the coming winter.
Introducing the project: SolOptXMR – Solar Optimal mining of XMR.
Why
The following YT video is able to bring you more context why this idea has a chance for being a trending subject and is here to stay. The outline of this video is the following: Because of high initial, as well as perpetual investments required for producing power from solar collectors in Sahara/Nevada and delivering it to northern states, such an investment, contrived in the previous decade under the name "DESERTEC", turned out to be not the best possible investment. The project's relative profitability decreased across the years even more. The reason is, that the costs of production of solar panels dropped so low, that now it makes more sense to place them exactly where the produced power is most needed, even though the solar irradiation is lesser at these places.
On a sentiment / geopolitical side of things, with the rising prices of heating gas, I expect many people will be wanting to save on their gas costs by switching to electrical heating. Since the majority of such heaters demand high power, ranging from 2kW to 2.5kW and since a given population will typically want to switch them on all at the same time, I predict that we'll be facing a decent wave of blackouts in the coming winter. This has motivated me to build my solar farm.
If you are an individual who likes to take the matters into your own hands, you might have considered building a small solar farm yourself. Although our system won't assume a given user's setup, my particular farm is configured as an isolated island. This means, that I don't sell my overproduction to the grid for the pennies, that the state offers, but rather use the production for mining and cooking, while the overproduction is temporarily stored in a pair of cheap but reliable car batteries. My farm serves as a perfect test bed for the system, that we're going to design and automate for you. Optimized mining of Monero on such a system amortizes the installation costs over the long run, will help you learn about electricity and motivate you to become more resilient. Further compatible extensions are possible, such as small wind generators as well as bikes with flywheels, that recover kinetic energy. Such systems are already available on the market, that tends to think ahead of the time, so you don’t have to DIY.
System description
The dynamics of the system, that tries to optimize the mining efficiency has to take into account the following inputs and goals. Some of them are obvious while others maybe not so much.
- Static astronomic data, that describe the relative movement of the Sun across the sky in various latitudes and times of year - yielding the maximal expected power that can be reached
- Weather predictions - a cloudy sky can reduce the power output by more than 80%
- If the predicted power generation is off the measured generation, a warning might be issued to clean the panels
- Voltage (load state) of the batteries, as well as voltage that reaches the DC->AC transformer, as some of it might be lost in the cables, that are either too long or have accidentally formed an inductor
- Predicted load state of the batteries on the next day before the Sun rises and the charging starts, as emptying the chemical batteries below 50% tends to damage them
- Mining rigs' temperature readouts as it's not worth to meltdown your hardware for those couple of Piconeros being mined.
- A given computer's hashrate per Watt - use the most efficient ones first, but don't overheat them
- A given computer's idle power cost - sometimes it will make sense to switch on certain less efficient computers only if there's really way too much power production, and/or the most efficient ones overheat, or it's predicted that they'd overheat soon under the, also predicted, future conditions.
- Per core efficiency compared against multithreaded efficiency - for CPUs with smaller caches, the efficiency (in this case: hashrate) scales poorly with more threads working at the same time. In such cases it will make sense to spread the mining across longer periods of time on a single core.
- The owners' power usage habits - taking into account, that you might want to use a low powered, compatible electric heating system at a given time of day, by leaving some energy in the batteries for you to consume, before the predicted event – learned either from your usage habits, or from your manually scheduled input.
- The owners' setup – whether an isolated solar island is used, output connected to grid (for selling) or supplementing power losses from the grid (buying) or both of the last 2 options.
The control mechanisms of the system would be the following:
- Downclocking the CPUs - very easy to achieve under Linux with a high precision, thus allowing for a smooth control
- Restarting the mining with a smaller number of cores
- Putting computers to sleep and waking them up with Wake On LAN on the next day
While it would be easy enough to control the system via PID controller, and I know a lot about it, such a system would have little prediction power, delivered only by the first differential part of the controller (the D) and only between the current and the previous state. Using a Time Series Analysis and Prediction tool, such as the already available tsqsim, will give the system the ability to look ahead into the future (like at least 48 hours ahead) and plan accordingly to maximize the gain per the unit of energy used, while at the same time planning carefully against overheat of the hardware, under-voltage of the DC->AC transformer, leading to an immediate halt of the system, as well as against under-voltage (as in: capacity) of the batteries, which would otherwise quickly damage them.
The inputs to the system will come in a few forms – either assumed, input by the user or measured. The measurements can be easily supplied via smart plugs or measured directly. Ultimately it has to be left to the user to decide how much time to spend on providing additional inputs, that result in improvements of predictions, thus profitability. The goal here is however to keep the measurements optional, not to impose burden on the user.
Who
mj – working for Monero since 2 years on the purely technical level. Speeding up compilation and Continuous Integration on GitHub, keeping an eye on overall efficiency, generating Monero health report, reviewing technical PRs, helping out new developers. Lately developing Time Series Analysis tool for Monero Research Lab, called tsqsim. Here’s a list of all my contributions to Monero and to the supporting software.
Endor – Aerospace Engineering student and a junior Python developer on the side. I already have a few small projects under my belt - one of which related to automation and monitoring of a live system. I've been following the Monero project for over 5 years, learning the ins and outs of cryptocurrency mining and moderating the Supportxmr chat and the Monero Mining room on Matrix (#xmrmine:matrix.org). I am currently developing a comprehensive economic model of mining, and I plan to publish a small paper/series of articles over the next few weeks that will (hopefully) serve as "the ultimate guide to mining" for newcomers and professionals alike. A glimpse into my work can be seen in this minimalistic script, that dynamically calcylates the CPUs mining profitability and compares it across various other CPUs. An example output (though shortened) can be seen below:
python3 mining-profitability.py
Difficulty: 354634073114
Average blocktime @6500 H/s: 631 days, 11:18:08
Price: 201.98 - XXMRZUSD
Expected crypto income/s: 1.236e-08
Expected fiat income/s: 2.496e-06
Profitability: 38.2%
Miner efficiency: 100.0 H/s/W
ROI time: 12577.0 days to recover 750 USD
Breakeven efficiency @0.1/kWh: 72.3 H/s/W
Network power consumption @0.1/kWh: 40849693 W
+--------+--------+-------+--------+
| Period | Profit | Mined | Power |
+========+========+=======+========+
| Day | 0.060 | 0.001 | 0.156 |
| Week | 0.417 | 0.007 | 1.092 |
| Month | 1.789 | 0.032 | 4.680 |
| Year | 21.770 | 0.390 | 56.940 |
+--------+--------+-------+--------+
+-------------------------------+--------+-------+------------+--------+
| CPU | Speed | Power | Efficiency | Amount |
+===============================+========+=======+============+========+
| AMD EPYC 7763 (dual) | 100000 | 750 | 133.333 | 54467 |
| AMD Ryzen Threadripper 3990X | 65000 | 600 | 108.333 | 68083 |
| AMD Ryzen 3600X | 7500 | 70 | 107.143 | 583568 |
| Intel Xeon Silver 4216 (dual) | 16157 | 312 | 51.785 | 130929 |
| Intel Core i7-4720HQ | 1768 | 45 | 39.289 | 907771 |
+-------------------------------+--------+-------+------------+--------+Milestones
1 Initial setup
The initial setup, that is able to use static astronomic data inputs (let’s call it by its name used in the industry: seasonality) and simple weather forecasts (sunny/cloudy/in-between) to make predictions based mostly on user input as to how much power a given computer and its cores consume, taking into account how much power is expected to be consumed in house for other purposes, that have to be treated as of a higher priority. Initially a status message will be displayed about what to do manually to maximize the selected goals. Plots of the inputs and outputs will be displayed, that explain why such a suggestion was presented.
Will be done mostly by mj, since Endor shouldn’t be strained with too many foreign systems, only receive the system’s output signals. This will require about 2 weeks of work full time, that makes:
- 6 (Monday-Saturday) * 2 (weeks) * 8 (hours) = 96 work hours
- 96 work hours / 10h in a week = 9.6 weeks spread.
2 Profitability calculator
Profitability calculator, taking into account the power costs, grid’s buy-back prices (if available) and user’s price target, at which to sell the mined coins. Also a very important input will be the time-bound energy availability, delivered through the system from the Milestone 1
This milestone will be done mostly by Endor, due to his experience in the field. I will only support via tips such as obtaining user input and code reviews, design suggestions, etc. I won’t count my pay on this one. This can be started before milestone 1) is finished, on a basis of simple generated sinusoidal signals, that will soon enough be replaced with real data.
Endor’s:
- 6 (Monday-Saturday) * 3 (weeks) * 8 (hours) = 144 work hours
- 144 work hours / 15h in a week = 9.6 weeks spread.
3 Measurements
- Support for various kinds of measurements – smart plugs, digital converters, software temperature readouts and even simple image recognition of LCD/LED displays of the DC→AC transformer and MPPT regulator.
Both of us have experience in various parts here and wish to take equal part in this milestone. I’ll create up to two examples of image recognition. mj’s:
- 6 (Monday-Saturday) * 2 (weeks) * 8 (hours) = 96 work hours
- 96 work hours / 10h in a week = 9.6 weeks spread.
Endor’s:
- 6 (Monday-Saturday) * 3 (weeks) * 8 (hours) = 144 work hours
- 144 work hours / 15h in a week = 9.6 weeks spread.
4 Automation
Automation of the system, including remote CPU downclocking, selecting the number of threads, putting computers to sleep, and waking them up via the Wake On LAN technology. We will be promoting privacy oriented and non-controversial software, namely: p2pool and xmrig, accordingly.
Same as above, both of us have experience and ideas here. I already perform such tasks manually (via scripts) on my solar farm, so I know “the feel” of the system. Endor has many great ideas and a lot of expertise for this milestone as well, as this resonates with his prior experience with mining profitability and a variety of mining software + hardware.
mj’s:
- 6 (Monday-Saturday) * 2 (weeks) * 8 (hours) = 96 work hours
- 96 work hours / 10h in a week = 9.6 weeks spread.
Endor’s:
- 6 (Monday-Saturday) * 3 (weeks) * 8 (hours) = 144 work hours
- 144 work hours / 15h in a week = 9.6 weeks spread.
Legal & privacy
The project shall be released under AGPLv3. It will send no telemetry by default, unless the user wishes to have their data uploaded and analyzed. The data will surely serve as a stabilization asset of the project and will be included in the automated system tests as one of many test cases. The complete data set to be sent will always be shown to the user for a review and approval.
As I’m familiar with the EU laws regarding solar panels a bit, I will be providing the users non-legally-binding relevant advice.
Even though I’m not strictly an electrical engineer, I know enough tricks in this field and I understand many nuances about the inner workings of such systems. I know what hardware are compatible with each other and how to calculate the supply and demand of the produced power. Therefore I can provide advice in this field as well, yet again: non-legally-binding.
Proposal
As of 12 April, XMR/EUR is at 214.92 EUR.
mj:
I will be able to spend about 10 hours / week on the project. An amount so low, that it doesn’t collide with my maintenance assignment, nor Monero Research Laboratory, once I switch there for a while. My rate per hour is 45 EUR. The total number of work hours is: 96 + 96 + 96 = 288. 288 [h] * 45 [EUR/h] / 214.92 [XMR/EUR] = 60.3 XMR
Endor:
I will be able to spend about 15 hours / week on the project. My rate in my first ever proposal is at 25 EUR / h. The total number of work hours is: 144 + 144 + 144 = 432. 432 [h] * 25 [EUR] / 214.92 [XMR/EUR] = 50.25 XMR
Since in this case we are really talking about seasonal data, that doesn’t require a lot of readjustments, further maintenance will not be a burden for any of us. Unless many new features are requested, we will gladly maintain it for you without asking for compensation.
Summary:
60.3 + 50.25 = 110.55 XMR is what we're asking for. Thanks in advance!
Deadline calculation
A rough deadline calculation based on the spread of the hours and including the tasks that can be done in parallel: 9.6 [weeks] * 3 / 4.1 [weeks in a month] ~ 7 months. This coincides with the promise to finalize the project until winter if all goes well with Endor’s plan. If not, I can take over most of the responsibilities, as there are always ~1-1.5 month breaks between my maintenance proposals.
Cheers!
Expiration date
31 Oct, 2022
Merge request reports
Activity
@mj small correction:
In the Who section:
Here’s a list of all my contributions to Monero and to the supporting software.
The correct link should probably be: https://github.com/search?q=is%3Apr+author%3Amj-xmr+archived%3Afalse+is%3Aclosed
I think its an awesome idea in general. I am an engineer with some experience around that field and I can tell you that such a tool would be great! It could planned/implemented into hundreds of companies and households each year because they typically take anything that increases the profitability of their plant and speeds up its amortization time. Especially if they already own the PC's required for mining.
However, static astronomic data, weather predictions or display recognition wont get you there. To make this work properly and actually be applicable, I think (experience not speculation) it needs to read out the inverter directly. Thats where you get the actual information of how much excess energy is available at the given moment. Based on this, the workload of the miner should be adjusted.
The key challenge in my opinion is that there are a lot of inverters (with different controllers) on the market. To cover a significant portion of this market, a lot of different read out routines need to be created i assume. Good thing is that most of these devices already have online functionalities and support a read out via API to enable remote monitoring (Some even come with their own smartphone app). I would recommend to focus on these remotely monitorable devices because everything else is dying out anyway.
For example I like to use all in one solutions from E3/DC (inverter + battery) when planning for private customers (most people like the design of these devices and it enables the use of a financial loophole in some EU-countries because they can get government funding for batteries for the whole device because it includes a battery). However, these E3/DC devices have online functionalities and provide an API where you can pull the inverter data.
They are also able to factor in weather data and so on but this is only about power peak management. Power-grids usually only take up to 70% of the peak power theoretically generated by your plant. If you want them to take 100%, you need to enable them to remotely shut down your plant to protect their grid from overloading in case of peak performance. If you dont want them to be able to do that, they wont take more than about 70% of your theoretical peak performance at once (depends a little on the local law). So the goal is to have spare battery storage or to switch on power consumers when peak performance kicks in to absorb the 30% which you cant sell. Thats why they try to work with weather forecast and other astronomical data there. But in reality this data is rarely precise enough and most of the time it wont work properly. A single cloud in the wrong spot is all it takes to completely mess with those calculations. To be able to move excess energy into crypto is basically an alternative and (IMHO) superior concept that makes this semi functional weather data BS obsolete.
e.g. https://sunspec.org/project/e3-dc-gmbh/
Another big point here is to level out the playing field when it comes to selling energy to the grid. It depends a little on the specific contracts but most of the time, it is brazen how bad they pay you for your energy in comparison to the official exchange prices and even worse compared to the price you pay when buying energy. In a lot of (1st world) countries it will be more profitable to only self consume and make crypto out of your generated energy and not sell anything to the grid at all.
To be able to create crypto out of excess energy and out of energy you would have sold (they are not the same) will increase the efficiency and decrease the amortization time of decentralized renewable energy systems. Therefore it could really boost the spread of those technologies.
I am absolutely in favor of this proposal if it is done right, which means to focus on inverter data and not on weather and so on.
Edited by OrneryMaintenance-
Thank you very much. This is an awesome input, esp. since you come from the field.
Maybe I'm wrong but I need to dispute this one:
However, static astronomic data, weather predictions or display recognition wont get you there. To make this work properly and actually be applicable, I think (experience not speculation) it needs to read out the inverter directly. Thats where you get the actual information of how much excess energy is available at the given moment. Based on this, the workload of the miner should be adjusted.
While reading the inverter gives a lot of information about available energy at the given moment, it cannot tell how fast you are allowed to use it up, if you don't know what the weather will be. If your batteries need a day to load fully and it's forecasted that tomorrow will be cloudy, and knowing that such weather would lead to 80% reduction of your production, you will definitely want to reduce the consumption of energy today to spare in for tomorrow. Moreover, if a particular desktop computer in your whole mining rig takes >40% of typical total power when it mines, just to keep it running in idle mode, and you know that you have to reduce its mining intensity because of the forecasted production shortages, it makes more sense to just completely switch it off ahead of the cloudy day. Include the prevention of overheating of mining rigs in an opposite situation, where you're blessed with overproduction and you've got a whole lot of variables to optimize to maximize your profit. Because if you can forecast, that the next x days will be sunny (weather) in the summer (static astronomic data), you'll definitely want to spread the mining across maybe even the entire 24h, not to overheat your rig. Also, in such situation, depending on how much CPU cache a given PC has, it might be smarter to mine on 1 of 4 available threads (for low cache CPUs) or on 2 of 4 available threads (for a high cache CPUs), since the threads compete for the cache, if it's low for a given CPU.
Perhaps I'm still missing something that should be obvious for me? IDK.
They are also able to factor in weather data and so on but this is only about power peak management.
This is a very interesting input, that I didn't take into account, but I can fully imagine it, because I read years ago about the grid's overload at peaks and how it becomes a problem. This will definitely have to be included in the model as one of the question from the app's setup wizard. Please bear in mind though, that the profitability of a given decision (whether to mine only on excess power or all the time) is a function of the buyback price at the given time, so we have another parameter to optimize against and our system should adapt to changes in the relative profit between mining and selling to the grid.
Regarding E3/DC: While I fully appreciate their solution, that would make our system more popular, I need to state here, that focusing on a single manufacturer would be a terrible blow towards healthy competition. And since the product is already practically subsidized, using a legal loophole, I'd only add to the problem of unhealthy competition if I focused on supporting only their solution. I'd like to leverage the situation and your interest, by asking you to run a couple of tests on the hardware that you have access to, keeping in mind, that this effort has to benefit you and your customers. However I will NOT publish (= tag) a version of our software without allowing other suppliers to deliver their implementations, so that they (or their user base) can perform equivalent readouts for their systems. Technically speaking, this will be realized through a plugin interface. I've already done something very similar in order to benchmark
tsqsim. See the following examples:To be able to create crypto out of excess energy and out of energy you would have sold (they are not the same) will increase the efficiency and decrease the amortization time of decentralized renewable energy systems. Therefore it could really boost the spread of those technologies.
I'm happy that you understand the basic point. It's also worth mentioning, that this improves decentralization of Monero's mining, while profits from mining from the free energy are just a token of gratitude from the Monero's users (= amortization).
Edited by mj While reading the inverter gives a lot of information about available energy at the given moment, it cannot tell how fast you are allowed to use it up, if you don't know what the weather will be. If your batteries need a day to load fully and it's forecasted that tomorrow will be cloudy, and knowing that such weather would lead to 80% reduction of your production, you will definitely want to reduce the consumption of energy today to spare in for tomorrow.
You are right with this. There can be some benefits in taking weather into account. But for me this is already 3 steps further. The first thing that is required is a software that can adjust the power draw of the miner according to the excess energy available at a given moment. This data is key and obtained by reading out the inverter.
Once this works, you could start to try to increase profitability by implementing estimations and forecasts. But from my experience this is really a questionable thing. Weather data is unprecise and sometimes even unreliable. I experienced this countless times ... e.g. it wont fill battery or charge the car in the morning because there is more sun expected at noon but then its a little cloudy at noon and you wasted your energy in the morning. Or you expect a rainy day tomorrow and reschedule energy heavy processes just to find out that the neighbor town got the rain und you still have sun....
What I am trying to say is that this is a feature which can be useful but its als questionable and sometimes cost you money rather than saving it. In my opinion this is something that can be added once the real functionality is there.
Regarding E3/DC: While I fully appreciate their solution, that would make our system more popular, I need to state here, that focusing on a single manufacturer would be a terrible blow towards healthy competition.
I fully agree with this and I did not mean to focus on E3/DC devices in particular. I just brought them up as an example of what kind of devices I would focus. I would suggest to focus on devices where the manufacturer already provides network capability and a remote read out of the inverter. Because devices that dont enable remote monitoring will die out anyway and I think its unnecessary to put much effort into them.
I'd like to leverage the situation and your interest, by asking you to run a couple of tests on the hardware that you have access to, keeping in mind, that this effort has to benefit you and your customers.
I am happy to help in any way I can and I would absolutely engage in testing and evaluating the software.
Technically speaking, this will be realized through a plugin interface.
I think this is a great approach here.
Edited by OrneryMaintenance-
Once this works, you could start to try to increase profitability by implementing estimations and forecasts. But from my experience this is really a questionable thing. Weather data is unprecise and sometimes even unreliable. I experienced this countless times ... e.g. it wont fill battery or charge the car in the morning because there is more sun expected at noon but then its a little cloudy at noon and you wasted your energy in the morning. Or you expect a rainy day tomorrow and reschedule energy heavy processes just to find out that the neighbor town got the rain und you still have sun....
I understand. Well, any estimator's output can be modeled with confidence bands. I'll keep it all in mind and not be overly enthusiastic about this part of input.
Because devices that dont enable remote monitoring will die out anyway and I think its unnecessary to put much effort into them.
I agree. That's why I just wanted to make an example or two on how to deal with the situation where there's no remote monitoring, and leave it up to the users to write their own specific plugins.
-
Once this works, you could start to try to increase profitability by implementing estimations and forecasts. But from my experience this is really a questionable thing. Weather data is unprecise and sometimes even unreliable.
The longer I think about it, the clearer it becomes to me, that the weather part of the "Sun" signal should be optional and selectable by the user.
-
Reddit post with a lot more positive feedback.
I can tell you some local figures
-
my government has been promoting rural PV for the last 10 years, all in cost is $1500 for 5KW, average power generation is 15kWh per day.
-
connected to the national grid, the government will buy back excess power, and will give green energy subsidies, if you use 5 kWh per day, the remaining power sold, almost 3 years to pay back the capital.
-
as to whether the mining profit, may be more or depends on the price of xmr.
-
anyway, this is a new application scenario, good.
-
-
- as to whether the mining profit, may be more or depends on the price of xmr.
And depends on what a given government currently "isn't planning".
One way or another, thanks for the local figures. It does bring a perspective.
Edited by mj
I think this is a great idea and has the potential to help a lot of people with monetizing their PV systems the best way possible for them. It seems that a lot of countries did a bait and switch when it came to solar and incentivized people with subsidies and promises of prices per KW/h being pushed into the grid and then they changed their minds about it.
-
Exactly. I know only about the spanish case from 2016:
The only thing that's predictable about governments is, that through subsidies they will first lure you into something that you think is a lucrative business, then screw you over, like that old man with candies.
Therefore in this context me, Endor and the entire Ecosystem will serve as:
- Regulator (or creator) of the free market, as anybody can choose whether to sell the overproduction for the currently promised candy, for pennies guaranteed in the future or for an increasing XMR/USD price. I assume the XMR price will only increase since we've been postponing both hyperinflation and energy crisis since at least 2008 (The Housing Bubble), if not since 2000 (The DotCom Bubble). This means that with less energy available cheaply fewer people will mine, increasing your own odds of success, plus that tiny XMR inflation is nothing compared to the inflation that awaits the Petrodollar.
- Safeguards of promises made by the governments. They will know, that as soon as they start speculating about decreasing the tariffs (A.K.A. "We're NOT planning to decrease the tariffs"), as it was in Spain, the Producers will already be able to switch to mining anytime... still enjoying the subsidies that they were awarded before.
Especially in the context of 2., it's absolutely critical to use @SChernykh p2pool, to be able to stay under the radar and pretend that you're simply not selling, only merely waiting for the government to raise the prices.
Edited by mj
@mj can you update the price?
-
@luigi1111 done. Thanks.
I'm keeping the price update commit separately. Pls let me know if it has to be squashed.
mentioned in commit 249c8789
mentioned in merge request !311 (closed)
