Offline Signing Library for XmrSigner Production

Offline Signing Library for XmrSigner Production

About

This proposal aims to create a minimal library for offline signing on air-gapped devices, and make XmrSigner production ready to actual use secure, focusing on essential features:

• Seed phrase generation (including polyseed)
• Address and key generation
• Account and sub-address management
• Address verification
• Output importing and Key Image exporting (raw and encrypted)*
• Unsigned transaction handling (description, sanity checks, signing)
• Block height and date estimation

The library will be implemented in C++ with a C ABI, allowing use in multiple languages. It will be based on the current Monero source but without relying on wallet2, aiming for minimal external dependencies, not inventing the wheel again, nor rolling own crypto.

Key objectives:

• Comprehensive documentation (OTS library, OTS Python library, buildroot packages desired integrated into monero docs for a easy and quick start to build something)
• Quick start guide for offline and hardware wallet developers
• Documentation of challenges in stripping down and cross-compiling
• Buildroot package for easy target system integration
• Python module for library usage
• Test code
• Modification of XmrSigner to use this new library

Who

Thor (vThor/DiosDelRayo), completed the XmrSigner (MoneroSigner Resurrection) proposal.

Why

XmrSigner, while functional, is currently more of a proof of concept than a production-ready tool. This library addresses several key issues:

1. Performance: The current implementation using wallet RPC is slow, especially on resource-constrained devices.
2. Security: Minimizing dependencies and code base improves auditability and reduces attack surface.
3. Flexibility: A dedicated library allows for easier integration into various offline signing solutions.
4. Resource efficiency: Stripping down to essential features enables use on lower-power devices.

By creating this library, we will:

• Bring XmrSigner to production readiness
• Provide a foundation for future hardware wallet development
• Explore the viability of using even more resource-constrained devices (e.g., MCUs) for offline signing
• Build the basis for a lib which could also serve for quick payment way (seems there exists something like it called pocket change, was not aware of before) even if the customer is offline. Talked about that in #monero-community

A proof of concept has been developed to validate this approach: c_abi_for_cpp_code_PoC

This library will significantly improve XmrSigner's performance, security, and usability while opening doors for more diverse Monero hardware wallet solutions in the future.

Milestones and Timeline

Offline Signing Library for XmrSigner Production

Milestones and Timeline

1. Setting up documentation site (4 hours)

All OTS library related will go into a fork of docs.get-monero.org and should take me no more than a blink of an eye to get going. For the XmrSigner related part I will setup the same mkdocs via github pages. This is apart from the Doxygen and Sphinx documentation, which I need to see how to make it best available on the way.

View tasks

Task	Hours	Sessions
Set up initial documentation structure	2	1
Configure documentation site	2	1
Use existing Monero documentation through PR's	0	0

2. MVP library with C ABI (164 hours)

The main task of this proposal, creating an ots folder in monero/src and nit pick the parts the lib needs together to create and compile a actual library. On top add the C ABI and make the build script as easy and robust as possible.

View tasks

Task	Hours	Sessions
Initial ABI design	16	8
Seed phrase generation	8	4
Address and key generation	10	5
Account and sub-address management	10	5
Address verification	10	5
Output importing and Key Image exporting	30	15
Unsigned transaction handling	30	15
Write Doxygen documentation	14	7
Create detailed documentation of the ABI and library internals	14	7
Develop unit tests with Check	14	7
PR process revisions and merge	8	4

3. Python cffi classes (44 hours)

Building the clue code between the C ABI and Python in a pythonic way.

View tasks

Task	Hours	Sessions
Design Python class structure	10	5
Implement Python wrapper classes	20	10
Write rich inline documentation	8	4
Create Sphinx documentation	2	1
Develop unit tests for Python classes	4	2

4. Building library in buildroot (44 hours)

buildroot is a build kit to create an embedded linux with the help of packages which get cross compiled for the target platform. Here the dependencies to build the OTS library from the monero sources need to be fulfilled.

Show more... A very critical part to use the library on the pi zero (or other target device - maybe modifications will be needed here fore). I build for XmrSigner before buildroot packages, but struggled to get monero-wallet-rpc as binary running on the pi via Buildroot (how I was building on the Pi itself and there are so many dependencies that finally I pulled the plug on that version, becauseit was clear that monero-wallet-rpc was only a temporary solution to finish the CCS in a running state)

View tasks

Task	Hours	Sessions
Adapt CMakeLists.txt for the library and its dependencies	20	10
Create buildroot package for downloading and compiling	12	6
Write documentation for builders/developers	12	6

5. Buildroot package for Python library (12 hours)

The same but a lot easier then for the OTS library, here the Python OTS library will get compiled via CPython, how there are only two dependencies (the OTS library and cffi) it should be much easier even with a deterministic build.

View tasks

Task	Hours	Sessions
Create buildroot package for the Python library	8	4
Write documentation for builders/developers	4	2

6. XmrSigner integration (34 hours)

Here the XmrSigner code get's the upgrade, monero-wallet-rpc, python-monero, python-polyseed will be replaced by the Python OTS library.

View tasks

Task	Hours	Sessions
Remove existing dependencies and integrate new Python library	26	13
Update XmrSigner documentation	8	4

7. XmrSigner on XmrSigner OS (buildroot2) (42 hours)

Here all gets assembled together to create the actual XmrSigner production image.

Show more... How I did that already outside of the last CSS, I need only to make some modifications, get the dependencies right, the biggest part will be testing and debugging if something goes wrong - I had this issue already, but hope that this time is minimal to use the calculated time for Milestone 9 because I really like to improve there. But 30 hours can vanish quick debugging issues there because it took like 30min to build on a 4c ore i7 with 16GB (lack at the moment something more powerful) and then flashing to microSD and test. But how from the buildroot image was forked from SeedSignerOS which stripped handy everything unnecessary away - good for security and speed - but terrible to debug.

View tasks

Task	Hours	Sessions
Create buildroot package for XmrSigner	8	4
Test and debug XmrSigner in buildroot environment	30	15
Write documentation for XmrSigner OS integration	4	2

8. Unify XmrSigner-related components (48 hours)

Getting all the XmrSigner stuff on one place together and the OTS library stuff on docs.get-monero.org, or at least in close reach.

Show more... Through the excessive amount of time the last CCS took the rebranding to XmrSigner and putting documentation and linking nice together suffered a bit. So here everything will get separated what is actual XmrSigner and what is needed to build a different Offline Signer from the OTS library and BCUR. So that XmrSigner is XmrSigner, and the building blocks a close to get-monrero.org as possible. All development resources should be on monero docs after the PR got merged, so there is a quickstart and all documents to create an offline signer pretty quick.

View tasks

Task	Hours	Sessions
Reorganize and consolidate XmrSigner-related content on GitHub	8	4
Create separate documentation for UR codes	20	10
Register UR types used	6	3
Final review and organization of all documentation	14	7

9. (Optional/Stretch) Clean-up and Optimization (82 hours)

Here is where my heart is, but after carefully calculating through the time, and catching myself, arguing on almost each point it could be done faster, and almost halving most points... I still with 138h more then the original 240 hours estimated. And instead of doubling or triple the time to have more air for unforeseen challenges I'm now even tighter, so that I probably will end up again working a lot of hours for free to get all done. But I still hope to safe enough time to get this milestone done, too.

View tasks

Task	Hours	Sessions
Analyze Monero source dependencies	20	10
Remove unnecessary dependencies	20	10
Optimize remaining code	10	5
Document changes and rationale	16	8
Analyze minimal resources needed for OTS library on restricted devices	8	4
PR process revisions and merge	8	4

10. (Stretch) Xmr Signer Pure (Rewrite in C++) (98 hours)

If I have the mega flow, I will continue here to move substitute the Python code to C++. Why C++ and not Go or not Rust? First I'm still learning daily in C++, but the more important reason, I think there is a benefit in thinking on a bare metal XmrSigner NG in future. circle and libcamera would make it possible to run XmrSigner on bare metal what is a C++ environment... So done this here well, most of the work would be already done for the next step.

View tasks

Task	Hours	Sessions
Build environment	6	3
Screen	8	4
UI	12	6
Logic	20	10
OTS	4	2
CBOR	8	4
BCUR	8	4
Zxing	12	6
Cam	20	10

11. (Stretch) JNI classes (44 hours)

This is another thing I wanted to include but the time constraints will probably not allow it. One to have a second language using the C ABI with documentation, and second to be used for Android apps.

View tasks

Task	Hours	Sessions
Design Java class structure	10	-
Implement Java wrapper classes	20	-
Create Java documentation	10	-
Develop unit tests for Java classes	4	44

Total Estimated Time

Core Milestones (1-8): 392 hours Optional/Stretch Milestones (9-11): 224 hours Grand Total: 616 hours

Note: This timeline is an estimate and may be subject to adjustments as the project progresses. The sessions are based on the Pomodoro technique to maintain high focus and productivity throughout the project.

Given the significant time investment and potential challenges ahead, I propose a shift from my usual value-based pricing to an monthly rate for this project.

Proposed terms:

• 65 Promodoro Sessions (130 hours) per month for three months (196 sessions in total, first month one extra session)
• Compensation: 1 XMR per session, making 66 XMR first and 65 XMR second and 65 third month. (196 in total)
• I would kindly request a 20 xmr upfront on start from the first month.
• Total compensation: 196 XMR

Hours worked beyond in the running month shall roll over to the next month. If I can work some time out I will work on the optional/strech goals. If time is missing to get milestone 1-8 done in the calculated hours, I will work hours for free to get it done - I always do.

I will commit every day my work, on starting each repository I will advertise them here in a comment an push my commits every day. And I will create a repository/blog or something as a daily logbook to document the work done and the daily progress.

I set up a website which syncronizes my actual pomodoro sessions, and the last 48 hours the work done

on which tasks and how many sessions, even each unit time, so it is more transparent. On the way I plan to render also all tasks with the total time and a calendar. But the raw data is available in the issues of the github repository of the site. The time worked on each task can be calculated by the label and unlabeling of `WIP` label is start to work and unlabel end of work on it.

vThorOfflineSigningLibrary_XmrSignerToProduction.md

@@ -3,18 +3,26 @@ layout: fr
 title: Offline Signing Library for XmrSigner to Production
 date: August 31, 2024
 author: Thor a.k.a. vThor a.k.a DiosDelRayo
-amount: 97
+amount: 197
 milestones:
-  - name: Material
-    funds: 1 XMR
+  - name: Material and Upfront
+    funds: 21 XMR
     done:
     status: unfinished
-  - name: First month
-    funds: 48 XMR
+  - name: First month, minus upfront
+    funds: 46 XMR
     done:
     status: unfinished
   - name: Second month
-    funds: 48 XMR
+    funds: 65 XMR
+    done:
+    status: unfinished
+  - name: Third month
+    funds: 65 XMR
+    done:
+    status: unfinished
+  - name: Optional strech goals overfunding up to 112
+    funds: 0 XMR
     done:
     status: unfinished
 payouts:
@@ -24,6 +32,10 @@ payouts:
     amount:
   - date:
     amount:
+  - date:
+    amount:
+  - date:
+    amount:
 ---
 
 # Offline Signing Library for XmrSigner Production
@@ -31,7 +43,7 @@ payouts:
 ## About
 
 This proposal aims to create a minimal library for offline signing on
-air-gapped devices, focusing on essential features:
+air-gapped devices, and make XmrSigner production ready to actual use secure, focusing on essential features:
 
 - Seed phrase generation (including polyseed)
 - Address and key generation
@@ -47,7 +59,7 @@ without relying on wallet2, aiming for minimal external dependencies,
 not inventing the wheel again, nor rolling own crypto.
 
 Key objectives:
-- Comprehensive documentation
+- Comprehensive documentation (OTS library, OTS Python library, buildroot packages desired integrated into monero docs for a easy and quick start to build something)
 - Quick start guide for offline and hardware wallet developers
 - Documentation of challenges in stripping down and cross-compiling
 - Buildroot package for easy target system integration
@@ -57,7 +69,7 @@ Key objectives:
 
 ## Who
 
-Thor (vThor/DiosDelRayo), currently completing the [XmrSigner (MoneroSigner Resurrection) proposal](https://ccs.getmonero.org/proposals/%20MoneroSignerResurrection.html).
+Thor (vThor/DiosDelRayo), completed the [XmrSigner (MoneroSigner Resurrection) proposal](https://ccs.getmonero.org/proposals/%20MoneroSignerResurrection.html).
 
 ## Why
 
@@ -73,7 +85,7 @@ various offline signing solutions.
 4. Resource efficiency: Stripping down to essential features enables use on
 lower-power devices.
 
-By creating this library, we can:
+By creating this library, we will:
 - Bring XmrSigner to production readiness
 - Provide a foundation for future hardware wallet development
 - Explore the viability of using even more resource-constrained devices
@@ -87,18 +99,263 @@ and usability while opening doors for more diverse Monero hardware wallet
 solutions in the future.
 
 ## Milestones and Timeline
+# Offline Signing Library for XmrSigner Production
+
+## Milestones and Timeline
+
+### 1. Setting up documentation site (4 hours)
+
+All OTS library related will go into a fork of docs.get-monero.org and should take me no more than a blink of an eye to get going.
+For the XmrSigner related part I will setup the same mkdocs via github pages.
+This is apart from the Doxygen and Sphinx documentation, which I need to see how to make it best available on the way.
+<details>
+<summary>View tasks</summary>
+
+| Task | Hours | Sessions |
+|------|-------|----------|
+| Set up initial documentation structure | 2 | 1 |
+| Configure documentation site | 2 | 1 |
+| Use existing Monero documentation through PR's | 0 | 0 |
+
+</details>
+
+### 2. MVP library with C ABI (164 hours)
+
+The main task of this proposal, creating an ots folder in monero/src and nit pick the parts the lib needs together
+to create and compile a actual library. On top add the C ABI and make the build script as easy and robust as possible.
+<details>
+<summary>View tasks</summary>
+
+| Task | Hours | Sessions |
+|------|-------|----------|
+| Initial ABI design | 16 | 8 |
+| Seed phrase generation | 8 | 4 |
+| Address and key generation | 10 | 5 |
+| Account and sub-address management | 10 | 5 |
+| Address verification | 10 | 5 |
+| Output importing and Key Image exporting | 30 | 15 |
+| Unsigned transaction handling | 30 | 15 |
+| Write Doxygen documentation | 14 | 7 |
+| Create detailed documentation of the ABI and library internals | 14 | 7 |
+| Develop unit tests with Check | 14 | 7 |
+| PR process revisions and merge | 8 | 4 |
+
+</details>
+
+### 3. Python cffi classes (44 hours)
+
+Building the clue code between the C ABI and Python in a pythonic way.
+
+<details>
+<summary>View tasks</summary>
+
+| Task | Hours | Sessions |
+|------|-------|----------|
+| Design Python class structure | 10 | 5 |
+| Implement Python wrapper classes | 20 | 10 |
+| Write rich inline documentation | 8 | 4 |
+| Create Sphinx documentation | 2 | 1 |
+| Develop unit tests for Python classes | 4 | 2 |
+
+</details>
+
+### 4. Building library in buildroot (44 hours)
+
+buildroot is a build kit to create an embedded linux with the help of
+packages which get cross compiled for the target platform. Here the dependencies
+to build the OTS library from the monero sources need to be fulfilled.
+
+<details>
+<summery>Show more...</summary>
+A very critical part to use the library on the pi zero (or other target device - maybe modifications will be needed here fore).
+I build for XmrSigner before buildroot packages, but struggled to get monero-wallet-rpc as binary running on
+the pi via Buildroot (how I was building on the Pi itself and there are so many dependencies that finally I pulled
+the plug on that version, becauseit was clear that monero-wallet-rpc was only a temporary solution to finish the CCS
+in a running state)
+</details>
+
+<details>
+<summary>View tasks</summary>
+
+| Task | Hours | Sessions |
+|------|-------|----------|
+| Adapt CMakeLists.txt for the library and its dependencies | 20 | 10 |
+| Create buildroot package for downloading and compiling | 12 | 6 |
+| Write documentation for builders/developers | 12 | 6 |
+
+</details>
+
+### 5. Buildroot package for Python library (12 hours)
+
+The same but a lot easier then for the OTS library, here the Python OTS
+library will get compiled via CPython, how there are only two dependencies (the OTS library and
+cffi) it should be much easier even with a deterministic build.
+
+<details>
+<summary>View tasks</summary>
+
+| Task | Hours | Sessions |
+|------|-------|----------|
+| Create buildroot package for the Python library | 8 | 4 |
+| Write documentation for builders/developers | 4 | 2 |
+
+</details>
 
+### 6. XmrSigner integration (34 hours)
+
+Here the XmrSigner code get's the upgrade, monero-wallet-rpc, python-monero, python-polyseed will
+be replaced by the Python OTS library.
+
+<details>
+<summary>View tasks</summary>
+
+| Task | Hours | Sessions |
+|------|-------|----------|
+| Remove existing dependencies and integrate new Python library | 26 | 13 |
+| Update XmrSigner documentation | 8 | 4 |
+
+</details>
+
+### 7. XmrSigner on XmrSigner OS (buildroot2) (42 hours)
+
+Here all gets assembled together to create the actual XmrSigner production image.
+
+<details>
+<summery>Show more...</summary>
+How I did that already outside of the last CSS, I need only to make some modifications,
+get the dependencies right, the biggest part will be testing and debugging if something
+goes wrong - I had this issue already, but hope that this time is minimal to use the calculated
+time for Milestone 9 because I really like to improve there. But 30 hours can vanish quick debugging
+issues there because it took like 30min to build on a 4c ore i7 with 16GB (lack at the moment something more powerful)
+and then flashing to microSD and test. But how from the buildroot image was forked from SeedSignerOS which stripped handy
+everything unnecessary away - good for security and speed - but terrible to debug.
+</details>
+
+<details>
+<summary>View tasks</summary>
+
+| Task | Hours | Sessions |
+|------|-------|----------|
+| Create buildroot package for XmrSigner | 8 | 4 |
+| Test and debug XmrSigner in buildroot environment | 30 | 15 |
+| Write documentation for XmrSigner OS integration | 4 | 2 |
+
+</details>
+
+### 8. Unify XmrSigner-related components (48 hours)
+
+Getting all the XmrSigner stuff on one place together and the OTS library stuff on docs.get-monero.org, or
+at least in close reach.
+<details>
+<summery>Show more...</summary>
+Through the excessive amount of time the last CCS took the rebranding to XmrSigner and
+putting documentation and linking nice together suffered a bit. So here everything will
+get separated what is actual XmrSigner and what is needed to build a different Offline Signer
+from the OTS library and BCUR. So that XmrSigner is XmrSigner, and the building blocks a close to get-monrero.org
+as possible. All development resources should be on monero docs after the PR got merged, so there is a quickstart
+and all documents to create an offline signer pretty quick.
+</details>
+
+<details>
+<summary>View tasks</summary>
+
+| Task | Hours | Sessions |
+|------|-------|----------|
+| Reorganize and consolidate XmrSigner-related content on GitHub | 8 | 4 |
+| Create separate documentation for UR codes | 20 | 10 |
+| Register UR types used | 6 | 3 |
+| Final review and organization of all documentation | 14 | 7 |
+
+</details>
+
+### 9. (Optional/Stretch) Clean-up and Optimization (82 hours)
+
+Here is where my heart is, but after carefully calculating through the time, and catching myself,
+arguing on almost each point it could be done faster, and almost halving most points... I still with 138h more
+then the original 240 hours estimated. And instead of doubling or triple the time to have more air for unforeseen
+challenges I'm now even tighter, so that I probably will end up again working a lot of hours for free to get all done.
+But I still hope to safe enough time to get this milestone done, too.
+
+<details>
+<summary>View tasks</summary>
+
+| Task | Hours | Sessions |
+|------|-------|----------|
+| Analyze Monero source dependencies | 20 | 10 |
+| Remove unnecessary dependencies | 20 | 10 |
+| Optimize remaining code | 10 | 5 |
+| Document changes and rationale | 16 | 8 |
+| Analyze minimal resources needed for OTS library on restricted devices | 8 | 4 |
+| PR process revisions and merge | 8 | 4 |
+
+</details>
+
+### 10. (Stretch) Xmr Signer Pure (Rewrite in C++) (98 hours)
+
+If I have the mega flow, I will continue here to move substitute the Python code
+to C++. Why C++ and not Go or not Rust? First I'm still learning daily in C++, but
+the more important reason, I think there is a benefit in thinking on a bare metal
+XmrSigner NG in future. [circle](https://github.com/rsta2/circle) and [libcamera](https://github.com/rsta2/libcamera)
+would make it possible to run XmrSigner on bare metal what is a C++ environment...
+So done this here well, most of the work would be already done for the next step.
+
+<details>
+<summary>View tasks</summary>
+
+| Task | Hours | Sessions |
+|------|-------|----------|
+| Build environment | 6 | 3 |
+| Screen | 8 | 4 |
+| UI | 12 | 6 |
+| Logic | 20 | 10 |
+| OTS | 4 | 2 |
+| CBOR | 8 | 4 |
+| BCUR | 8 | 4 |
+| Zxing | 12 | 6 |
+| Cam | 20 | 10 |
+
+</details>
+
+### 11. (Stretch) JNI classes (44 hours)
+
+This is another thing I wanted to include but the time constraints will probably not allow it.
+One to have a second language using the C ABI with documentation, and second to be used
+for Android apps.
+
+<details>
+<summary>View tasks</summary>
+
+| Task | Hours | Sessions |
+|------|-------|----------|
+| Design Java class structure | 10 | - |
+| Implement Java wrapper classes | 20 | - |
+| Create Java documentation | 10 | - |
+| Develop unit tests for Java classes | 4 | 44 |
+</details>
+
+## Total Estimated Time
+
+Core Milestones (1-8): 392 hours
+Optional/Stretch Milestones (9-11): 224 hours
+Grand Total: 616 hours
+
+*Note: This timeline is an estimate and may be subject to adjustments as the project progresses. The sessions are based on the Pomodoro technique to maintain high focus and productivity throughout the project.*
+  
 Given the significant time investment and potential challenges ahead, I
-propose a shift from my usual value-based pricing to an hourly rate for
+propose a shift from my usual value-based pricing to an monthly rate for
 this project. 
 
 Proposed terms:
-- 120 hours per month for two months (240 hours total)
-- Compensation: 48 XMR per month (96 XMR total)
-- Additional 1 XMR for printed literature (requested at project start)
-- Total compensation: 97 XMR
-
-Hours worked beyond 120 in the first month will roll over to the second. If
-the project completes before 240 hours, unused funds can be allocated to
-future proposals or returned to the general fund. Should more time be
-needed, I'll submit a new proposal.
+- 65 Promodoro Sessions (130 hours) per month for three months (196 sessions in total, first month one extra session)
+- Compensation: 1 XMR per session, making 66 XMR first and 65 XMR second and 65 third month. (196 in total)
+- Additional 1 XMR for printed literature (requested at project start), and I would kindly request a 20 xmr upfront on start from the first month.
+- Total compensation: 197 XMR
+
+Hours worked beyond in the running month shall roll over to the next month. If I can work some time out I will work
+on the optional/strech goals. If time is missing to get milestone 1-8 done in the calculated hours, I will work hours for
+free to get it done - I always do.
+
+I will commit every day my work, on starting each repository I will advertise them here in a comment an push my commits every day.
+And I will create a repository/blog or something as a daily logbook to document the work done and the daily progress.
+
+I also kindly request in case the proposal would get over funded that each XMR up to 308 will fund the optional/strech goals by one session (2 hours) per XMR. Where I will try to do up to 90 sessions/month and the rest flowing over in a fourth month. I will add this milestones in the front matter to the end, to be paid at the end in case it will happen.