robosats/scripts/traditional

Robosats traditional environment

Robosats backend development and testing without docker and containers.

Binaries needed:

• postgresql (postgres, initdb, psql)
• redis (redis-server)
• bitcoin (bitcoind, bitcoin-cli)
• cln (lightningd, lightning-cli, holdinvoice)
• lnd (lnd, lncli)

Preparation

Postgresql and redis can be found in all linux distros and bsds.

Some distros do not put postgresql binaries in PATH, if this is the case simply link them somewhere in your PATH.

Example on debian:

ln -sf /usr/lib/postgresql/16/bin/postgres ~/.local/bin/
ln -sf /usr/lib/postgresql/16/bin/initdb ~/.local/bin/
ln -sf /usr/lib/postgresql/16/bin/psql ~/.local/bin/

Bitcoin nodes if not already installed need to be manually downloaded.

• bitcoin core binaries can be found here: https://bitcoincore.org/en/download
• cln binaries can be found here: https://github.com/ElementsProject/lightning/releases
• holdinvoice binary can be found here: https://github.com/daywalker90/holdinvoice/releases
• lnd binaries can be found here: https://github.com/lightningnetwork/lnd/releases

Example preparation:

$ python3 -m venv venv
$ . venv/bin/activate
$ pip install -r requirements_dev.txt
$ pip install -r requirements.txt

$ mkdir traditional
$ mkdir traditional/programs
$ cd traditional/programs

# if you do not have them already installed
$ mkdir bitcoin cln lnd
# download bitcoin, cln (and holdinvoice) and lnd binaries

# follow https://github.com/hoytech/strfry#compile
$ git clone https://github.com/hoytech/strfry
$ cd strfry
$ git submodule update --init
$ git checkout 1.0.4
$ make setup-golpe
$ make -j4

# if you want to use roboauto
# still in traditional/programs
$ git clone https://github.com/jerryfletcher21/roboauto
$ cd roboauto
$ pip install -r requirements.txt
$ pip install .

env file

$ cp .env-sample .env

Edit .env, both robosats and traditional scripts will read from there.

Variables to change:

LNVENDOR = "CLN"
# LNVENDOR = "LND"

BITCOIND_RPCURL = "http://127.0.0.1:18443"
BITCOIND_RPCUSER = "test"
BITCOIND_RPCPASSWORD = "test"
CLN_DIR = "traditional/nodes/cln-coord/regtest/"
CLN_GRPC_HOST = "localhost:9999"
CLN_GRPC_HOLD_HOST = "localhost:9998"
LND_DIR = "traditional/nodes/lnd-coord/"
MACAROON_PATH = "data/chain/bitcoin/regtest/admin.macaroon"
LND_GRPC_HOST = "localhost:10009"
# POSTGRES_DB should not be postgres
POSTGRES_DB = "robosats"
# POSTGRES_USER should not be the same as $USER environment variable
POSTGRES_USER = "robosats"
POSTGRES_PASSWORD = "robosats"
USE_TOR = False
LOG_TO_CONSOLE = True
LOGGER_LEVEL = "INFO"

Variables to add:

DEVELOPMENT = True
TESTING = True
SKIP_FRONTEND_TESTS = True
TIMING_EXTRA_IN_TESTS = True

# LNVENDOR_USER is what robosats calls robot node in tests
LNVENDOR_USER = "LND"
# LNVENDOR_USER = "CLN"

DAPHNE_PORT = 9000
GUNICORN_PORT = 8080
RUNSERVER_PORT = 8000
STRFRY_PORT = 7778

BITCOIND_BIN = "traditional/programs/bitcoin/bin/bitcoind"
BITCOIN_CLI_BIN = "traditional/programs/bitcoin/bin/bitcoin-cli"
LIGHTNINGD_BIN = "traditional/programs/cln/bin/lightningd"
LIGHTNING_CLI_BIN = "traditional/programs/cln/bin/lightning-cli"
HOLDINVOICE_PLUGIN_BIN = "traditional/programs/cln/holdinvoice"
LND_BIN = "traditional/programs/lnd/lnd"
LNCLI_BIN = "traditional/programs/lnd/lncli"
STRFRY_GIT_DIR = "traditional/programs/strfry"
ROBOAUTO_GIT_DIR = "traditional/programs/roboauto"
TRADITIONAL_NODES_DIR = "traditional/nodes"
TRADITIONAL_SERVICES_DIR = "traditional/services"
TRADITIONAL_LOGS_DIR = "traditional/logs"
GNUPG_DIR = "traditional/services/gnupg"

BITCOIN_TEST_ZMQ_BLOCK_PORT = 28432
BITCOIN_TEST_ZMQ_TX_PORT = 28433

# CLN_GRPC_HOST and CLN_GRPC_HOLD_HOST are for coord
CLN_TEST_COORD_LISTEN_PORT = 19846
CLN_TEST_USER_GRPC_PORT = 9989
CLN_TEST_USER_LISTEN_PORT = 19836

# LND_GRPC_HOST is for coord
LND_TEST_COORD_LISTEN_PORT = 19746
LND_TEST_COORD_REST_PORT = 8181
LND_TEST_COORD_MACAROON_PATH = "traditional/nodes/lnd-coord/data/chain/bitcoin/regtest/admin.macaroon"

LND_TEST_USER_GRPC_PORT = 10010
LND_TEST_USER_LISTEN_PORT = 19736
LND_TEST_USER_REST_PORT = 8182
LND_TEST_USER_MACAROON_PATH = "traditional/nodes/lnd-user/data/chain/bitcoin/regtest/admin.macaroon"

Paths can be relative or absolute. Binaries should be paths, they are not resolved with PATH.

Roboauto can be disabled by not setting ROBOAUTO_GIT_DIR or setting it to false.

If some ports are already in use, change their value.

To check which port are already in use, netstat -tulnp with root privileges can be used.

For example if there is alread an instance of postgresql running on the default port, change POSTGRES_PORT = "5433".

Usage

For development and testing two scripts are provided:

• traditional-services for non bitcoin related services
• regtest-nodes for bitcoin and lightning nodes

traditional-services sets up the database and manages the services.

Everything is done locally, so no root privileges/service managers are needed.

regtest-nodes is a script that should be sourced, it sets up a regtest environment, with bitcoin core, cln, lnd and roboauto, connecting them and creating channels. It then exposes the functions btc_reg, cln_coord, cln_user, lnd_coord, lnd_user, ra_reg to interact with the nodes and roboauto.

If the script is sourced in a bash shell, it will also source completions for all the functions.

regtest-nodes can also be run without arguments to simply expose the functions to start and remove the nodes and to create the channels between them, without setting up a specific environment.

Setup:

# change .env file

$ . venv/bin/activate

# generate cln and lnd grpc
$ scripts/generate_grpc.sh

$ . scripts/traditional/regtest-nodes test

$ scripts/traditional/traditional-services postgres-setup

$ scripts/traditional/traditional-services postgres-database

$ scripts/traditional/traditional-services strfry-setup

# remove the nodes
$ robosats_regtest_stop_and_remove_all

# postgres-database will create the database specified by
# POSTGRES_DB in .env, it can be run multiple times with
# different values of POSTGRES_DB for different databases

Testing:

# edit .env setting LNVENDOR to either "CLN" or "LND"
# LNVENDOR_USER while running tests should be set to "LND"

# in the main window
$ . venv/bin/activate
$ . scripts/traditional/regtest-nodes test

# in a secondary window
$ . venv/bin/activate
# can be stopped with Control-C
$ scripts/traditional/traditional-services test

# back in the main window
$ python3 manage.py test

# after having run the tests run
$ robosats_regtest_stop_and_remove_all
# to remove the nodes, they will be recreated when
# running the tests again

# python3 manage.py test can be run multiple times with the same database
# to have a clean database either use a different value
# of POSTGRES_DB or use a different directory (and run again the setup)
# by moving traditional/services/postgres somewhere and the moving it back when
# you want to use the old database again

Development:

# edit .env setting LNVENDOR to either "CLN" or "LND"
# and LNVENDOR_USER to either "CLN" or "LND"

# in the main window
$ . venv/bin/activate
$ . scripts/traditional/regtest-nodes server

# in a secondary window
$ . venv/bin/activate
# can be stopped with Control-C
$ scripts/traditional/traditional-services server

# to see the output of the django runserver command
# in a third window
$ tail -f traditional/logs/runserver

# if roboauto is active, use it to test the backend
# back in the main window
$ ra_reg --help
...
$ ra_reg create-order "$(ra_reg generate-robot --loc)" type=buy currency=btc min_amount=0.001 max_amount=0.002 payment_method="On-Chain BTC" premium=-1.5
...
$ ra_reg take-order $(ra_reg generate-robot --loc) order-id
...
$ ra_reg escrow-pay RobotName
...

Production:

# edit .env setting LNVENDOR to either "CLN" or "LND"
DEVELOPMENT = False
TESTING = False
USE_TOR = True
LOG_TO_CONSOLE = False
LOGGER_LEVEL = "WARNING"
HOST_NAME = "coordinator_onion_address.onion"
HOST_NAME2 = "*"
ONION_LOCATION = "coordinator_onion_address.onion"

$ . venv/bin/activate

$ scripts/traditional/traditional-services postgres-setup

$ scripts/traditional/traditional-services postgres-database-production

$ scripts/traditional/traditional-services strfry-setup
$ scripts/traditional/traditional-services nginx-setup

# change commit_sha

$ scripts/traditional/traditional-services production

Update:

$ . venv/bin/activate

$ pip install -r requirements_dev.txt
$ pip install -r requirements.txt

$ scripts/generate_grpc.sh

# for testing:
# start just postgres and redis in an other window
$ scripts/traditional/traditional-services test
# in the main window
$ . scripts/traditional/regtest-nodes test

# for production:
# backup traditional folder
# update commit_sha
$ scripts/traditional/traditional-services production

# then for both testing and production, in the main window
$ python3 manage.py migrate