Merge pull request #19 from Groestlcoin/0.8.0

0.8.0

.gitattributes

@@ -4,3 +4,7 @@
 Makefile text eol=lf
 configure text eol=lf
 .gitmodules text eol=lf
+# The following files are generated and should not be shown in the
+# diffs by default on Github.
+doc/lightning*.7 linguist-generated=true
+

.gitignore

@@ -18,6 +18,7 @@ ccan/ccan/cdump/tools/cdump-enumstr
 gen_*
 cli/lightning-cli
 tools/check-bolt
+tools/hsmtool
 coverage
 ccan/config.h
 __pycache__

.travis/build.sh

@@ -10,7 +10,7 @@ export SOURCE_CHECK_ONLY=${SOURCE_CHECK_ONLY:-"false"}
 export COMPAT=${COMPAT:-1}
 export PATH=$CWD/dependencies/bin:"$HOME"/.local/bin:"$PATH"
 export PYTEST_PAR=2
-export PYTHONPATH=$PWD/contrib/pylightning:$PYTHONPATH
+
 # If we're not in developer mode, tests spend a lot of time waiting for gossip!
 # But if we're under valgrind, we can run out of memory!
 if [ "$DEVELOPER" = 0 ] && [ "$VALGRIND" = 0 ]; then

CHANGELOG.md

@@ -4,9 +4,80 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.8.0] - 2019-12-16: "Blockchain Good, Orange Coin Bad"
+
+This release was named by Michael Schmoock @m-schmoock.
+
+### Added
+
+ - JSON API: Added `createonion` and `sendonion` JSON-RPC methods allowing the implementation of custom protocol extensions that are not directly implemented in c-lightning itself. ([3260](https://github.com/ElementsProject/lightning/pull/3260))
+ - JSON API: `listinvoices` now displays the payment preimage if the invoice was paid. ([3295](https://github.com/ElementsProject/lightning/pull/3295))
+ - JSON API: `listpeers` channels now include `close_to` and `close_to_addr` iff a `close_to` address was specified at channel open ([3223](https://github.com/ElementsProject/lightning/pull/3223))
+ - The new `pyln-testing` package now contains the testing infrastructure so it can be reused to test against c-lightning in external projects ([3218](https://github.com/ElementsProject/lightning/pull/3218))
+ - config: configuration files now support `include`. ([3268](https://github.com/ElementsProject/lightning/pull/3268))
+ - options: Allow the Tor inbound service port differ from 9735 ([3155](https://github.com/ElementsProject/lightning/pull/3155))
+ - options: Persistent Tor address support ([3155](https://github.com/ElementsProject/lightning/pull/3155))
+ - plugins: A new plugin hook, `rpc_command` allows a plugin to take over `lightningd` for any RPC command. ([2925](https://github.com/ElementsProject/lightning/pull/2925))
+ - plugins: Allow the `accepter` to specify an upfront_shutdown_script for a channel via a `close_to` field in the openchannel hook result ([3280](https://github.com/ElementsProject/lightning/pull/3280))
+ - plugins: Plugins may now handle modern TLV-style payloads via the `htlc_accepted` hook ([3260](https://github.com/ElementsProject/lightning/pull/3260))
+ - plugins: libplugin now supports writing plugins which register to hooks ([3317](https://github.com/ElementsProject/lightning/pull/3317))
+ - plugins: libplugin now supports writing plugins which register to notifications ([3317](https://github.com/ElementsProject/lightning/pull/3317))
+ - protocol: Payment amount fuzzing is restored, but through shadow routing. ([3212](https://github.com/ElementsProject/lightning/pull/3212))
+ - protocol: We now signal the network we are running on at init. ([3300](https://github.com/ElementsProject/lightning/pull/3300))
+ - protocol: can now send and receive TLV-style onion messages. ([3335](https://github.com/ElementsProject/lightning/pull/3335))
+ - protocol: can now send and receive BOLT11 payment_secrets. ([3335](https://github.com/ElementsProject/lightning/pull/3335))
+ - protocol: can now receive basic multi-part payments. ([3335](https://github.com/ElementsProject/lightning/pull/3335))
+ - JSON RPC: low-level commands sendpay and waitsendpay can now be used to manually send multi-part payments. ([3335](https://github.com/ElementsProject/lightning/pull/3335))
+ - quirks: Workaround LND's `reply_channel_range` issues instead of sending error. ([3264](https://github.com/ElementsProject/lightning/pull/3264))
+ - tools: A new command, `guesstoremote`, is added to the hsmtool. It is meant to be used to recover funds after an unilateral close of a channel with `option_static_remotekey` enabled. ([3292](https://github.com/ElementsProject/lightning/pull/3292))
+
+### Changed
+
+:warning: The default network and the default location of the lightning home directory changed. Please make sure that the configuration, key file and database are moved into the network-specific subdirectory.
+
+ - config: Default network (new installs) is now bitcoin, not testnet. ([3268](https://github.com/ElementsProject/lightning/pull/3268))
+ - config: Lightning directory, plugins and files moved into `<network>/` subdir ([3268](https://github.com/ElementsProject/lightning/pull/3268))
+ - JSON API: The `fundchannel` command now tries to connect to the peer before funding the channel, no need to `connect` before `fundchannel` if an address for the peer is known ([3314](https://github.com/ElementsProject/lightning/pull/3314))
+ - JSON API: `htlc_accepted` hook has `type` (currently `legacy` or `tlv`) and other fields directly inside `onion`. ([3167](https://github.com/ElementsProject/lightning/pull/3167))
+ - JSON API: `lightning_` prefixes removed from subdaemon names, including in listpeers `owner` field. ([3241](https://github.com/ElementsProject/lightning/pull/3241))
+ - JSON API: `listconfigs` now structures plugins and include their options ([3283](https://github.com/ElementsProject/lightning/pull/3283))
+ - JSON API: the `raw_payload` now includes the first byte, i.e., the realm byte, of the payload as well. This allows correct decoding of a TLV payload in the plugins. ([3261](https://github.com/ElementsProject/lightning/pull/3261))
+ - logging: formatting made uniform: [NODEID-]SUBSYSTEM: MESSAGE ([3241](https://github.com/ElementsProject/lightning/pull/3241))
+ - options: `config` and `<network>/config` read by default. ([3268](https://github.com/ElementsProject/lightning/pull/3268))
+ - options: log-level can now specify different levels for different subsystems. ([3241](https://github.com/ElementsProject/lightning/pull/3241))
+ - protocol: The TLV payloads for the onion packets are no longer considered an experimental feature and generally available. ([3260](https://github.com/ElementsProject/lightning/pull/3260))
+ - quirks: We'll now reconnect and retry if we get an error on an established channel. This works around lnd sending error messages that may be non-fatal. ([3340](https://github.com/ElementsProject/lightning/pull/3340))
+
+:warning: If you don't have a config file, you now may need to specify the network to `lightning-cli` ([3268](https://github.com/ElementsProject/lightning/pull/3268))
+
+### Deprecated
+
+Note: You should always set `allow-deprecated-apis=false` to test for changes.
+
+ - JSON API: `listconfigs` duplicated "plugin" paths ([3283](https://github.com/ElementsProject/lightning/pull/3283))
+ - JSON API: `htlc_accepted` hook `per_hop_v0` object deprecated, as is `short_channel_id` for the final hop. ([3167](https://github.com/ElementsProject/lightning/pull/3167))
+
+### Removed
+
+ - JSON: `listpays` won't shown payments made via sendpay without a bolt11 string, or before 0.7.0. ([3309](https://github.com/ElementsProject/lightning/pull/3309))
+
+### Fixed
+
+ - JSON API: #3231 `listtransactions` crash ([3256](https://github.com/ElementsProject/lightning/pull/3256))
+ - JSON API: `listconfigs` appends '...' to truncated config options. ([3268](https://github.com/ElementsProject/lightning/pull/3268))
+ - `pyln-client` now handles unicode characters in JSON-RPC requests and responses correctly. ([3018](https://github.com/ElementsProject/lightning/pull/3018))
+ - bitcoin: If bitcoind goes backwards (e.g. reindex) refuse to start (unless forced with --rescan). ([3274](https://github.com/ElementsProject/lightning/pull/3274))
+ - bug: `gossipd` crash on huge number of unknown channels. ([3273](https://github.com/ElementsProject/lightning/pull/3273))
+ - gossip: No longer discard most `node_announcements` (fixes #3194) ([3262](https://github.com/ElementsProject/lightning/pull/3262))
+ - options: We disable all dns even on startup the scan for bogus dns servers, if `--always-use-proxy` is set true ([3251](https://github.com/ElementsProject/lightning/pull/3251))
+ - protocol: "Bad commitment signature" closing channels when we sent back-to-back update_fee messages across multiple reconnects. ([3329](https://github.com/ElementsProject/lightning/pull/3329))
+ - protocol: Unlikely corner case is simultanous HTLCs near balance limits fixed. ([3286](https://github.com/ElementsProject/lightning/pull/3286))
+
+### Security
+
 ## [0.7.3] - 2019-10-18: "Bitcoin's Proof of Stake"
 
-This release named by @trueptolemy.
+This release was named by @trueptolemy.
 
 ### Added
 
@@ -69,7 +140,7 @@ Note: You should always set `allow-deprecated-apis=false` to test for changes.
 
 ## [0.7.2.1] - 2019-08-19: "Nakamoto's Pre-approval by US Congress"
 
-This release named by Antoine Poinsot @darosior.
+This release was named by Antoine Poinsot @darosior.
 
 (Technically a .1 release, as it contains last-minute fixes after 0.7.2 was tagged)
 
@@ -121,7 +192,7 @@ changes.
 
 ## [0.7.1] - 2019-06-29: "The Unfailing Twitter Consensus Algorithm"
 
-This release named by (C-Lightning Core Team member) Lisa Neigut @niftynei.
+This release was named by (C-Lightning Core Team member) Lisa Neigut @niftynei.
 
 ### Added
 
@@ -186,7 +257,7 @@ changes.
 
 ## [0.7.0] - 2019-02-28: "Actually an Altcoin"
 
-This release named by Mark Beckwith @wythe.
+This release was named by Mark Beckwith @wythe.
 
 ### Added
 
@@ -267,7 +338,7 @@ provide appropriate suffixes for JSON input fields.
 
 ## [0.6.3] - 2019-01-09: "The Smallblock Conspiracy"
 
-This release named by @molxyz and [@ctrlbreak](https://twitter.com/ctrlbreak).
+This release was named by @molxyz and [@ctrlbreak](https://twitter.com/ctrlbreak).
 
 ### Added
 
@@ -315,7 +386,7 @@ changes.
 
 ## [0.6.2] - 2018-10-20: "The Consensus Loving Nasal Daemon"
 
-This release named by practicalswift.
+This release was named by practicalswift.
 
 ### Added
 
@@ -372,7 +443,7 @@ changes.
 
 ## [0.6.1] - 2018-09-11: "Principled Opposition To Segwit"
 
-This release named by ZmnSCPxj.
+This release was named by ZmnSCPxj.
 
 ### Added
 
@@ -458,7 +529,7 @@ This release named by ZmnSCPxj.
 In the prehistory of c-lightning, no changelog was kept.  But major
 JSON API changes are tracked.
 
-This release named by Fabrice Drouin.
+This release was named by Fabrice Drouin.
 
 ### Deprecated
 
@@ -496,7 +567,8 @@ There predate the BOLT specifications, and are only of vague historic interest:
 6. [0.5.1] - 2016-10-21
 7. [0.5.2] - 2016-11-21: "Bitcoin Savings & Trust Daily Interest II"
 
-[Unreleased]: https://github.com/ElementsProject/lightning/compare/v0.7.3...HEAD
+[Unreleased]: https://github.com/ElementsProject/lightning/compare/v0.8.0...HEAD
+[0.8.0-rc1]: https://github.com/ElementsProject/lightning/releases/tag/v0.8.0-rc1
 [0.7.3]: https://github.com/ElementsProject/lightning/releases/tag/v0.7.3
 [0.7.2.1]: https://github.com/ElementsProject/lightning/releases/tag/v0.7.2.1
 [0.7.1]: https://github.com/ElementsProject/lightning/releases/tag/v0.7.1

Dockerfile

@@ -43,26 +43,29 @@ RUN mkdir /opt/groestlcoin && cd /opt/groestlcoin \
 FROM debian:stretch-slim as builder
 
 ENV LIGHTNINGD_VERSION=master
-RUN apt-get update && apt-get install -y --no-install-recommends \
-    ca-certificates \
-    autoconf \
-    automake \
-    build-essential \
-    git \
-    libtool \
-    python \
-    python3 \
-    python3-mako \
-    wget \
-    gnupg \
-    dirmngr \
-    git \
-    gettext \
-    unzip \
-    tclsh \
-    libsqlite3-dev \
-    libgmp-dev \
-    zlib1g-dev
+RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates autoconf automake build-essential git libtool python python3 python3-mako wget gnupg dirmngr git gettext
+
+RUN wget -q https://zlib.net/zlib-1.2.11.tar.gz \
+&& tar xvf zlib-1.2.11.tar.gz \
+&& cd zlib-1.2.11 \
+&& ./configure \
+&& make \
+&& make install && cd .. && rm zlib-1.2.11.tar.gz && rm -rf zlib-1.2.11
+
+RUN apt-get install -y --no-install-recommends unzip tclsh \
+&& wget -q https://www.sqlite.org/2018/sqlite-src-3260000.zip \
+&& unzip sqlite-src-3260000.zip \
+&& cd sqlite-src-3260000 \
+&& ./configure --enable-static --disable-readline --disable-threadsafe --disable-load-extension \
+&& make \
+&& make install && cd .. && rm sqlite-src-3260000.zip && rm -rf sqlite-src-3260000
+
+RUN wget -q https://gmplib.org/download/gmp/gmp-6.1.2.tar.xz \
+&& tar xvf gmp-6.1.2.tar.xz \
+&& cd gmp-6.1.2 \
+&& ./configure --disable-assembly \
+&& make \
+&& make install && cd .. && rm gmp-6.1.2.tar.xz && rm -rf gmp-6.1.2
 
 WORKDIR /opt/lightningd
 COPY . /tmp/lightning
@@ -74,12 +77,7 @@ RUN ./configure --prefix=/tmp/lightning_install --enable-static && make -j3 DEVE
 
 FROM debian:stretch-slim as final
 COPY --from=downloader /opt/tini /usr/bin/tini
-RUN apt-get update && apt-get install -y --no-install-recommends \
-    socat \
-    inotify-tools \
-    libgmp10 \
-    libsqlite3-0 \
-    zlib1g \
+RUN apt-get update && apt-get install -y --no-install-recommends socat inotify-tools \
     && rm -rf /var/lib/apt/lists/*
 
 ENV LIGHTNINGD_DATA=/root/.lightning

Makefile

@@ -15,7 +15,7 @@ CCANDIR := ccan
 
 # Where we keep the BOLT RFCs
 BOLTDIR := ../lightning-rfc/
-BOLTVERSION := 2afe3559e89520ba28b24ff5739491313217ae13
+BOLTVERSION := 35f6376f2050191081b148fb540f604092be59e1
 
 -include config.vars
 
@@ -47,7 +47,7 @@ endif
 
 ifeq ($(COMPAT),1)
 # We support compatibility with pre-0.6.
-COMPAT_CFLAGS=-DCOMPAT_V052=1 -DCOMPAT_V060=1 -DCOMPAT_V061=1 -DCOMPAT_V062=1 -DCOMPAT_V070=1
+COMPAT_CFLAGS=-DCOMPAT_V052=1 -DCOMPAT_V060=1 -DCOMPAT_V061=1 -DCOMPAT_V062=1 -DCOMPAT_V070=1 -DCOMPAT_V073=1
 endif
 
 # Timeout shortly before the 600 second travis silence timeout
@@ -263,7 +263,7 @@ ifeq ($(PYTEST),)
 	exit 1
 else
 # Explicitly hand DEVELOPER and VALGRIND so you can override on make cmd line.
-	PYTHONPATH=`pwd`/contrib/pylightning:$$PYTHONPATH TEST_DEBUG=1 DEVELOPER=$(DEVELOPER) VALGRIND=$(VALGRIND) $(PYTEST) tests/ $(PYTEST_OPTS)
+	PYTHONPATH=`pwd`/contrib/pyln-client:`pwd`/contrib/pyln-testing:`pwd`/contrib/pylightning:$$PYTHONPATH TEST_DEBUG=1 DEVELOPER=$(DEVELOPER) VALGRIND=$(VALGRIND) $(PYTEST) tests/ $(PYTEST_OPTS)
 endif
 
 # Keep includes in alpha order.
@@ -316,7 +316,7 @@ check-python:
 	@# W503: line break before binary operator
 	@flake8 --ignore=E501,E731,W503 --exclude=contrib/pylightning/lightning/__init__.py ${PYSRC}
 
-	PYTHONPATH=contrib/pylightning:$$PYTHONPATH $(PYTEST) contrib/pylightning/
+	PYTHONPATH=contrib/pyln-client:$$PYTHONPATH $(PYTEST) contrib/pyln-client/
 
 check-includes:
 	@tools/check-includes.sh

README.md

@@ -12,6 +12,7 @@ c-lightning is a lighweight, highly customizable and [standard compliant][std] i
 	* [Configuration File](#configuration-file)
 * [Further Information](#further-information)
     * [Pruning](#pruning)
+    * [HD wallet encryption](#hd-wallet-encryption)
 	* [Developers](#developers)
 
 ## Project Status
@@ -92,8 +93,6 @@ Once you've started for the first time, there's a script called
 `contrib/bootstrap-node.sh` which will connect you to other nodes on
 the lightning network.
 
-You can encrypt the BIP32 root seed (what is stored in `hsm_secret`) by passing the `--encrypted-hsm` startup argument. You can start `lightningd` with `--encrypted-hsm` on an already existing `lightning-dir` (with a not encrypted `hsm_secret`). If you pass that option, you __will not__ be able to start `lightningd` (with the same wallet) again without the password, so please beware with your password management. Also beware of not feeling too safe with an encrypted `hsm_secret`: unlike for `groestlcoind` where the wallet encryption can restrict the usage of some RPC command, `lightningd` always need to access keys from the wallet which is thus __not locked__ (yet), even with an encrypted BIP32 master seed.
-
 There are also numerous plugins available for c-lightning which add
 capabilities: in particular there's a collection at:
 
@@ -102,7 +101,14 @@ capabilities: in particular there's a collection at:
 Including [helpme][helpme-github] which guides you through setting up
 your first channels and customizing your node.
 
-### Opening a channel on the Groestlcoin testnet
+For a less reckless experience, you can encrypt the HD wallet seed:
+ see [HD wallet encryption](#hd-wallet-encryption).
+
+You can also chat to other users at [#c-lightning @ freenode.net][irc2];
+we are always happy to help you get started!
+
+
+### Opening A Channel
 
 First you need to transfer some funds to `lightningd` so that it can
 open a channel:
@@ -165,8 +171,8 @@ interfaces) for more sophisticated use.
 `lightningd` can be configured either by passing options via the command line, or via a configuration file.
 Command line options will always override the values in the configuration file.
 
-To use a configuration file, create a file named `config` within your lightning directory
-(eg. `~/.lightning/config`).  See `man -l doc/lightningd-config.5`.
+To use a configuration file, create a file named `config` within your top-level lightning directory or network subdirectory
+(eg. `~/.lightning/config` or `~/.lightning/bitcoin/config`).  See `man -l doc/lightningd-config.5`.
 
 ## Further information
 
@@ -179,6 +185,12 @@ If `groestlcoind` prunes a block that c-lightning has not processed yet, e.g., c
 In order to avoid this situation you should be monitoring the gap between c-lightning's blockheight using `lightning-cli getinfo` and `groestlcoind`'s blockheight using `groestlcoin-cli getblockchaininfo`.
 If the two blockheights drift apart it might be necessary to intervene.
 
+### HD wallet encryption
+
+You can encrypt the `hsm_secret` content (which is used to derive the HD wallet's master key) by passing the `--encrypted-hsm` startup argument, or by using the `hsmtool` (which you can find in the `tool/` directory at the root of this repo) with the `encrypt` method. You can unencrypt an encrypted `hsm_secret` using the `hsmtool` with the `decrypt` method.
+
+If you encrypt your `hsm_secret`, you will have to pass the `--encrypted-hsm` startup option to `lightningd`. Once your `hsm_secret` is encrypted, you __will not__ be able to access your funds without your password, so please beware with your password management. Also beware of not feeling too safe with an encrypted `hsm_secret`: unlike for `bitcoind` where the wallet encryption can restrict the usage of some RPC command, `lightningd` always need to access keys from the wallet which is thus __not locked__ (yet), even with an encrypted BIP32 master seed.
+
 ### Developers
 
 Developers wishing to contribute should start with the developer guide [here](doc/HACKING.md).

bitcoin/chainparams.c

@@ -1,6 +1,7 @@
 #include "chainparams.h"
 #include <ccan/array_size/array_size.h>
 #include <ccan/str/str.h>
+#include <common/utils.h>
 #include <string.h>
 
 /* Version codes for BIP32 extended keys in libwally-core.
@@ -119,6 +120,14 @@ const struct chainparams *chainparams_for_network(const char *network_name)
 	return NULL;
 }
 
+const struct chainparams **chainparams_for_networks(const tal_t *ctx)
+{
+	const struct chainparams **params = tal_arr(ctx, const struct chainparams*, 0);
+	for (size_t i = 0; i < ARRAY_SIZE(networks); i++)
+		tal_arr_expand(&params, &networks[i]);
+	return params;
+}
+
 const struct chainparams *chainparams_by_chainhash(const struct bitcoin_blkid *chain_hash)
 {
 	for (size_t i = 0; i < ARRAY_SIZE(networks); i++) {

bitcoin/chainparams.h

@@ -46,6 +46,12 @@ struct chainparams {
  */
 const struct chainparams *chainparams_for_network(const char *network_name);
 
+/**
+ * chainparams_for_networks - Get blockchain parameters for all known networks,
+ *                            as a tal array.
+ */
+const struct chainparams **chainparams_for_networks(const tal_t *ctx);
+
 /**
  * chainparams_by_bip173 - Helper to get a network by its bip173 name
  *