feat: command to convert `peers.txt` into `addrbook.json` #2623

rootulp · 2023-10-06T22:07:09Z

Description

Adds a new command to celestia-appd called addrbook.

$ ./build/celestia-appd addrbook --help
Convert a list of peers into an address book.
The first argument (peers.txt) should contain a new line separated list of peers. The format for a peer is `id@ip:port` or `id@domain:port`.
The second argument (addrbook.json) should contain the complete file path, including both the directory path and the desired output file name, in the following format: `path/to/filename`. The address book is saved to the output file in JSON format.

Usage:
  celestia-appd addrbook peers.txt addrbook.json [flags]

Testing

I was able to run a node on arabica-10 with an addrbook.json that was created from arabica-10/peers.txt. Note, testing failed on mocha-4 due to celestiaorg/networks#349

rootulp · 2023-10-06T22:11:35Z

Running using a modified mocha-4.sh script and encountered

panic: Error reading file /Users/rootulp/.celestia-app/config/addrbook.json: invalid IP address: rpc-1.celestia.nodes.guru

codecov-commenter · 2023-10-06T22:23:15Z

Codecov Report

Merging #2623 (f51ece6) into main (819bb25) will decrease coverage by 0.31%.
Report is 6 commits behind head on main.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #2623      +/-   ##
==========================================
- Coverage   19.93%   19.62%   -0.31%     
==========================================
  Files         139      139              
  Lines       16702    16999     +297     
==========================================
+ Hits         3329     3336       +7     
- Misses      13051    13341     +290     
  Partials      322      322

see 5 files with indirect coverage changes

rootulp · 2023-10-06T22:26:18Z

6:19PM INF Add our address to book addr={"id":"3e3b5452509b2ee02802c4bbd7a1b7636d2d7ef0","ip":"0.0.0.0","port":26656} book=/Users/rootulp/.celestia-app/config/addrbook.json module=p2p
...
6:19PM INF service start book=/Users/rootulp/.celestia-app/config/addrbook.json impl=AddrBook module=p2p msg={}
6:19PM INF Saving AddrBook to file book=/Users/rootulp/.celestia-app/config/addrbook.json module=p2p size=4
6:19PM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
6:19PM INF starting API server... module=api-server
6:19PM INF serve module=api-server msg={}

just hanging here... so I don't think it's actually connecting to any of these peers

still hangs

rootulp · 2023-10-09T14:22:10Z

Moved this from Mainnet milestone to Post-Mainnet milestone b/c it doesn't need to be merged to celestia-app prior to cutting the v1.x release.

~~We may not even need to merge it if it's a one-off tool and only used for mainnet.~~ Let's plan on merging this PR so that it may be used by future testnets.

rootulp · 2023-10-10T14:41:02Z

Enabled debug logs

10:40AM INF service start book=/Users/rootulp/.celestia-app/config/addrbook.json impl=AddrBook module=p2p msg={}
10:40AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
10:40AM DBG Consensus ticker inbound=0 module=blockchain numPending=0 outbound=0 total=0
10:40AM DBG Blockpool has no peers module=blockchain
10:40AM DBG Consensus ticker inbound=0 module=blockchain numPending=0 outbound=0 total=0
10:40AM DBG Blockpool has no peers module=blockchain
10:40AM DBG Consensus ticker inbound=0 module=blockchain numPending=0 outbound=0 total=0
10:40AM DBG Blockpool has no peers module=blockchain

rootulp · 2023-10-10T14:51:17Z

Was able to get the node to start by specifying:

A monotonically increasing bucket value for each peer in addrbook.json (e.g. "buckets": [1], "buckets": [2], etc.)
Bucket type of new for each peer in addrbook.json (e.g. "bucket_type": 1)
And then observing that after the node starts up and connects to these peers, it modifies the buckets assigned to each peer in the addrbook.json so that they end up varying (e.g. "buckets": [4, 91]).

rootulp · 2023-10-10T15:18:47Z

Testing again and encountering a lot of this

11:17AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
11:18AM INF Error reconnecting to peer. Trying again addr={"id":"f9e950870eccdb40e2386896d7b6a7687a103c99","ip":"88.99.219.120","port":43656} err="auth failure: secret conn failed: read tcp 10.239.122.14:64466->88.99.219.120:43656: i/o timeout" module=p2p tries=7
11:18AM INF Error reconnecting to peer. Trying again addr={"id":"daf2cecee2bd7f1b3bf94839f993f807c6b15fbf","ip":"65.109.92.79","port":11656} err="auth failure: secret conn failed: read tcp 10.239.122.14:64468->65.109.92.79:11656: i/o timeout" module=p2p tries=3
11:18AM INF Error reconnecting to peer. Trying again addr={"id":"43e9da043318a4ea0141259c17fcb06ecff816af","ip":"141.94.73.39","port":43656} err="auth failure: secret conn failed: read tcp 10.239.122.14:64469->141.94.73.39:43656: i/o timeout" module=p2p tries=3
11:18AM INF Error reconnecting to peer. Trying again addr={"id":"f9e950870eccdb40e2386896d7b6a7687a103c99","ip":"88.99.219.120","port":43656} err="auth failure: secret conn failed: read tcp 10.239.122.14:64472->88.99.219.120:43656: i/o timeout" module=p2p tries=8

Inspired by celestiaorg/networks#351

rootulp · 2023-10-10T15:24:05Z

Relevant: celestiaorg/networks#349

rootulp · 2023-10-10T17:59:43Z

Ayy this works for arabica-10:

1:59PM INF indexed block exents height=1766 module=txindex
1:59PM INF minted coins from module account amount=28537864698497544652899591utia from=mint module=x/bank
1:59PM INF executed block height=1767 module=state num_invalid_txs=0 num_valid_txs=0
1:59PM INF commit synced commit=436F6D6D697449447B5B323420323430203537203832203131332036312031322039362039382032333020393820323336203232382032323320392031383520323131203720313535203536203930203139362031333120313836203232342031323420313233203137372032303720323435203134362037335D3A3645377D
1:59PM INF committed state app_hash=18F03952713D0C6062E662ECE4DF09B9D3079B385AC483BAE07C7BB1CFF59249 height=1767 module=state num_txs=0
...

rach-id

LGTM.

Would it make sense to add more documentation to this? For example, in which format the input file should be and also the output

rootulp · 2023-10-20T17:18:39Z

Yea can add more docs pre merge, good point! I had a README.md before but deleted it in the migration to be a subcommand under celestia-appd.

staheri14

LGTM, included some suggestions.

cmd/celestia-appd/cmd/addrbook.go

Co-authored-by: Sanaz Taheri <[email protected]>

staheri14

LGTM!

…g#2623) Closes celestiaorg#2619 ## Description Adds a new command to celestia-appd called `addrbook`. ``` ./build/celestia-appd addrbook --help Convert a list of peers into an address book. The first argument (peers.txt) should contain a new line separated list of peers. The format for a peer is `id@ip:port` or `id@domain:port`. The second argument (addrbook.json) should be the output filename. The address book is saved to the output file in JSON format. Usage: celestia-appd addrbook peers.txt addrbook.json [flags] ``` ## Testing I was able to run a node on arabica-10 with an `addrbook.json` that was created from [arabica-10/peers.txt](https://github.com/celestiaorg/networks/blob/master/arabica-10/peers.txt). Note, testing failed on mocha-4 due to celestiaorg/networks#349

Backports #2623 and #2783 ## Testing ``` $ ./build/celestia-appd addrbook ~/git/rootulp/celestia/networks/celestia/peers.txt ~/Downloads/addrbook.json Error adding [email protected]:26656: Cannot add non-routable address [email protected]:26656 Converted /Users/rootulp/git/rootulp/celestia/networks/celestia/peers.txt into /Users/rootulp/Downloads/addrbook.json $ head -n 10 /Users/rootulp/Downloads/addrbook.json { "key": "8957397890a66cd0f716cbc9", "addrs": [ { "addr": { "id": "a41b4bc451b1b71d537aa1018226e08d7fa7e44e", "ip": "5.255.77.44", "port": 26656 }, "src": { ```

feat: tool to generate addrbook.json

bddd9e4

rootulp added the enhancement New feature or request label Oct 6, 2023

rootulp added this to the Mainnet milestone Oct 6, 2023

rootulp self-assigned this Oct 6, 2023

feat: convert domains to IP addresses

7d2afc2

rootulp added 4 commits October 6, 2023 18:28

debug: attempt to add seeds to peers list

1718d01

still hangs

fix: need to populate remainder of Entry with empty fields

25afcf3

debug: attempt to populate empty buckets

9e141a0

debug: check mocha-4 script into version control

00af9c9

rootulp modified the milestones: Mainnet, Post-mainnet Oct 9, 2023

feat: rand bucket index + bucket type new

d171f0f

debug: add AlexToTheMoon peer

1ead5b7

Inspired by celestiaorg/networks#351

rootulp added 2 commits October 10, 2023 13:50

debug: arabica-10 script

71aab84

debug: works for arabica-10

ea78914

rootulp added 3 commits October 10, 2023 14:02

fix: name of output file

6bac3f4

remove scripts used for testing

ff1b2fc

fix: lint

84429b4

rootulp marked this pull request as ready for review October 10, 2023 18:17

rootulp requested review from evan-forbes and cmwaters as code owners October 10, 2023 18:17

rootulp requested review from staheri14 and rach-id October 20, 2023 15:18

rootulp modified the milestones: Post-mainnet, Mainnet Oct 20, 2023

rach-id previously approved these changes Oct 20, 2023

View reviewed changes

docs: add more docs to command

dc49124

rootulp dismissed stale reviews from rach-id and evan-forbes via dc49124 October 20, 2023 17:32

rootulp requested review from evan-forbes and rach-id October 20, 2023 17:32

celestia-bot requested a review from a team October 20, 2023 17:32

rootulp enabled auto-merge (squash) October 20, 2023 17:32

staheri14 previously approved these changes Oct 20, 2023

View reviewed changes

cmd/celestia-appd/cmd/addrbook.go Outdated Show resolved Hide resolved

cmd/celestia-appd/cmd/addrbook.go Outdated Show resolved Hide resolved

rootulp dismissed staheri14’s stale review via f51ece6 October 20, 2023 20:16

Update cmd/celestia-appd/cmd/addrbook.go

f51ece6

Co-authored-by: Sanaz Taheri <[email protected]>

celestia-bot requested a review from a team October 20, 2023 20:16

rootulp requested a review from staheri14 October 20, 2023 20:19

staheri14 previously approved these changes Oct 20, 2023

View reviewed changes

refactor: use default config for routabilityStrict

6a9f28e

rootulp dismissed staheri14’s stale review via 6a9f28e October 20, 2023 20:33

celestia-bot requested a review from a team October 20, 2023 20:33

rootulp requested a review from staheri14 October 20, 2023 20:34

staheri14 approved these changes Oct 20, 2023

View reviewed changes

rach-id approved these changes Oct 20, 2023

View reviewed changes

rootulp merged commit 03beaef into celestiaorg:main Oct 20, 2023
28 checks passed

rootulp mentioned this pull request Oct 31, 2023

feat: backport addrbook command (backport #2623 and #2783) #2796

Merged

rootulp deleted the rp/address-book branch February 5, 2024 13:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: command to convert `peers.txt` into `addrbook.json` #2623

feat: command to convert `peers.txt` into `addrbook.json` #2623

rootulp commented Oct 6, 2023 •

edited

Loading

rootulp commented Oct 6, 2023

codecov-commenter commented Oct 6, 2023 •

edited

Loading

rootulp commented Oct 6, 2023

rootulp commented Oct 9, 2023 •

edited

Loading

rootulp commented Oct 10, 2023

rootulp commented Oct 10, 2023 •

edited

Loading

rootulp commented Oct 10, 2023

rootulp commented Oct 10, 2023

rootulp commented Oct 10, 2023

rach-id left a comment

rootulp commented Oct 20, 2023

staheri14 left a comment

staheri14 left a comment

feat: command to convert peers.txt into addrbook.json #2623

feat: command to convert peers.txt into addrbook.json #2623

Conversation

rootulp commented Oct 6, 2023 • edited Loading

Description

Testing

rootulp commented Oct 6, 2023

codecov-commenter commented Oct 6, 2023 • edited Loading

Codecov Report

rootulp commented Oct 6, 2023

rootulp commented Oct 9, 2023 • edited Loading

rootulp commented Oct 10, 2023

rootulp commented Oct 10, 2023 • edited Loading

rootulp commented Oct 10, 2023

rootulp commented Oct 10, 2023

rootulp commented Oct 10, 2023

rach-id left a comment

Choose a reason for hiding this comment

rootulp commented Oct 20, 2023

staheri14 left a comment

Choose a reason for hiding this comment

staheri14 left a comment

Choose a reason for hiding this comment

feat: command to convert `peers.txt` into `addrbook.json` #2623

feat: command to convert `peers.txt` into `addrbook.json` #2623

rootulp commented Oct 6, 2023 •

edited

Loading

codecov-commenter commented Oct 6, 2023 •

edited

Loading

rootulp commented Oct 9, 2023 •

edited

Loading

rootulp commented Oct 10, 2023 •

edited

Loading