2. Reading Version Messages.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading Version Messages\n",
    "\n",
    "In the last lesson we encountered the Bitcoin protocol's [Version Handshake](https://en.bitcoin.it/wiki/Version_Handshake). We saw how Bitcoin network peers won't respond if you don't start the conversion with a `version` message.\n",
    "\n",
    "But _we cheated_. I gave you a serialized `version` message and didn't tell you how I created it.\n",
    "\n",
    "_We were lazy_: we didn't parse the cryptic `payload` of the `version` message that our peer sent us.\n",
    "\n",
    "_We were rude_! After listening for our peer's `version` message we stopped listening and never received or responded to their `verack` message -- completing the handshake. Our peer was left hanging ...\n",
    "\n",
    "So you see, we have much to fix!\n",
    "\n",
    "### Housekeeping\n",
    "\n",
    "In the root of the `bitcoincorps` project directory you will now see a `ibd` (\"initial block download\") folder, and inside it a `one` folder. This folder represents all the code we wrote during the first lesson. With each now lesson, another such folder will show up. By the end you will have an `ibd` python package which does initial block download.\n",
    "\n",
    "Check out the complete code from the first lesson: [`ibd/one/complete.py`](./ibd/one/complete.py). I changed a few thing, which I'll point out as we go."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the code from lesson 1\n",
    "from ibd.one.complete import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Jupyter has an [\"autoreload extensioin\"](https://ipython.readthedocs.io/en/stable/config/extensions/autoreload.html?highlight=reload#module-IPython.extensions.autoreload) that allows us to reimport `.py` files when we change them. Now that we have added a library of `.py` files this will be a helpful thing for us to do at the beginning of each lesson."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding Our Place\n",
    "\n",
    "Now we're back where we left off last time: we can send a hard-coded `version` message to a Bitcoin peer and make sense of the outermost attributes of the binary response we receive: \"network magic\", \"command\", \"payload length\", \"payload checksum\" and \"payload\" itself. But the information contained within the `payload` remains hidden from us.\n",
    "\n",
    "### Lesson 2 Objective\n",
    "\n",
    "The goal of this lesson is to make sense of the payload you see after executing the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import socket\n",
    "\n",
    "PEER_IP = \"35.198.151.21\"\n",
    "PEER_PORT = 8333\n",
    "\n",
    "# magic \"version\" bytestring\n",
    "VERSION = b'\\xf9\\xbe\\xb4\\xd9version\\x00\\x00\\x00\\x00\\x00j\\x00\\x00\\x00\\x9b\"\\x8b\\x9e\\x7f\\x11\\x01\\x00\\x0f\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x93AU[\\x00\\x00\\x00\\x00\\x0f\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0f\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00rV\\xc5C\\x9b:\\xea\\x89\\x14/some-cool-software/\\x01\\x00\\x00\\x00\\x01'\n",
    "\n",
    "sock = socket.socket()\n",
    "sock.connect((PEER_IP, PEER_PORT))\n",
    "\n",
    "# initiate the \"version handshake\"\n",
    "sock.send(VERSION)\n",
    "\n",
    "# receive their \"version\" response\n",
    "version_message = Packet.from_socket(sock)\n",
    "\n",
    "print(version_message)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some Refactoring\n",
    "\n",
    "Almost the same as last lesson except for:\n",
    "* I renamed `Message` to `Packet` because we're going to start defining things like `VersionMessage` and `VerackMessage` and so this seemed a little more clear.\n",
    "* `Packet` instances look like pretty tables when printed. This is because I added a `Network.__str__` method to [ibd/one/complete.py](./ibd/one/complete.py). `object.__str__` is the function Python calls when determining _how_ to print a given `object`. This `Packet.__str__` method simply runs a few of its values through this [tabulate](https://bitbucket.org/astanin/python-tabulate) program, which I added to our requirements.txt file.\n",
    "\n",
    "### The Payload\n",
    "\n",
    "Our next task is to parse this payload. Besides the \"/Satoshi:0.16.0/\" -- clearly a user agent -- the rest of the payload isn't human readable.\n",
    "\n",
    "But have no fear -- we will decode the message payload in the same manner as we decoded the overall message structure in our `Packet.from_socket` method from last class.\n",
    "\n",
    "[This chart](https://en.bitcoin.it/wiki/Protocol_documentation#version) from the protocol documentation will act as our blueprint. A reprint:\n",
    "\n",
    "![image](./images/version-message.png)\n",
    "\n",
    "### Old Types\n",
    "\n",
    "Here we encounter some \"types\" we are now familiar with from the first lesson -- `int32_t` / `uint64_t` / `int64_t` -- which are different types in a \"low-level\" language like C++, but are all equivalent to the `int` type in Python. Our previously implemented `bytes_to_int` can handle these just fine.\n",
    "\n",
    "### New Types\n",
    "\n",
    "But we also encounter some new types: `net_addr`, `varstr`, and `bool`. \n",
    "\n",
    "Even worse, if we click on the [`varstr` link](https://en.bitcoin.it/wiki/Protocol_documentation#Variable_length_string) we see that it contains one additional type: `varint`. \n",
    "\n",
    "Worse still, the [`net_addr` link](https://en.bitcoin.it/wiki/Protocol_documentation#Network_address) contains `time`, `services` and `IPv6/4` fields nominally of types `uint32`, `uint64_t` and `char[16]` but in order for us to make sense of what they hell them mean each requires parsing: the `time` integer as a Unix timestamp, the `services` integer as a damn \"bitfield\" (whatever that is!), and `IPv6/4` IP address as a 16 digit bytestring where the first 12 digits are always `00 00 00 00 00 00 00 00 00 00 FF FF` and only the last 4 matter! \n",
    "\n",
    "Oh, and remember how I mentioned that Satoshi usually, but not always, encoded his integers in \"little endian\" byte order (least significant digits is on the left)? Well, the `port` attribute of `net_addr` is encoded \"big endian\", where the *most* significant digit is on the left. Yes, the exact opposite of everything else!!!\n",
    "\n",
    "Hunker down for a looooooong lesson!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# VersionMessage\n",
    "\n",
    "Here's the outline of a `VersionMessage` class which will abstract the `version` [message type](https://en.bitcoin.it/wiki/Protocol_documentation#version):\n",
    "\n",
    "* It has an `__init__` constructor method, which allows us to pass a different set of `version`, `services`, `timestamp`, `addr_recv`, `addr_from`, `nonce`, `user_agent`, `start_height`, and `relay` values to each new instance.\n",
    "* It will have a hard-coded [\"class variable\"](https://www.toptal.com/python/python-class-attributes-an-overly-thorough-guide) of `command = b\"version\"`. With this decision we are setting a convention: any instance of this class or the other 26 `Message` classes we still have to implement will have a `msg.command` attribute  to tell us what kind of message we're dealing with.\n",
    "* `VersionMessage.from_bytes` is also a convention that all 26 other `Message` classes will implement. Let's assume we are trying to handle an incoming `Packet` instance, which we will call `pkt`. We observe that `pkt.command` is `b\"version`, so we're dealing with a version message and need to turn `pkt.payload` into an instance of the `VersionMessage` class defined in the cell above. The purpose of this `VersionMessage.from_bytes` classmethod is facilitate this: `msg = VersionMessage.from_bytes(pkt.payload)`. It's magic!\n",
    "* We will do some operations many times -- such as reading `n` bytes and interpreting them as a Python `int` -- so it makes sense to implement so-called \"helper methods\" to simplify our code, make it more testable and readable. `read_int`, `read_var_str`, `read_var_int` and `read_bool` are some such methods waiting to be implemented."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_int(stream, n):\n",
    "    raise NotImplementedError()\n",
    "    \n",
    "def read_var_int(stream):\n",
    "    raise NotImplementedError()\n",
    "    \n",
    "def read_var_str(stream):\n",
    "    raise NotImplementedError()\n",
    "    \n",
    "def read_bool(stream):\n",
    "    raise NotImplementedError()\n",
    "    \n",
    "class VersionMessage:\n",
    "\n",
    "    command = b\"version\"\n",
    "\n",
    "    def __init__(self, version, services, timestamp, addr_recv, addr_from, \n",
    "                 nonce, user_agent, start_height, relay):\n",
    "        self.version = version\n",
    "        self.services = services\n",
    "        self.timestamp = timestamp\n",
    "        self.addr_recv = addr_recv\n",
    "        self.addr_from = addr_from\n",
    "        self.nonce = nonce\n",
    "        self.user_agent = user_agent\n",
    "        self.start_height = start_height\n",
    "        self.relay = relay\n",
    "\n",
    "    @classmethod\n",
    "    def from_bytes(cls, payload):\n",
    "        stream = BytesIO(payload)\n",
    "        \n",
    "        version = read_int(stream, 4)\n",
    "        services = read_int(stream, 8)\n",
    "        timestamp = read_int(stream, 8)\n",
    "        addr_recv = stream.read(26)\n",
    "        addr_from = stream.read(26)\n",
    "        nonce = read_int(stream, 8)\n",
    "        user_agent = read_var_str(stream)\n",
    "        start_height = read_int(stream, 4)\n",
    "        relay = read_bool(stream)\n",
    "\n",
    "        return cls(version, services, timestamp, addr_recv, addr_from, \n",
    "                   nonce, user_agent, start_height, relay)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# \"Integer\" Fields\n",
    "\n",
    "In the last lesson we implemented `bytes_to_int(n)`. We'll start this lesson by implementing a small helper method `read_int(stream)` atop `bytes_to_int(n)` which first reads `n` from `stream` and then calls `bytes_to_int` with the bytes it read.\n",
    "\n",
    "And we're going to create an argument `byte_order`, which defaults to `little`, because almost every integer our program deals with will be little-endian encoded. But IP ports -- and soon other -- are big-endian encoded so we must allow callers to override this `bytes_order=\"little\"` default value if they have a big-endian endcoded integer on their hands.\n",
    "\n",
    "### Exercise #1: Implement `read_int(stream, n)`\n",
    "\n",
    "Read `n` bytes and interpret it as an `int` with `byte_order` byte-order:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This cell is just some reminders of what kind of data we're dealing with\n",
    "# Exercise is in the next cell ...\n",
    "\n",
    "# stream will always be a BytesIO instance:\n",
    "from io import BytesIO\n",
    "stream = BytesIO(b'\\x7f\\x11\\x01\\x00\\r\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\xb4\\x9dZ[\\x00\\x00\\x00\\x00\\x0f\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\r\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00b\\x8f\\xc9N~]\\x00\\xb2\\x10/Satoshi:0.16.0/p%\\x08\\x00\\x01')\n",
    "\n",
    "# reminder: this is how you read 10 bytes from a stream / BytesIO instance:\n",
    "stream.read(10) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_int(stream, n, byte_order='little'):\n",
    "    # step 1: read the correct number of bytes from `stream` according to \"field size\" column above\n",
    "    # step 2: interpret these bytes according to the \"data type\" column above\n",
    "    ### replace `raise NotImplementedError()` with your code ###\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ipytest, pytest\n",
    "from io import BytesIO\n",
    "\n",
    "data = [\n",
    "    # number, number of bytes used to encode number, byte-order used to encode number\n",
    "    [22, 10, 'little'],\n",
    "    [1_000_000, 7, 'big'],\n",
    "]\n",
    "\n",
    "def test_read_int_0():\n",
    "    for number, num_bytes, byte_order in data:\n",
    "        bytes_ = number.to_bytes(num_bytes, byte_order)\n",
    "        stream = BytesIO(bytes_)\n",
    "        result = read_int(stream, num_bytes, byte_order)\n",
    "        assert number == result\n",
    "    \n",
    "ipytest.run_tests(doctest=True)\n",
    "ipytest.clean_tests(\"test_read_int*\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### To Help You Cheat\n",
    "\n",
    "If you can't get the tests to pass, example solutions for all exercises can be found in [ibd/two/answers.py](./ibd/two/answers.py)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise #2: Read the version field contained within the payload of the version message (a mouthful, I know!)\n",
    "\n",
    "hint: use `read_int` + protocol docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_version(stream):\n",
    "    ### replace `raise NotImplementedError()` with your code ###\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ipytest, pytest\n",
    "import test_data\n",
    "\n",
    "version_streams = test_data.make_version_streams()\n",
    "\n",
    "def test_read_version_0():\n",
    "    n = read_version(version_streams[0])\n",
    "    assert n == 70015\n",
    "\n",
    "def test_read_version_1():\n",
    "    n = read_version(version_streams[1])\n",
    "    assert n == 60001\n",
    "\n",
    "def test_read_version_2():\n",
    "    n = read_version(version_streams[2])\n",
    "    assert n == 106\n",
    "    \n",
    "ipytest.run_tests(doctest=True)\n",
    "ipytest.clean_tests(\"test_read_version*\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Exercise #3: Given a version message binary stream, tell me whether the node that sent it can send a `pong` message \n",
    "\n",
    "This exercise should give you a taste of the kind of information the version number encodes. [This table](https://bitcoin.org/en/developer-reference#protocol-versions) will show you the way!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def can_send_pong(stream):\n",
    "    ### replace `raise NotImplementedError()` with your code ###\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "version_streams = test_data.make_version_streams()\n",
    "\n",
    "def test_can_send_pong_0():\n",
    "    result = can_send_pong(version_streams[0])\n",
    "    assert result == True\n",
    "\n",
    "def test_can_send_pong_1():\n",
    "    result = can_send_pong(version_streams[1])\n",
    "    assert result == True\n",
    "\n",
    "def test_can_send_pong_2():\n",
    "    result = can_send_pong(version_streams[2])\n",
    "    assert result == False\n",
    "    \n",
    "ipytest.run_tests(doctest=True)\n",
    "ipytest.clean_tests(\"test_can_send_pong*\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# \"Boolean\" Fields\n",
    "\n",
    "After the `int32_t` / `uint64_t` / `int64_t` integer-type fields, `bool` is the next simplest: it's a `1` or it's `0`. Actually, it's even simpler, huh? But we're going to resuse the code above so I'm introducing it second.\n",
    "\n",
    "In fact, we could just use `read_int` and pass around `1`'s and `0`'s and our program would work just fine. After all, the statement `1 == True and 0 == False` evaluates to `True` in Python. But Python gives us a built-in `bool` class for dealing with true-or-false, 1-or-0 values because it gives our programs greater clarity and readability. Let's use it.\n",
    "\n",
    "### Exercise #4: implement `read_bool`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_bool(stream):\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import test_data\n",
    "\n",
    "def test_read_bool_0():\n",
    "    stream = test_data.make_stream(test_data.true_bytes)\n",
    "    result = read_bool(stream)\n",
    "    assert type(result) == bool\n",
    "    assert result is True\n",
    "    \n",
    "def test_read_bool_1():\n",
    "    stream = test_data.make_stream(test_data.false_bytes)\n",
    "    result = read_bool(stream)\n",
    "    assert type(result) == bool\n",
    "    assert result is False\n",
    "    \n",
    "ipytest.run_tests(doctest=True)\n",
    "ipytest.clean_tests(\"test_read_bool_*\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# \"Timestamp\" Fields\n",
    "\n",
    "Network messages use [\"Unix timestamps\"](https://en.wikipedia.org/wiki/Unix_time) whenever they communicate some notion of \"time\". \"Unix time\" is just a running count of the number of seconds elapsed since the start of the year 1970 -- so it is represented as an integer.\n",
    "\n",
    "Here's how we interpret a Unix timestamp in Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "def read_timestamp(stream):\n",
    "    timestamp = read_int(stream, 8)\n",
    "    return datetime.fromtimestamp(timestamp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# \"Variable Length\" fields\n",
    "\n",
    "Next comes `var_str`, the type of the \"User Agent\", which is basically an advertisement of the Bitcoin software implementation that the node is using. You can see a listing of popular values [here](https://bitnodes.earn.com/nodes/).\n",
    "\n",
    "[\"Variable Length Strings\"](https://en.bitcoin.it/wiki/Protocol_documentation#Variable_length_string) are used for string fields of unpredictible length. This technique strives to use only the space it needs. It does so by prepending a \"variable length integer\" in front of the string value being communicated, which tells the receiver how many bytes they should read in order to read the encoded string value. This is kind of similar to how the payload bytes are handled in our `Packet.from_socket` -- first we read `length` and then we read `length`-many bytes to get our raw payload. Same idea here, but now the length of the string isn't an integer, but a \"variable length integer\".\n",
    "\n",
    "How does this `var_int` work?\n",
    "\n",
    "The first byte of a `var_int` is a marker which says how many bytes come after it:\n",
    "* `0xFF`: 8 byte integer follows\n",
    "* `0xFE`: 4 byte integer follows\n",
    "* `0xFD`: 2 byte integer follows\n",
    "* < `0xFD`: 0 bytes follow. Interpret first byte as a 1 byte integer.\n",
    "\n",
    "### Exercise #5:  Implement `read_var_int`, since `read_var_str` will depend on it.\n",
    "\n",
    "Since this is a somewhat complicated function, I've outlined it for you. Replace the `\"FIXME\"`s:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_var_int(stream):\n",
    "    i = read_int(stream, 1)\n",
    "    if i == 0xff:\n",
    "        return read_int(stream, 8)\n",
    "    elif i == 0xfe:\n",
    "        return \"FIXME\"\n",
    "    elif \"FIXME\":\n",
    "        return \"FIXME\"\n",
    "    else:\n",
    "        \"FIXME\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ipytest, pytest\n",
    "import test_data as td\n",
    "\n",
    "enumerated = (\n",
    "    (td.eight_byte_int, td.eight_byte_var_int),\n",
    "    (td.four_byte_int, td.four_byte_var_int),\n",
    "    (td.two_byte_int, td.two_byte_var_int),\n",
    "    (td.one_byte_int, td.one_byte_var_int),\n",
    ")\n",
    "\n",
    "def test_read_var_int():\n",
    "    for correct_int, var_int in enumerated:\n",
    "        stream = td.make_stream(var_int)\n",
    "        calculated_int = read_var_int(stream)\n",
    "        assert correct_int == calculated_int\n",
    "\n",
    "ipytest.run_tests(doctest=True)\n",
    "ipytest.clean_tests(\"test_read_var_int*\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have that out of the way:\n",
    "\n",
    "### Exercise #6: Implement `read_var_str`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_var_str(stream):\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ipytest, pytest\n",
    "import test_data as td\n",
    "\n",
    "enumerated = (\n",
    "    (td.short_str, td.short_var_str),\n",
    "    (td.long_str, td.long_var_str),\n",
    ")\n",
    "\n",
    "def test_read_var_str():\n",
    "    for correct_byte_str, var_str in enumerated:\n",
    "        stream = td.make_stream(var_str)\n",
    "        calculated_byte_str = read_var_str(stream)\n",
    "        assert correct_byte_str == calculated_byte_str\n",
    "\n",
    "ipytest.run_tests(doctest=True)\n",
    "ipytest.clean_tests(\"test_read_var_str*\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# \"Services\" Field\n",
    "\n",
    "[The version section of the protocol docs](https://en.bitcoin.it/wiki/Protocol_documentation#version) provides us with the following guide for interpreting the `services` field of the `version` payload:\n",
    "\n",
    "![image](images/services.png)\n",
    "\n",
    "The type of this field is \"bitfield\". [Check out the wikipedia entry](https://en.wikipedia.org/wiki/Bit_field) for a more detailed explanation that I can provide.\n",
    "\n",
    "A bitfield is an integer. Every bit of the base-2 representation (e.g. \"101\" is base-2 representation of 5) holds some pre-defined meaning. This particular bitfield is 8 bytes / 64 bits (remember, a byte is just a collection of 8 bits so 8 bytes is 8*8=64 bits).\n",
    "\n",
    "From the table above we can see that the least significant digit in the binary representation (decimal value `2^0=1`) represents `NODE_NETWORK`, or whether this peer \"can be asked for full blocks or just headers\".\n",
    "\n",
    "The second least-significant digit (decimal value `2^1=2`): `NODE_GETUTXO`\n",
    "\n",
    "The third least-significant digit (decimal value `2^2=4`): `NODE_BLOOM`\n",
    "\n",
    "The fourth least-significant digit (decimal value `2^3=8`): `NODE_WITNESS`\n",
    "\n",
    "The eleventh least-significant digit (decimal value `2^10=1024`): `NODE_NETWORK_LIMITED`\n",
    "\n",
    "The rest of the bits (decimal values `2*n` where n in {4, 5, 6, 7, 8, 9, 11, 12, ..., 63} have no meaning, yet.\n",
    "\n",
    "So, in order to interpret this field we need to look up the nth bit in the table above and see if it means anything.\n",
    "\n",
    "So, our Python code could produce a dictionary like this for every node we connect to. This would allow us to look up what services that node offers _by name_ (which is why it's called a dictionary!):\n",
    "\n",
    "```\n",
    "{\n",
    "    'NODE_NETWORK': True,\n",
    "    'NODE_GETUTXO': False,\n",
    "    'NODE_BLOOM': True,\n",
    "    'NODE_WITNESS': False,\n",
    "    'NODE_NETWORK_LIMITED': True,\n",
    "}\n",
    "```\n",
    "\n",
    "Furthermore, we could write a function that produces this lookup table for us given an integer bitfield and a magical `check_bit(n)` function:\n",
    "\n",
    "```\n",
    "def read_services(stream):\n",
    "    n = read_int(stream, 4)\n",
    "    return {\n",
    "        'NODE_NETWORK': check_bit(services_int, 0),           # 1    = 2**0\n",
    "        'NODE_GETUTXO': check_bit(services_int, 1),           # 2    = 2**1\n",
    "        'NODE_BLOOM': check_bit(services_int, 2),             # 4    = 2**2\n",
    "        'NODE_WITNESS': check_bit(services_int, 3),           # 8    = 2**3\n",
    "        'NODE_NETWORK_LIMITED': check_bit(services_int, 10),  # 1024 = 2**10\n",
    "    }\n",
    "```\n",
    "\n",
    "For now, I'm just going to give you a definition of the magical `check_bit` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_bit(number, index):\n",
    "    \"\"\"See if the bit at `index` in binary representation of `number` is on\"\"\"\n",
    "    mask = 1 << index\n",
    "    return bool(number & mask)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise #7: Fill out the remainder of the `services_int_to_dict` and `read_services` functions:\n",
    "\n",
    "Replace each occurrence of `FIXME` with correct numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def services_int_to_dict(services_int):\n",
    "    return {\n",
    "        'NODE_NETWORK': check_bit(services_int, \"FIXME\"),\n",
    "        'NODE_GETUTXO': check_bit(services_int, \"FIXME\"),\n",
    "        'NODE_BLOOM': check_bit(services_int, \"FIXME\"),\n",
    "        'NODE_WITNESS': check_bit(services_int, \"FIXME\"),\n",
    "        'NODE_NETWORK_LIMITED': check_bit(services_int, \"FIXME\"),\n",
    "    }\n",
    "\n",
    "def read_services(stream):\n",
    "    services_int = read_int(stream, \"FIXME\")\n",
    "    return \"FIXME\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ipytest, pytest\n",
    "import test_data as td\n",
    "\n",
    "def test_read_services():\n",
    "    services = 1 + 2 + 4 + 1024\n",
    "    answer = {\n",
    "        'NODE_NETWORK': True,\n",
    "        'NODE_GETUTXO': True,\n",
    "        'NODE_BLOOM': True,\n",
    "        'NODE_WITNESS': False,\n",
    "        'NODE_NETWORK_LIMITED': True,\n",
    "    }\n",
    "    stream = BytesIO(int_to_bytes(services, 8))\n",
    "    assert read_services(stream) == answer\n",
    "\n",
    "ipytest.run_tests(doctest=True)\n",
    "ipytest.clean_tests(\"test_read_services*\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To give you a better idea what's going on here, check out these `read_services` outputs for some possible inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint\n",
    "\n",
    "bitfields = [\n",
    "    1,\n",
    "    8,\n",
    "    1 + 8,\n",
    "    1024,\n",
    "    8 + 1024,\n",
    "    1 + 2 + 4 + 8 + 1024,\n",
    "    2**5 + 2**9 + 2**25,\n",
    "]\n",
    "\n",
    "for bitfield in bitfields:\n",
    "    pprint(f\"(n={bitfield})\")\n",
    "    stream = BytesIO(int_to_bytes(bitfield, 4))\n",
    "    pprint(read_services(stream))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise #8: Complete these function definitions to hammer home you understanding of this strange `services` \"bitfield\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def offers_node_network_service(services_bitfield):\n",
    "    # given integer services_bitfield, return whether the NODE_NETWORK bit is on\n",
    "    raise NotImplementedError()\n",
    "\n",
    "def offers_node_bloom_and_node_witness_services(services_bitfield):\n",
    "    # given integer services_bitfield, return whether the \n",
    "    # NODE_BLOOM and NODE_WITNESS bits are on\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ipytest, pytest\n",
    "\n",
    "def test_services_0():\n",
    "    assert offers_node_network_service(1) is True\n",
    "    assert offers_node_network_service(1 + 8) is True\n",
    "    assert offers_node_network_service(4) is False\n",
    "    \n",
    "\n",
    "def test_services_1():\n",
    "    assert offers_node_bloom_and_node_witness_services(1) is False\n",
    "    assert offers_node_bloom_and_node_witness_services(1 + 8) is False\n",
    "    assert offers_node_bloom_and_node_witness_services(4 + 8) is True\n",
    "    \n",
    "ipytest.run_tests(doctest=True)\n",
    "ipytest.clean_tests(\"test_services*\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# \"Network Address\" Type\n",
    "\n",
    "[`net_addr`](https://en.bitcoin.it/wiki/Protocol_documentation#Network_address) is the most complicated new type we encounter this lesson, so we'll handle it last. Plus, it builds on the `timestamp` and `services` types we learned to read above.\n",
    "\n",
    "![image](images/network-address.png)\n",
    "\n",
    "Network addresses require we interpret 4 new kinds of data:\n",
    "\n",
    "1. `time`: Unix timestamp. Already done.\n",
    "2. `services`: integer bitfield. Already done.\n",
    "3. `IP address`: complicated ...\n",
    "4. `port`: big-endian encoded `int`\n",
    "\n",
    "Here's a Python class abstracting this \"Network Address\" type. \n",
    "\n",
    "* `read_ip` and `read_port` functions await implementation\n",
    "* `net_addr` doesn't contain a `time` when it's inside a `version` message. Yup, that's confusing to me too. The `if-else` statement in `Address.from_stream` is my best attempt at translating what the documentation describes. Once again, we'll employ an argument with a default value `version_msg=False` to allow whoever calls `Address.from_stream` to tell it whether it's inside a version message or not. We'll see if this works!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_ip(stream):\n",
    "    raise NotImplementedError()\n",
    "\n",
    "def read_port(stream):\n",
    "    raise NotImplementedError()\n",
    "    \n",
    "class Address:\n",
    "\n",
    "    def __init__(self, services, ip, port, time):\n",
    "        self.services = services\n",
    "        self.ip = ip\n",
    "        self.port = port\n",
    "        self.time = time\n",
    "\n",
    "    @classmethod\n",
    "    def from_stream(cls, stream, version_msg=False):\n",
    "        if version_msg:\n",
    "            time = None\n",
    "        else:\n",
    "            time = read_timestamp(stream)\n",
    "        services = read_services(stream)\n",
    "        ip = read_ip(stream)\n",
    "        port = read_port(stream)\n",
    "        return cls(services, ip, port, time)\n",
    "    \n",
    "    def __repr__(self):\n",
    "        return f\"<Address {self.ip}:{self.port}>\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise #9: Implement `read_ip`\n",
    "\n",
    "hint: read n bytes where n is in the chart above, and return `ip_address(those_bytes)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipaddress import ip_address\n",
    "\n",
    "def read_ip(stream):\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ipytest, pytest\n",
    "\n",
    "def test_read_ip_0():\n",
    "    ipv4 = '10.10.10.10'\n",
    "    ipv4_mapped = b'\\x00'*10 + b'\\xff'*2 + ip_address(ipv4).packed\n",
    "    stream = BytesIO(ipv4_mapped)    \n",
    "    assert read_ip(stream).ipv4_mapped.compressed == ipv4\n",
    "    \n",
    "ipytest.run_tests(doctest=True)\n",
    "ipytest.clean_tests(\"test_read_ip*\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## \"Network Address > Port\" Field\n",
    "\n",
    "This is just 2 byte integer -- but it's encoded with the opposite byte order of what we usually read using `read_bytes`. But have no fear, `read_bytes` takes an optional `byte_order` parameter which defaults to `\"little\"` -- since we're usually reading little-endian encoded messages. But if we set it to `\"big\"`, then `read_int` will successfully read the \"big endian\" / \"network byte order\" port integer.\n",
    "\n",
    "In order to have clean, testable code we will define another helper method: `read_port`\n",
    "\n",
    "### Exercise #10: Implement `read_port`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_port(stream):\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ipytest, pytest\n",
    "from io import BytesIO\n",
    "\n",
    "ports = [8333, 55555]\n",
    "\n",
    "def test_read_port_0():\n",
    "    for port in ports:\n",
    "        bytes_ = port.to_bytes(2, 'big')\n",
    "        stream = BytesIO(bytes_)\n",
    "        result = read_port(stream)\n",
    "        assert port == result\n",
    "    \n",
    "ipytest.run_tests(doctest=True)\n",
    "ipytest.clean_tests(\"test_read_port*\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parsing a complete Version response\n",
    "\n",
    "Let's put together all the little helper function and helper classes we've so dilligently written and parse the payload of the `version` message we downloaded at the beginning of this lesson."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import socket\n",
    "from ibd.two.complete import Packet, VersionMessage # get the final version ...\n",
    "\n",
    "PEER_IP = \"35.198.151.21\"\n",
    "\n",
    "PEER_PORT = 8333\n",
    "\n",
    "# magic \"version\" bytestring\n",
    "VERSION = b'\\xf9\\xbe\\xb4\\xd9version\\x00\\x00\\x00\\x00\\x00j\\x00\\x00\\x00\\x9b\"\\x8b\\x9e\\x7f\\x11\\x01\\x00\\x0f\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x93AU[\\x00\\x00\\x00\\x00\\x0f\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0f\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00rV\\xc5C\\x9b:\\xea\\x89\\x14/some-cool-software/\\x01\\x00\\x00\\x00\\x01'\n",
    "\n",
    "sock = socket.socket()\n",
    "sock.connect((PEER_IP, PEER_PORT))\n",
    "\n",
    "# initiate the \"version handshake\"\n",
    "sock.send(VERSION)\n",
    "\n",
    "# receive their \"version\" response\n",
    "pkt = Packet.from_socket(sock)\n",
    "\n",
    "msg = VersionMessage.from_bytes(pkt.payload)\n",
    "print(msg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Boom! This is basically the same code we finished the last lesson with, but our magical `version_report` function and all the functions it calls are able to decipher what this cryptic message _means_!\n",
    "\n",
    "Some work is left. What the heck do the `net_addr` fields mean? What kind of `versions` are most people running. Are most people `relay`ing, or not?\n",
    "\n",
    "In the next lesson we'll connect with every Bitcoin full node we can find and try to answer some of these questions!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}