Skip to content

Configuring Conseil (Up to Tezos Hangzhou)

Jump to bottom
Siddharth Vaderaa edited this page Nov 29, 2021 · 5 revisions

Configuring Conseil

**This document is currently a work in progress. It should be finalized within a few hours. **

Conseil and Lorre need to be configured correctly or they won't run at all.

Basic Configuration

In the repository the file conseil-common/src/main/resources/reference.conf contains common configuration items. Configuration for specific applications can be found in relevant modules (conseil-api/src/main/resources/application.conf and conseil-lorre/src/main/resources/application.conf). Most of the default values should suffice for everyone. However, you will need to use a custom configuration file or environment variables which will override reference.conf and application.conf with the particular details of your own deployment.

Configuration file

The below is a reasonable custom configuration file to run Conseil and Lorre with. Some of these entries, such as addresses, ports, usernames and passwords, should be edited to reflect specific deployments.

TODO: Siddharth to paste actual config file.

include required(classpath("application.conf"))


platforms: [
  {
    name: "tezos"
    network: "mainnet"
    enabled: true
    node: {
        #Replace the below entries with the address of your Tezos node
        protocol: "http"
        hostname: "localhost"
        port: 8732
        path-prefix: ""
    }
     db:
     {
      dataSourceClass = "org.postgresql.ds.PGSimpleDataSource"
      properties
      {
        # Replace the below lines with details of your database configured with the Conseil database schema
        databaseName = "conseil"
        user = "myuser"
        password = "mypassword"
        url = "jdbc:postgresql://localhost:5432/conseil"
      }
      numThreads = 20
      maxConnections = 20
     }
  }
]

conseil {
 hostname: "0.0.0.0"
 port: 1337


 security: {
      api-keys: {
         keys: ["galleon", "arronax", "mininax", "hooman"]
   }
 }
}

lorre.chain-events: [
  {
    type: accountsRefresh,
    levels: {
      ".*"   : [655361]
    }
  }
]

lorre.db:
     {
      dataSourceClass = "org.postgresql.ds.PGSimpleDataSource"
      properties
      {
        # Replace the below lines with details of your database configured with the Conseil database schema
        databaseName = "conseil"
        user = "myuser"
        password = "mypassword"
        url = "jdbc:postgresql://localhost:5432/conseil"
      }
      numThreads = 20
      maxConnections = 20
     }

lorre.enabled-features.metadata-fetching-is-on: false

See the comments in the extract above for values which you will need to customize. This file will pull in defaults from reference.conf and application.conf and only override the settings mentioned in it.

Once you are satisfied with the file you can use it for Running Conseil.

Environment variables

The most comment properties can be also changed by exporting specific environment variables before running Conseil modules.

Here you can find list of properties which can be changed:

Common module:

  • CONSEIL_XTZ_NETWORK - The name of the Tezos network.
  • CONSEIL_XTZ_ENABLED - Indicates if this network should be enabled. If disabled all components will skip it.
  • CONSEIL_XTZ_NODE_PROTOCOL - The protocol used by Tezos Node.
  • CONSEIL_XTZ_NODE_HOSTNAME - The hostname used by Tezos Node.
  • CONSEIL_XTZ_NODE_PORT - The port used by Tezos Node.
  • CONSEIL_XTZ_NODE_PATH_PREFIX - The path prefix used by Tezos Node.
  • CONSEIL_BTC_NETWORK - The name of the Bitcoin network.
  • CONSEIL_BTC_ENABLED - Indicates if this network should be enabled. If disabled all components will skip it.

Conseil-api module:

  • CONSEIL_API_HOSTNAME - The hostname under which API will be hosted.
  • CONSEIL_API_PORT - The port under which API will be exposed.
  • CONSEIL_API_DB_USER - The name of the user, used for making database connection.
  • CONSEIL_API_DB_PASSWORD - The password of the user, used for making database connection.
  • CONSEIL_API_DB_URL - The connection string (url) for making database connection, e.g: jdbc:postgresql://localhost:5432/postgres.
  • CONSEIL_API_KEY - The api key used to authorize queries for API.
  • CONSEIL_API_ALLOW_BLANK_KEYS - Indicates if API allows blank api keys.

Conseil-lorre module:

  • CONSEIL_LORRE_DB_USER - The name of the user, used for making database connection.
  • CONSEIL_LORRE_DB_PASSWORD - The password of the user, used for making database connection.
  • CONSEIL_LORRE_DB_URL - The connection string (url) for making database connection, e.g: jdbc:postgresql://localhost:5432/postgres.
  • CONSEIL_LORRE_DEPTH - Indicates how many blocks to synchronize, starting from head hash. Use -1 or all for everything and 0 or new to only get new ones.
  • CONSEIL_LORRE_HEAD_HASH - Indicates the head hash, as starting point for making blocks synchronization.

Once you are satisfied with the environment variables you can use it for Running Conseil.

Advanced Configuration

Additional fine-grained configuration

You might want to override, only after careful testing, part of the following configuration entries

# Runtime settings for Conseil server
conseil {
  hostname: "0.0.0.0"
  port: 1337
}

lorre {
  batched-fetches {
    # The following configs defines how many concurrent requests we'd like to run
    #   against a tezos node to speed up fetching of data

    #Used when getting accounts from tezos
    account-concurrency-level: 5

    #Used when getting operation data for each new block just fetched
    block-operations-concurrency-level: 10

    #Used to paginate blocks read from tezos before each db storage
    block-page-size: 500

  }
}

# Customization on the underlying actor system
akka {

  # custom host pool for akka-http client connections used for streaming request/responses
  # tune the configuration based on load-handling capability of blockchain nodes
  # refer to host-connection-pool section in
  # https://doc.akka.io/docs/akka-http/current/configuration.html
  # for available properties and their meaning
  #
  # The current configuration is based on local benchmarking against zeronet
  # On the streaming http client pool we expect a max of:
  #   30 connections x
  #    7 requests/conn ~=
  #  210 ongoing requests at each moment
  # The pipelining on each connection might slow down for slow responses, but they should be rare
  streaming-client {
    max-connections: 30
    # The 2048 limit is thus overestimated by a factor of roughly 10x, to allow room for
    # reuse of the same pool from different threads at the same time, up to that factor
    max-open-requests: 2048
    # essentially keep connections alive across lorre's cycles
    idle-timeout: 10 minutes
    pipelining-limit: 7
    # give more room for async response in head-of-line blocking on the same connection or other slow responses
    response-entity-subscription-timeout: 5 seconds
  }

  # akka-http-caching config described here:
  # https://doc.akka.io/docs/akka-http/current/common/caching.html#frequency-biased-lfu-cache
  http {
    # this is essentially available to enable composition of database operations
    # current configuration is based upon the default-blocking-io-dispatcher in akka
    dispatcher {
      type: "Dispatcher"
      executor: "thread-pool-executor"
      throughput: 1

      thread-pool-executor {
        fixed-pool-size: 16
      }
    }

    caching.lfu-cache {
      max-capacity: 500
      initial-capacity: 100
      time-to-live: 5 minutes
      time-to-idle: 3 minutes
    }
  }
}

TODO: Siddharth to update to new format.

In addition to that, both lorre.db and conseil.db allows finer configuration details for the database access, where the properties can be any of those described in the java postgres driver documentation

An example of an SSL remote connection might look like

lorre.db {
  dataSourceClass: "org.postgresql.ds.PGSimpleDataSource"
  properties {
    serverName: "my-remote-host"
    ssl: true
    sslMode: "require"
    databaseName: "conseil-dbname"
    user: "username"
    password: "password"
    portNumber: 26000
    reWriteBatchedInserts: true
  }
  numThreads: 10
  maxConnections: 10
}

Fine-grained debug logs for remote node communications

When there is some issue while the Lorre indexer connects to the blockchain node and gets the data, it might turn useful to trace the actual requests/responses between the two.

You can enable a fine-grained logging of those by setting on an additional flag attribute named trace-calls in your configuration entry for the node, as shown in the following example

  node {
    protocol: "http"
    hostname: "localhost"
    port: 50002
    path-prefix: ""
    trace-calls: "true"
  }

Turning calls tracing on will print a debug log entry for each rpc call and response, including detailed descriptions of the http calls and payloads of request and response.

Consider this as a temporary tool because it will result in a significant amount of logging being produced.

Chain's "one-off" events handling

To support out-of-the ordinary processing on the data collected and exposed by Conseil, the Lorre configuration supports special handling for generally-named "Chain Events".

Such events might be identified by the Block Level when they need to be handled. Currently we only support one type of event, namely

  • AccountsRefresh: upon reaching certain levels, Lorre plans a full or partial reload of accounts data from the rpc node. This is instrumental, for example, to take into account the Tezos airdrop after Babylon protocol switch, that impacted many delegated contracts.

The configuration entry specifies, for the running Lorre instance, after which levels such "refresh" is needed, and allows to define a regular expression pattern to select only part of the accounts' hashes.

It will look like the following excerpt

lorre.chain-events: [
  {
    type: accountsRefresh,
    levels: {
      "tz1.*": [655000],
      ".*"   : [655361]
    }
  }
]

Here there's a request for tz1 accounts after level 655000, and a global one at level 655361.

Clone this wiki locally