From 7a9057b0a261c09d53af0fe908cc7f4181704824 Mon Sep 17 00:00:00 2001 From: David Whittington Date: Mon, 9 Sep 2024 12:55:43 -0500 Subject: [PATCH] docs(changel): add WIP notes for release 17 PE-6562 --- CHANGELOG.md | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 74096284..cf4116a4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,62 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). ## [Unreleased] +### Fixed + +- Use the correct environment variable to populate WEBHOOK_BLOCK_FILTER in + `docker-compose.yaml`. +- Don't cache data regions retrieved to satisfy range requests to avoid + unnecessary storage overhead and prevent inserting invalid ID to hash + mappings into the data DB. + +### Added + +- Added a new ClickHouse based DB backend. It can be used in combination with + the SQLite DB backend to enable batch loading of historical data from + Parquet. It also opens up the possibility of higher DB performance and + scalability. In its current state it should be considered a technology + preview. It won't be useful to most users until we either provide Parquet + files to load into it or automate flushing of the SQLite DB to it (both are + planned in future release). It is not intended to be standalone solution. It + supports bulk loading and efficient GraphQL querying of transactions and data + items, but it relies on SQLite (or potentially another OLTP in the future) to + index recent data. These limitations allow greatly simplified schema and + query construction. Querying the new ClickHouse DB for transaction and data + items via GraphQL is enabled by setting the 'CLICKHOUSE_URL' environment + variable. +- Added the ability to skip storing transaction signatures in the DB by setting + WRITE_TRANSACTION_DB_SIGNATURES to false. Missing signatures are fetched from + the trusted Arweave node when needed for GraphQL results. +- Added a Redis backed signature cache to support retrieving optimistically + indexed data item signatures in GraphQL queries when writing data items + signatures to the DB has been disabled. +- Added on-demand and composite ArNS resolvers. The on-demand resolver + fetches results directly from an AO CU. The composite resolver attempts + resolution in the order specified by the ARNS_RESOLVER_PRIORITY_ORDER + environment variable (defaults to 'on-demand,gateway'). +- Added a queue_length Prometheus metric to fasciliate monitoring queues and + inform future optimizations +- Added SQLite WAL cleanup worker to help manage the size of the `data.db-wal` + file. Future improvements to `data.db` usage are also planned to further + improve WAL management. + +### Changed + +- Handle data requests by ID on ArNS sites. This enables ArNS sites to use + relative links to data by ID. +- Replaced ARNS_RESOLVER_TYPE with ARNS_RESOLVER_PRIORITY_ORDER (defaults to + 'on-demand,gateway'). +- Introduced unbundling back pressure. When either data item data or GraphQL + indexing queue depths are more than the value specified by the + MAX_DATA_ITEM_QUEUE_SIZE environment variable (defaults to 100000), + unbundling is paused until the queues length falls bellow that threshold. + This prevents the gateway from running out of memory when the unbundling rate + exceeds the indexing rate while avoiding wasteful bundle reprocessing. +- Prioritized optimistic data item indexing by inserting optimistic data items + at the front of the indexing queues. +- Prioritized nested bundle indexing by inserting nested bundles at the front + of the unbundling queue. + ## [Release 16] - 2024-08-09 ### Fixed