Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pre-release branch for N.2.9.1 #695

Merged
merged 142 commits into from
Jan 27, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
142 commits
Select commit Hold shift + click to select a range
28cd902
Introduce 2.9 replication format (HF)
JamesPiechota Nov 26, 2024
43b4c89
Initialize poa caches for initial block cache blocks
vird Oct 26, 2024
6f93b59
fix: use exor instead of feistel for 2.9 enciphering
JamesPiechota Dec 12, 2024
755200c
fix: bump randomx commit
JamesPiechota Dec 12, 2024
ba50564
Optimize and Improve Github CI/CD
humaite Dec 10, 2024
013a470
fix: update submodule before building
JamesPiechota Dec 12, 2024
8f2ba9e
Add arweave papers
JamesPiechota Dec 12, 2024
56cd4dd
Bump release number for 2.9.0-early-adopter
JamesPiechota Dec 12, 2024
0d46cc9
fix: add benchmark-2.9 to the release
JamesPiechota Dec 12, 2024
4eaa4bb
Improve Download Partition Script
humaite Dec 13, 2024
63fbcb6
Revert Download Partition Script
humaite Dec 18, 2024
bf954b4
increase timeouts
humaite Dec 18, 2024
db56b2c
Revert "increase timeouts"
humaite Dec 18, 2024
2825b7b
refactor: move several replica_2_9 functions into ar_replica_2_9
JamesPiechota Dec 15, 2024
fea52b9
refactor: replace ENTROPY_SUB_CHUNK_COUNT with ENTROPY_SIZE
JamesPiechota Dec 15, 2024
101bb8f
refactor: replace get_entropy_bucket_start with get_chunk_bucket_start
JamesPiechota Dec 15, 2024
3298f90
refactor: drop the "mask" term, I don't think it's needed anymore?
JamesPiechota Dec 16, 2024
b577c94
docs: write ar_replica_2_9 module docs
JamesPiechota Dec 16, 2024
30fe3f2
refactor: revert to having a distinct get_entropy_bucket_start/1
JamesPiechota Dec 17, 2024
59985e9
docs: update some comments
JamesPiechota Dec 17, 2024
0ac0300
Improve Download Partition Script
humaite Dec 13, 2024
111f7ab
Do not unpack 2.9 modules for other modules
Dec 18, 2024
36b64a4
Request unpacked data in syncing
Dec 18, 2024
e925fa9
Do not be silent on failure when creating new wallets (including bin/…
Oct 22, 2024
ef6b3ce
Generate missing entropy after preparation on the fly
Dec 18, 2024
d86599b
Update replica 2.9 sync record asynchronously
Dec 18, 2024
6388075
Fix bug where repeated 2.9 writes erase sync record
Dec 19, 2024
a095e28
Add support for storing 2.9 unpacked chunks during preparation
Dec 19, 2024
fc74809
Increase ar_chunk_storage:put timeout
Dec 19, 2024
84e3b2c
feat: add some more metrics for 2.9
JamesPiechota Dec 19, 2024
0a97dca
fix syntax
vird Dec 24, 2024
9013986
Merge pull request #672 from ArweaveTeam/fix/syntax
vird Dec 24, 2024
26132d5
test: get tests to pass without setting NONCE_LIMITER_RESET_FREQUENCY
JamesPiechota Dec 18, 2024
ca0dfb7
fix: fix a race condition in ar_multiple_txs_per_wallet_tests
JamesPiechota Dec 20, 2024
9ac47b1
fix: see if increaing the shutdown timeout avoid db lock issues
JamesPiechota Dec 23, 2024
8c76a2b
fix: fix a race condition in ar_coordinated_mining_tests
JamesPiechota Dec 24, 2024
0716234
test: assert when wait_until_height skips a block
JamesPiechota Dec 24, 2024
0b9b69b
test: fix race condition in ar_mining_stats
JamesPiechota Dec 26, 2024
54a9bfc
test: ensure we always reset the config after changing it
JamesPiechota Dec 26, 2024
a66d921
test: better mock load/unload handling
JamesPiechota Dec 26, 2024
f158119
(re)Configure Github Workflow (#678)
humaite Dec 27, 2024
85179d6
Disable fail-fast feature from Github Matrix Workflow
humaite Dec 30, 2024
ac218f4
feat: make the genesis data deterministic
JamesPiechota Nov 15, 2024
0bb7e41
feat: add support for an e2e test mode
JamesPiechota Nov 17, 2024
666edd3
test: fix ar_mining_stats race condition
JamesPiechota Dec 27, 2024
5fd8d34
fix: set the default sync format back to `any` from `unpacked`
JamesPiechota Dec 27, 2024
e896023
Allow Github Workflow to be Executed on Pull Requests
humaite Jan 3, 2025
b8ef8fe
Add missing console script in release
humaite Jan 3, 2025
7943110
test: add ar_repack_in_place_mine_test to the e2e tests
JamesPiechota Dec 30, 2024
c40f483
refactor: some renaming
JamesPiechota Dec 30, 2024
4fa81e5
fix: reduce impact of entropy overhang + fix some bugs
JamesPiechota Dec 31, 2024
2ac784a
test: fix the test implementation for enciphering
JamesPiechota Jan 2, 2025
4a46092
refactor: refactor or chunk validation errors are logged
JamesPiechota Jan 2, 2025
9683de3
fix: handle fact that partitions often end in th emiddle of a chunk
JamesPiechota Jan 2, 2025
0681297
test: add replica_2_9 to e2e sync_pack_mine tests
JamesPiechota Jan 2, 2025
0da18e1
fix: block syncing of replica_2_9 chunks
JamesPiechota Jan 2, 2025
80a811d
Fully replace randomx squared with fused implementation (#681)
vird Jan 4, 2025
3f7b881
Add e2e tests support in Github Pipelines
humaite Jan 3, 2025
190a81d
Fix bug where index of 2.9 chunks waiting for entropy is not updated
Jan 2, 2025
60dab40
Repack (in place) unpacked chunks stored in RocksDB
Nov 6, 2024
4db96da
Add support for in-place repacking to the 2.9 format
Jan 2, 2025
61f3770
refactor: introduce ar_entropy_storage.erl for replica.2.9 entropy code
JamesPiechota Jan 6, 2025
14eafc6
test: use AR_TEST instead of TEST since eunit.hrl sets TEST (!!)
JamesPiechota Jan 7, 2025
f24e945
fix: bug fixes for replica.2.9 repack_in_place
Jan 7, 2025
8dc83b0
fix: fix crash when read_range2 returns recast
JamesPiechota Jan 7, 2025
1707a7b
fix: correctly block replica.2.9 syncing server-side
Jan 7, 2025
ab1de7a
test: add replica_2_9 to the ar_repack_mine_tests
JamesPiechota Jan 7, 2025
01d938a
fix: store 2.9 entropy in all relevant storage modules
Jan 8, 2025
1c4a6d7
test: get repack_mine_tests working in e2e test suite
JamesPiechota Jan 8, 2025
7f7fed2
Improve Download Partition Script
humaite Dec 13, 2024
68a107e
Feature/ecdsa (#686)
JamesPiechota Jan 9, 2025
641d426
fix: some fixes for replica_2_9 packing
JamesPiechota Jan 9, 2025
63f83f4
refactor: pull repack-in-place logic out into ar_repack module
JamesPiechota Jan 10, 2025
158356c
refactor: encapsulate access to the chunks_data_db
JamesPiechota Jan 10, 2025
00d9e62
refactor: encapsulate access to the chunks_index rocksdb
JamesPiechota Jan 10, 2025
ab5a6b3
refactor: move more code from ar_chunk_storage to ar_repack
JamesPiechota Jan 10, 2025
8d01732
fix: fix several repack in place bugs
JamesPiechota Jan 13, 2025
166e96e
mac build
samantehrani Jan 13, 2025
324923a
test: rebar.config for testnet
JamesPiechota Jan 13, 2025
c336e3d
fix: ECDSA revert key size (#690)
vird Jan 16, 2025
375c5ce
Fix blacklist url parsing bug
humaite Jan 16, 2025
db8de86
Do not run shell if there are running beams
Jan 16, 2025
74307f5
Test slashing 4 doube-signing with ECDSA keys
Jan 16, 2025
1181a0e
Assert key size before signing blocks
Jan 16, 2025
39e0448
Test blocks signed with small RSA keys
Jan 16, 2025
afc4abe
Rename ar_multiple_txs_per_wallet_tests -> ar_tx_tests
Jan 16, 2025
f6ae3e4
Add ECDSA-signed txs to accepts_and_gossips_txs_test
Jan 16, 2025
85c43e1
Use a small RSA key in keeps_txs_after_new_block_test_
Jan 16, 2025
489a6e3
Add a way to use ar_test_node functions with external peers
Jan 17, 2025
f251a6b
Fix chunk discovery during 2.9 entropy generation
Jan 17, 2025
1030019
Update testnet-5 config (migrate to an ECDSA key)
Jan 17, 2025
fcc108d
Switch storage module 40 to new packing on testnet-5
Jan 17, 2025
01c5d93
Assert double-signing proof sizes early
Jan 18, 2025
932711b
Fix many small bugs in download script
humaite Jan 22, 2025
b0692c7
fix: introduce ar_device_lock to manage IO by system device
JamesPiechota Jan 9, 2025
12ebfb3
fix: entropy preparation performance improvements
JamesPiechota Jan 18, 2025
365d6af
feature: expose the packing_worker and replica_2_9_workers flags
JamesPiechota Jan 19, 2025
6b3fded
fix: add delete_async, clean up some metrics, cache StoreIDLabel
JamesPiechota Jan 20, 2025
1b8e384
fixup parse packing_workers and replica_2_9_workers in ar_config
Jan 20, 2025
2f7cffd
fixup collect repacked modules in ar_device_lock
Jan 20, 2025
e59255f
fixup handle timed out entropy generated replies
Jan 20, 2025
91c881c
fixup report all read_range tasks as localhost
Jan 20, 2025
905a94d
test: add some more logging to tests
JamesPiechota Jan 20, 2025
4e69e51
Record invalid mining solution data
Jan 20, 2025
877efe6
test: fix ar_device_lock test
JamesPiechota Jan 20, 2025
52d797f
fix: revert delete_async
JamesPiechota Jan 20, 2025
18141a0
fix: handle an extra case statement in GET /chunk
JamesPiechota Jan 20, 2025
eaba490
refactor: remove some debugging output
JamesPiechota Jan 20, 2025
6370ef0
misc: fix case clause errors, better logs
JamesPiechota Jan 20, 2025
d936056
fix: make sure device lock status is printed at least once
JamesPiechota Jan 20, 2025
c6b02de
fix: spawn ar_entropy_storage workers for repack_in_place modules
JamesPiechota Jan 20, 2025
80e37ce
fixup chunk byte from bucket end
Jan 20, 2025
0fd3ab4
fix: handle an error condition without crashing
JamesPiechota Jan 20, 2025
bc60119
logging: add some logs to help diagnose the invalid padding error
JamesPiechota Jan 20, 2025
9e06471
test: add some more logging to diagnose e2e test failures
JamesPiechota Jan 21, 2025
b243357
Handle invalid_padding gracefully in GET /chunk
Jan 21, 2025
7e707ce
fix: miscellaneaous fixes for replica.2.9 packing
Jan 21, 2025
254a8bb
fix: fix offset handling in invalidate_data_record
Jan 21, 2025
d993343
test: fix test flakiness in ar_post_block_tests
Jan 21, 2025
c38072f
refactor: refactor some of the offset code, add tests
JamesPiechota Jan 21, 2025
9641c9f
fix: block repacking in place to any format other than replica.2.9
JamesPiechota Jan 21, 2025
69f3ed6
Add some debug logs tracking stored 2.9 chunk data
Jan 22, 2025
4fb0de7
Fix 2.9 corruption recovery
Jan 22, 2025
0476041
Check chunk packing during mining early
Jan 22, 2025
c54e69d
Fix flaky reject_block_invalid_double_signing_proof_test
Jan 22, 2025
f958a4f
fix: fix some data corruption during packing
JamesPiechota Jan 23, 2025
a60baa3
fix: edge case crash during tests
JamesPiechota Jan 23, 2025
5099b1a
Revert "Add some debug logs tracking stored 2.9 chunk data"
JamesPiechota Jan 23, 2025
91e70da
fix: cleanup logs, remove unnecessary wait
JamesPiechota Jan 23, 2025
a15fcb7
Revert "Check chunk packing during mining early"
JamesPiechota Jan 23, 2025
b726a5f
cleanup: remove some dead code, add some logs
JamesPiechota Jan 23, 2025
fc99921
fix: handle read_range tasks in one worker per storage_module
JamesPiechota Jan 24, 2025
174c73d
Improve invalidate bad data record interface
Jan 24, 2025
57bdc36
fix: fix race condition during repack in place
JamesPiechota Jan 24, 2025
811375e
fix: move entropy generation out to its own process (#693)
JamesPiechota Jan 25, 2025
e88c467
fix: don't log "repack complete" until both repack and entropy are done
JamesPiechota Jan 25, 2025
6b9cc4a
fix: fix some iteration errors in repack_in_place
JamesPiechota Jan 26, 2025
87d364f
Fix chunk discovery during entropy generation
Jan 27, 2025
fc92124
Fix chunk_bucket_test after 6b9cc4a6
Jan 27, 2025
d487d08
test: fix some e2e test failures
JamesPiechota Jan 27, 2025
1017911
fix: fix localnet
JamesPiechota Jan 27, 2025
0c36089
test: fix and logging for e2e tests
JamesPiechota Jan 27, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
196 changes: 196 additions & 0 deletions .github/workflows/e2e-test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,196 @@
name: "Arweave e2e Tests Suites"
on:
workflow_dispatch:
schedule:
- cron: "0 13 * * 1"
pull_request_target:
types: [opened]
branches:
- 'master'
- 'release/**'
- 'releases/**'

jobs:
build:
runs-on: self-hosted
steps:
- uses: actions/checkout@v4
with:
submodules: "recursive"

# only arweave dependencies are being cached,
# those are not updated everyday and this is
# unecessary to fetch them everytime.
- uses: actions/cache@v4
id: cache
with:
path: |
_build/default/lib/accept
_build/default/lib/b64fast
_build/default/lib/cowboy
_build/default/lib/cowlib
_build/default/lib/graphql
_build/default/lib/gun
_build/default/lib/jiffy
_build/default/lib/prometheus
_build/default/lib/prometheus_cowboy
_build/default/lib/prometheus_httpd
_build/default/lib/prometheus_process_collector
_build/default/lib/quantile_estimator
_build/default/lib/ranch
_build/default/lib/.rebar3
_build/default/lib/recon
_build/default/lib/rocksdb
_build/default/plugins/
_build/default/plugins/aleppo
_build/default/plugins/geas
_build/default/plugins/geas_rebar3
_build/default/plugins/hex_core
_build/default/plugins/katana_code
_build/default/plugins/pc
_build/default/plugins/.rebar3
_build/default/plugins/rebar3_archive_plugin
_build/default/plugins/rebar3_elvis_plugin
_build/default/plugins/rebar3_hex
_build/default/plugins/samovar
_build/default/plugins/verl
_build/default/plugins/zipper
key: deps-cache-${{ hashFiles('rebar.lock') }}
restore-keys: |
deps-cache-${{ hashFiles('rebar.lock') }}

- name: Get dependencies
if: steps.cache.outputs.cache-hit != 'true'
run: ./ar-rebar3 test get-deps

- uses: actions/cache@v4
if: steps.cache.outputs.cache-hit != 'true'
with:
path: |
_build/default/lib/accept
_build/default/lib/b64fast
_build/default/lib/cowboy
_build/default/lib/cowlib
_build/default/lib/graphql
_build/default/lib/gun
_build/default/lib/jiffy
_build/default/lib/prometheus
_build/default/lib/prometheus_cowboy
_build/default/lib/prometheus_httpd
_build/default/lib/prometheus_process_collector
_build/default/lib/quantile_estimator
_build/default/lib/ranch
_build/default/lib/.rebar3
_build/default/lib/recon
_build/default/lib/rocksdb
_build/default/plugins/
_build/default/plugins/aleppo
_build/default/plugins/geas
_build/default/plugins/geas_rebar3
_build/default/plugins/hex_core
_build/default/plugins/katana_code
_build/default/plugins/pc
_build/default/plugins/.rebar3
_build/default/plugins/rebar3_archive_plugin
_build/default/plugins/rebar3_elvis_plugin
_build/default/plugins/rebar3_hex
_build/default/plugins/samovar
_build/default/plugins/verl
_build/default/plugins/zipper
key: deps-cache-${{ hashFiles('rebar.lock') }}

- name: Build arweave test sources
run: ./ar-rebar3 test compile

# some artifacts are compiled and only available
# in arweave directy (libraries)
- name: Prepare artifacts
run: |
chmod -R u+w ./_build
tar czfp _build.tar.gz ./_build
tar czfp apps.tar.gz ./apps

# to avoid reusing artifacts from someone else
# and generating issues, an unique artifact is
# produced using github checksum.
- name: upload artifacts
uses: actions/upload-artifact@v4
with:
name: build-${{ github.sha }}
if-no-files-found: error
retention-days: 1
overwrite: true
path: |
_build.tar.gz
apps.tar.gz

# some artifacts are compiled and only available
# in arweave directy (libraries)
- name: Prepare artifacts
run: |
chmod -R u+w ./_build
tar czfp _build.tar.gz ./_build
tar czfp apps.tar.gz ./apps

# to avoid reusing artifacts from someone else
# and generating issues, an unique artifact is
# produced using github checksum.
- name: upload artifacts
uses: actions/upload-artifact@v4
with:
name: build-${{ github.sha }}
if-no-files-found: error
include-hidden-files: true
retention-days: 7
overwrite: true
path: |
_build.tar.gz
apps.tar.gz

e2e-tests:
needs: build
runs-on: self-hosted
strategy:
max-parallel: 4
matrix:
core_test_mod: [
ar_sync_pack_mine_tests,
ar_repack_mine_tests,
ar_repack_in_place_mine_tests
]
steps:
- uses: actions/checkout@v4
with:
submodules: "recursive"

- name: Download artifact
uses: actions/download-artifact@v4
with:
name: build-${{ github.sha }}

# Both artifacts (_build and apps dir) are
# required.
- name: Extract artifact
run: |
tar zxfp _build.tar.gz
tar zxfp apps.tar.gz

- name: ${{ matrix.core_test_mod }}.erl
id: tests
run: bash scripts/github_workflow.sh "${{ matrix.core_test_mod }}"

# this part of the job produces test artifacts from logs
# generated by the tests. It also collect dumps and the files
# present in .tmp (temporary arweave data store)
- name: upload artifacts in case of failure
uses: actions/upload-artifact@v4
if: always() && failure()
with:
name: "logs-${{ matrix.core_test_mod }}-${{ github.run_attempt }}-${{ job.status }}-${{ runner.name }}-${{ github.sha }}"
retention-days: 7
overwrite: true
include-hidden-files: true
path: |
./logs
*.out
*.dump
9 changes: 8 additions & 1 deletion .github/workflows/gitstamp.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,14 @@
# See: https://github.com/weavery/gitstamp-action
---
name: Gitstamp
on: [push]
on:
push:
pull_request_target:
types: [opened]
branches:
- 'master'
- 'release/**'
- 'releases/**'
jobs:
gitstamp:
runs-on: ubuntu-latest
Expand Down
Loading
Loading