SPX v1.15.2-rc3 - Testing & Call Outs #8579

Reiers · 2022-05-02T16:46:30Z

Reiers
May 2, 2022

Hello SPX Fellows

Same old, you know the drill - upgrade to: v1.15.2-rc3 and fill out the Reply card - report any bugs, if any.

❗You need to upgrade Go to v1.17.9 or higher.

🚀 This post if you are participating in this testing round.
If you aren't able to join for reasons, please reach out to @jennijuju or me on Slack!!!

New features and Improvements

--max-piece-size in set-ask are no longer required when changing the price

For more improvements & bug fixes see here.

Reply card:

If you have successfully sealed some new sectors on this tag.
If you have successfully made some Storage Deals on this tag.
If you have successfully made some Snap Deals on this tag.
If you passed your first WindowPoSt on this tag.
If the log is back to normal.
The resource manager has not caused any connection problems.

Reminders and Risk Mitigations

Questions are welcomed in this discussion, the team will be monitoring this discussion and check-in regularly.

Issues can be reported in https://github.com/filecoin-project/lotus/issues. Issues should be submitted with:

Daemon logs / set RUST_LOG=debug if possible.
go routine dumps lotus pprof goroutines and lotus-miner pprof goroutines
the operation/step at which the problem occurred
The stack trace or logs as relevant. As always, for large log files, you may upload it in this Google Disk folder and link it in the issue!

❗ If you think the v1.15.2-rc3 branch is not working well, especially the resource manager, keep all the logs for describing the issue and disable the resource manager with LOTUS_RCMGR=0, if the problem still persists revert back to the stable v1.15.1 release.

tmyuu · 2022-05-02T20:43:03Z

tmyuu
May 2, 2022

go version

$ go version
go version go1.18.1 linux/amd64

lotus version

$ lotus-miner version
Daemon:  1.15.2-rc2+mainnet+git.94d9dee0d+api1.5.0
Local: lotus-miner version 1.15.2-rc2+mainnet+git.94d9dee0d
$ lotus version
Daemon:  1.15.2-rc2+mainnet+git.94d9dee0d+api1.5.0
Local: lotus version 1.15.2-rc2+mainnet+git.94d9dee0d

boost version

$ boostd -v
boostd version 0.0.1+git.067a458.dirty

check box

If you have successfully sealed some new sectors on this tag.
If you have successfully made some Storage Deals on this tag.
If you have successfully made some Snap Deals on this tag.
If you passed your first WindowPoSt on this tag.
If the log is back to normal.
The resource manager has not caused any connection problems.

0 replies

jamerduhgamer · 2022-05-02T23:36:16Z

jamerduhgamer
May 2, 2022

If you have successfully sealed some new sectors on this tag.
If you have successfully made some Storage Deals on this tag.
If you have successfully made some Snap Deals on this tag.
If you passed your first WindowPoSt on this tag.
If the log is back to normal.
The resource manager has not caused any connection problems.

0 replies

SBudo · 2022-05-03T09:39:26Z

SBudo
May 3, 2022

go version

$ go version
go version go1.18.1 linux/amd64

lotus version

$ lotus-miner version
Daemon:  1.15.2-rc3+mainnet+git.cfe201450.dirty+api1.5.0
Local: lotus-miner version 1.15.2-rc3+mainnet+git.cfe201450.dirty


$ lotus version
Daemon:  1.15.2-rc3+mainnet+git.cfe201450.dirty+api1.5.0
Local: lotus version 1.15.2-rc3+mainnet+git.cfe201450.dirty

boost version

$ boost -v
boost version 0.0.1+git.10c5ec8

Checks

If you have successfully sealed some new sectors on this tag.
If you have successfully made some Storage Deals on this tag.
If you have successfully made some Snap Deals on this tag.
If you passed your first WindowPoSt on this tag.
If the log is back to normal.
The resource manager has not caused any connection problems.

0 replies

cryptowhizzard · 2022-05-03T10:10:00Z

cryptowhizzard
May 3, 2022

go version

$ go version
go version go1.18.1 linux/amd64

lotus version

$ lotus-miner version
Local: lotus-miner version 1.15.2-rc3+mainnet+git.cfe201450

$ lotus version
Daemon:  1.15.2-rc2+mainnet+git.94d9dee0d+api1.5.0

boost version

$ boostd -v
boostd version 0.0.1+git.b11e3ea

check box

If you have successfully sealed some new sectors on this tag.
If you have successfully made some Storage Deals on this tag.
If you have successfully made some Snap Deals on this tag.
If you passed your first WindowPoSt on this tag.
If the log is back to normal.
The resource manager has not caused any connection problems.

Upgraded and running. Will populate the results whenever they are ready.

0 replies

benjaminh83 · 2022-05-03T14:16:53Z

benjaminh83
May 3, 2022

go version

$ go version
go version go1.17.9 linux/amd64

lotus version

$ lotus-miner version
Daemon:  1.15.2-rc3+mainnet+git.cfe201450+api1.5.0
Local: lotus-miner version 1.15.2-rc3+mainnet+git.cfe201450

boost version

$ boostd -v
boostd version 0.0.1+git.e9db5d6

check box

If you have successfully sealed some new sectors on this tag.
If you have successfully made some Storage Deals on this tag.
If you have successfully made some Snap Deals on this tag.
If you passed your first WindowPoSt on this tag.
If the log is back to normal.
The resource manager has not caused any connection problems.

Sealed sectors and WindowPoSt done on 1.15.2-rc2. Now upgraded and running again on rc3. Will populate the results whenever they are ready.

1 reply

benjaminh83 May 4, 2022

@Reiers Still seeing the error here in rc3:

[flexi_logger][ERRCODE::Time] flexi_logger has to work with UTC rather than with local time, caused by IndeterminateOffset
    See https://docs.rs/flexi_logger/latest/flexi_logger/error_info/index.html#time

But its not showing %Y-%m-%dT%H:%M:%S%.3f anymore, but does 2022-05-04T08:49:21.097, so I'm happy, as it is usable again.

Meest-NAM · 2022-05-04T02:40:46Z

Meest-NAM
May 4, 2022

lotus version

Daemon:  1.15.2-rc3+mainnet+git.cfe201450+api1.5.0
Local: lotus-miner version 1.15.2-rc3+mainnet+git.cfe201450

boost version

boostd version 0.0.1+git.067a458

go version

go version go1.17.9 linux/amd64

If you have successfully sealed some new sectors on this tag.
If you have successfully made some Storage Deals on this tag.
If you have successfully made some Snap Deals on this tag.
If you passed your first WindowPoSt on this tag.
If the log is back to normal.
The resource manager has not caused any connection problems.

2 replies

Meest-NAM May 6, 2022

2022-05-06T16:58:02.602+0900	WARN	storageminer	storage/wdpost_run.go:235	Checked sectors	{"checked": 2165, "good": 2165}
2022-05-06T16:58:02.649+0900	INFO	storageminer	storage/wdpost_run.go:647	running window post	{"chain-random": "7fToR5EOlu4sueQmV8Z7H/vpmjeIrQoF6rW32PX7RKo=", "deadline": {"CurrentEpoch":1783914,"PeriodStart":1783920,"Index":0,"Open":1783920,"Close":1783980,"Challenge":1783900,"FaultCutoff":1783850,"WPoStPeriodDeadlines":48,"WPoStProvingPeriod":2880,"WPoStChallengeWindow":60,"WPoStChallengeLookback":20,"FaultDeclarationCutoff":70}, "height": "1783914", "skipped": 0}
2022-05-06T16:58:02.649+0900	INFO	advmgr	sector-storage/manager_post.go:81	GenerateWindowPoSt run at lotus-miner
2022-05-06T07:58:02.680 INFO filcrypto::proofs::api > generate_window_post: start
2022-05-06T07:58:02.686 INFO filcrypto::proofs::api > generate_window_post: finish
2022-05-06T16:58:02.689+0900	INFO	storageminer	storage/wdpost_run.go:662	computing window post	{"batch": 0, "elapsed": 0.039357451, "skip": 0, "err": "could not read from path=\"p_aux\"\n\nCaused by:\n    No such file or directory (os error 2)\n\nStack backtrace:\n   0: anyhow::context::<impl anyhow::Context<T,E> for core::result::Result<T,E>>::with_context\n   1: filecoin_proofs::types::private_replica_info::PrivateReplicaInfo<Tree>::new\n   2: filecoin_proofs_api::post::generate_window_post_inner\n   3: filecoin_proofs_api::post::generate_window_post\n   4: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n   5: ffi_toolkit::catch_panic_response\n   6: fil_generate_window_post\n   7: _cgo_7059698500c4_Cfunc_fil_generate_window_post\n   8: runtime.asmcgocall\n             at /usr/local/go/src/runtime/asm_amd64.s:765", "errVerbose": "could not read from path=\"p_aux\"\n\nCaused by:\n    No such file or directory (os error 2)\n\nStack backtrace:\n   0: anyhow::context::<impl anyhow::Context<T,E> for core::result::Result<T,E>>::with_context\n   1: filecoin_proofs::types::private_replica_info::PrivateReplicaInfo<Tree>::new\n   2: filecoin_proofs_api::post::generate_window_post_inner\n   3: filecoin_proofs_api::post::generate_window_post\n   4: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n   5: ffi_toolkit::catch_panic_response\n   6: fil_generate_window_post\n   7: _cgo_7059698500c4_Cfunc_fil_generate_window_post\n   8: runtime.asmcgocall\n             at /usr/local/go/src/runtime/asm_amd64.s:765\ngithub.com/filecoin-project/filecoin-ffi.GenerateWindowPoSt\n\t/root/lotus/extern/filecoin-ffi/proofs.go:687\ngithub.com/filecoin-project/lotus/extern/sector-storage/ffiwrapper.(*Sealer).GenerateWindowPoSt\n\t/root/lotus/extern/sector-storage/ffiwrapper/verifier_cgo.go:48\ngithub.com/filecoin-project/lotus/extern/sector-storage.(*Manager).GenerateWindowPoSt\n\t/root/lotus/extern/sector-storage/manager_post.go:82\ngithub.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runPoStCycle\n\t/root/lotus/storage/wdpost_run.go:660\ngithub.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runGeneratePoST\n\t/root/lotus/storage/wdpost_run.go:96\ngithub.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).startGeneratePoST.func1\n\t/root/lotus/storage/wdpost_run.go:80\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581"}
2022-05-06T16:58:02.689+0900	ERROR	storageminer	storage/wdpost_run.go:664	error generating window post: could not read from path="p_aux"

Caused by:
    No such file or directory (os error 2)

Stack backtrace:
   0: anyhow::context::<impl anyhow::Context<T,E> for core::result::Result<T,E>>::with_context
   1: filecoin_proofs::types::private_replica_info::PrivateReplicaInfo<Tree>::new
   2: filecoin_proofs_api::post::generate_window_post_inner
   3: filecoin_proofs_api::post::generate_window_post
   4: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
   5: ffi_toolkit::catch_panic_response
   6: fil_generate_window_post
   7: _cgo_7059698500c4_Cfunc_fil_generate_window_post
   8: runtime.asmcgocall
             at /usr/local/go/src/runtime/asm_amd64.s:765
2022-05-06T16:58:02.689+0900	ERROR	storageminer	storage/wdpost_run.go:98	runPoStCycle failed: running window post failed:
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runPoStCycle
        /root/lotus/storage/wdpost_run.go:725
  - could not read from path="p_aux"
    
    Caused by:
        No such file or directory (os error 2)
    
    Stack backtrace:
       0: anyhow::context::<impl anyhow::Context<T,E> for core::result::Result<T,E>>::with_context
       1: filecoin_proofs::types::private_replica_info::PrivateReplicaInfo<Tree>::new
       2: filecoin_proofs_api::post::generate_window_post_inner
       3: filecoin_proofs_api::post::generate_window_post
       4: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
       5: ffi_toolkit::catch_panic_response
       6: fil_generate_window_post
       7: _cgo_7059698500c4_Cfunc_fil_generate_window_post
       8: runtime.asmcgocall
                 at /usr/local/go/src/runtime/asm_amd64.s:765
    github.com/filecoin-project/filecoin-ffi.GenerateWindowPoSt
    	/root/lotus/extern/filecoin-ffi/proofs.go:687
    github.com/filecoin-project/lotus/extern/sector-storage/ffiwrapper.(*Sealer).GenerateWindowPoSt
    	/root/lotus/extern/sector-storage/ffiwrapper/verifier_cgo.go:48
    github.com/filecoin-project/lotus/extern/sector-storage.(*Manager).GenerateWindowPoSt
    	/root/lotus/extern/sector-storage/manager_post.go:82
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runPoStCycle
    	/root/lotus/storage/wdpost_run.go:660
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runGeneratePoST
    	/root/lotus/storage/wdpost_run.go:96
    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).startGeneratePoST.func1
    	/root/lotus/storage/wdpost_run.go:80
    runtime.goexit
    	/usr/local/go/src/runtime/asm_amd64.s:1581

I don't know why
I'm doing this all of a sudden

Meest-NAM May 6, 2022

I checked all sectors of "Storage"

/storage/sealed/cache# ls | wc -l
2956

lotus-miner proving deadlines

deadline  partitions  sectors (faults)  proven partitions
0         1           2349 (2164)       0

jennijuju · 2022-05-04T05:59:45Z

jennijuju
May 4, 2022
Maintainer

Note - rc3 patch the bug where we got the wrong actor bundle that broke the FVM running.

0 replies

Angelo-gh3990 · 2022-05-04T08:43:14Z

Angelo-gh3990
May 4, 2022

Anybody getting or seen this error when compiling rc3 ? :

go build  -ldflags="-X=github.com/filecoin-project/lotus/build.CurrentCommit=+git.cfe201450" -o lotus ./cmd/lotus
go: downloading github.com/marten-seemann/qtls-go1-17 v0.1.1
# runtime/internal/sys
/usr/local/go/src/runtime/internal/sys/stubs.go:9:7: PtrSize redeclared in this block
        /usr/local/go/src/runtime/internal/sys/arch.go:24:38: previous declaration
/usr/local/go/src/runtime/internal/sys/stubs.go:10:24: undefined: Uintreg
/usr/local/go/src/runtime/internal/sys/stubs.go:16:7: StackGuardMultiplier redeclared in this block
        /usr/local/go/src/runtime/internal/sys/arch.go:27:74: previous declaration
make: *** [Makefile:87: lotus] Error 2

3 replies

Angelo-gh3990 May 4, 2022

/lotus# go version
go version go1.17.9 linux/amd64

Angelo-gh3990 May 4, 2022

Found it :

rm -rf /usr/local/go
wget -c https://golang.org/dl/go1.17.9.linux-amd64.tar.gz -O - | sudo tar -xz -C /usr/local

First I had to remove the previous install of Go (Is a known go issue apparently)

benjaminh83 May 4, 2022

Yes... you need to remove first. Same thing when we moved to v1.16 a while ago

TippyFlitsUK · 2022-05-04T19:54:31Z

TippyFlitsUK
May 4, 2022
Maintainer

If you have successfully sealed some new sectors on this tag.
If you have successfully made some Storage Deals on this tag.
If you have successfully made some Snap Deals on this tag.
If you passed your first WindowPoSt on this tag.
If the log is back to normal.
The resource manager has not caused any connection problems.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SPX v1.15.2-rc3 - Testing & Call Outs #8579

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 9 comments 6 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

SPX v1.15.2-rc3 - Testing & Call Outs #8579

Hello SPX Fellows

New features and Improvements

Reply card:

Reminders and Risk Mitigations

Replies: 9 comments · 6 replies

go version

lotus version

boost version

check box

go version

lotus version

boost version

Checks

go version

lotus version

boost version

check box

go version

lotus version

boost version

check box

lotus version

boost version

go version

jennijuju May 4, 2022 Maintainer

TippyFlitsUK May 4, 2022 Maintainer

Replies: 9 comments 6 replies

jennijuju
May 4, 2022
Maintainer

TippyFlitsUK
May 4, 2022
Maintainer