Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ValidateAvailability implementation #270

Merged
merged 48 commits into from
Apr 15, 2021
Merged
Show file tree
Hide file tree
Changes from 47 commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
3c24d62
implement GetLeafData
evan-forbes Mar 12, 2021
1f12d6b
use a real ipfs node instead of a mocked one during testing
evan-forbes Mar 16, 2021
d8f9190
Merge branch 'master' into evan/read-ipfs
evan-forbes Mar 16, 2021
1a5f9e3
use the ResolveNode method of the ipfs api instead of dag get
evan-forbes Mar 16, 2021
b69c0af
update CalcCIDPath tests
evan-forbes Mar 16, 2021
6dff524
first draft of RetrieveBlockData implementation
evan-forbes Mar 17, 2021
25a521d
Merge branch 'master' into evan/RetrieveBlockData
evan-forbes Mar 29, 2021
e692b0e
remove the parser interface
evan-forbes Mar 30, 2021
0d6b0f5
Merge branch 'master' into evan/RetrieveBlockData
evan-forbes Apr 6, 2021
e4be5d4
refactor leaf counter, fix bug
evan-forbes Apr 7, 2021
1b2335a
remove prints
evan-forbes Apr 7, 2021
28c6dee
update to the latest commit of rsmt2d
evan-forbes Apr 8, 2021
ca8c5b2
refactor RetrieveBlockData and fix bugs
evan-forbes Apr 8, 2021
cc7b7c4
increase test timeouts for CI
evan-forbes Apr 8, 2021
0b8c2ca
increase timeouts again :disappointed:
evan-forbes Apr 8, 2021
baf2fcc
add polish and change up timeouts for CI
evan-forbes Apr 8, 2021
af6a815
fix typo causing tests to fail
evan-forbes Apr 8, 2021
d10e141
revert remnant of debugging
evan-forbes Apr 8, 2021
60afd13
polish
evan-forbes Apr 8, 2021
4199979
re adjust timeouts
evan-forbes Apr 8, 2021
bb7fd2f
even longer timeout
evan-forbes Apr 8, 2021
f5677be
add buffer to hopefully help CI
evan-forbes Apr 8, 2021
152bb82
polish
evan-forbes Apr 8, 2021
98e1d72
Merge branch 'master' into evan/RetrieveBlockData
evan-forbes Apr 8, 2021
ffe1719
don't pin data to ipfs during testing
evan-forbes Apr 9, 2021
92e4132
use max block size
evan-forbes Apr 9, 2021
eba479b
retrieve block with minimum number of ipfs requests
evan-forbes Apr 12, 2021
0879f7a
go mod tidy + use cyrpto/rand instead of math/rand
evan-forbes Apr 12, 2021
43d420b
add no lint directive to weak random number generator
evan-forbes Apr 12, 2021
0510c67
set absurd timeout
evan-forbes Apr 12, 2021
69a7037
only run large tests with the race detector of
evan-forbes Apr 12, 2021
48360ca
review feedback: better docs
evan-forbes Apr 13, 2021
20d4c19
review feedback: better docs
evan-forbes Apr 13, 2021
50b691e
review feedback: smoother test sorting for a large tests
evan-forbes Apr 13, 2021
b44b303
feat(ipld): Sample and squareSampler implementation for random sampli…
Wondertan Apr 7, 2021
b1276a0
feat(ipld): ValidateAvailability implementation with minor test changes
Wondertan Apr 7, 2021
ad4e5b5
chore(ipld): go fmt && golangci-lint run
Wondertan Apr 7, 2021
cef85c7
chore(ipld): move out validation to separate files and fixes from rev…
Wondertan Apr 11, 2021
7171843
fix(ipld): add panic for invalid sampling case
Wondertan Apr 11, 2021
ac324ec
Update p2p/ipld/sample.go
Wondertan Apr 13, 2021
cf64d14
Update p2p/ipld/validate.go
Wondertan Apr 13, 2021
d93f9b7
Merge remote-tracking branch 'origin' into hlib/validate-avaiability
Wondertan Apr 13, 2021
23c9d06
fix(ipld): applied reveiw comments from @liamsi
Wondertan Apr 15, 2021
63e5acb
feat(crypto): Add new CRandSeed func for convinient and secure seed g…
Wondertan Apr 15, 2021
b517a38
fix(ipld): rework random number generation to be more secure
Wondertan Apr 15, 2021
adac1fb
tests(ipld): do not use Pinnning in TestValidateAvailability
Wondertan Apr 15, 2021
0d7669e
fix(ipld): use only crypto/rand for security critical ValidateAvailab…
Wondertan Apr 15, 2021
d2fede3
fix(crypto): remove CRandSeed function introduced recently that turne…
Wondertan Apr 15, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 13 additions & 2 deletions crypto/random.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ func randBytes(numBytes int) []byte {
return b
}

// This only uses the OS's randomness
// CRandBytes returns requested number of bytes from the OS's randomness.
func CRandBytes(numBytes int) []byte {
return randBytes(numBytes)
}
Expand All @@ -29,7 +29,18 @@ func CRandHex(numDigits int) string {
return hex.EncodeToString(CRandBytes(numDigits / 2))
}

// Returns a crand.Reader.
// CRandSeed returns a seed from the OS's randomness.
func CRandSeed(length int) int64 {
b := randBytes(length)
var seed uint64
for i := 0; i < 8; i++ {
seed |= uint64(b[i])
seed <<= 8
}
return int64(seed)
}
liamsi marked this conversation as resolved.
Show resolved Hide resolved

// CReader returns a crand.Reader.
func CReader() io.Reader {
return crand.Reader
}
12 changes: 1 addition & 11 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -599,13 +599,11 @@ github.com/kr/pretty v0.2.1 h1:Fmg33tUaq4/8ym9TJN1x7sLJnHVwhP33CNkpYV/7rwI=
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/pty v1.1.3/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/lazyledger/go-ipfs v0.8.0-lazypatch h1:8Dkw7Or6d0BmpFYFcxwgqWZ047BPGCsWtG7v9+H0ofk=
github.com/lazyledger/go-ipfs v0.8.0-lazypatch/go.mod h1:CE4cJkjUmwW5LwJP26KKEAZ11ZED0DxzSryfv5RMf6E=
github.com/lazyledger/go-leopard v0.0.0-20200604113236-298f93361181 h1:mUeCGuCgjZVadW4CzA2dMBq7p2BqaoCfpnKjxMmSaSE=
github.com/lazyledger/go-leopard v0.0.0-20200604113236-298f93361181/go.mod h1:v1o1CRihQ9i7hizx23KK4aR79lxA6VDUIzUCfDva0XQ=
github.com/lazyledger/go-leopard v0.0.0-20200724211609-50ec4b3fab41 h1:DTQODNWI71ZtqCT3wQg+RXl4K/zpu+hu5usISUhDq/E=
github.com/lazyledger/go-leopard v0.0.0-20200724211609-50ec4b3fab41/go.mod h1:v1o1CRihQ9i7hizx23KK4aR79lxA6VDUIzUCfDva0XQ=
Expand All @@ -615,7 +613,6 @@ github.com/lazyledger/merkletree v0.0.0-20201214195110-6901c4c3c75f h1:jbyPAH6o6
github.com/lazyledger/merkletree v0.0.0-20201214195110-6901c4c3c75f/go.mod h1:10PA0NlnYtB8HrtwIDQAyTKWp8TEZ0zBZCGlYC/7+QE=
github.com/lazyledger/nmt v0.3.1 h1:zP172RR33Es4dhb88GmUr9kBpAkH6Wcl7nQGJ3HQzu4=
github.com/lazyledger/nmt v0.3.1/go.mod h1:tY7ypPX26Sbkt6F8EbPl3AT33B5N0BJe4OVPbq849YI=
github.com/lazyledger/rsmt2d v0.1.1-0.20210327010029-ef1d6c54461e h1:3mwa4b4v9puYIFsfRN6TsXMbUagvhLtIzXWHN2GFHB4=
github.com/lazyledger/rsmt2d v0.1.1-0.20210327010029-ef1d6c54461e/go.mod h1:ORR2U7THCNr1fpUhwYqZN7QCFJ20iR2uiIWfXKz3KJ4=
github.com/lazyledger/rsmt2d v0.1.1-0.20210406153014-e1fd589bdb09 h1:5wpWhlalAm1vNpkR/L1BlWHyn8GT5XNieIZqTlKG9hc=
github.com/lazyledger/rsmt2d v0.1.1-0.20210406153014-e1fd589bdb09/go.mod h1:EbB1gGbX51gBNm0hC5lMcbkgEyO3Wj2RYwba9mCUvPA=
Expand Down Expand Up @@ -1168,7 +1165,6 @@ github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4k
github.com/sony/gobreaker v0.4.1/go.mod h1:ZKptC7FHNvhBz7dN2LGjPVBz2sZJmc0/PkyDJOjmxWY=
github.com/sourcegraph/annotate v0.0.0-20160123013949-f4cad6c6324d/go.mod h1:UdhH50NIW0fCiwBSr0co2m7BnFLdv4fQTgdqdJTHFeE=
github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e/go.mod h1:HuIsMU8RRBOtsCgI77wP899iHVBQpCmg4ErYMZB+2IA=
github.com/spacemonkeygo/errors v0.0.0-20171212215202-9064522e9fd1 h1:xHQewZjohU9/wUsyC99navCjQDNHtTgUOM/J1jAbzfw=
github.com/spacemonkeygo/errors v0.0.0-20171212215202-9064522e9fd1/go.mod h1:7NL9UAYQnRM5iKHUCld3tf02fKb5Dft+41+VckASUy0=
github.com/spacemonkeygo/openssl v0.0.0-20181017203307-c2dcc5cca94a/go.mod h1:7AyxJNCJ7SBZ1MfVQCWD6Uqo2oubI2Eq2y2eqf+A5r0=
github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572 h1:RC6RW7j+1+HkWaX/Yh71Ee5ZHaHYt7ZP4sQgUrm6cDU=
Expand Down Expand Up @@ -1229,7 +1225,6 @@ github.com/urfave/cli v1.22.1/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtX
github.com/urfave/cli/v2 v2.0.0/go.mod h1:SE9GqnLQmjVa0iPEY0f1w3ygNIYcIJ0OKPMoW2caLfQ=
github.com/viant/assertly v0.4.8/go.mod h1:aGifi++jvCrUaklKEKT0BU95igDNaqkvz+49uaYMPRU=
github.com/viant/toolbox v0.24.0/go.mod h1:OxMCG57V0PXuIP2HNQrtJf2CjqdmbrOx5EkMILuUhzM=
github.com/vivint/infectious v0.0.0-20190108171102-2455b059135b h1:dLkqBELopfQNhe8S9ucnSf+HhiUCgK/hPIjVG0f9GlY=
github.com/vivint/infectious v0.0.0-20190108171102-2455b059135b/go.mod h1:5oyMAv4hrBEKqBwORFsiqIrCNCmL2qcZLQTdJLYeYIc=
github.com/vivint/infectious v0.0.0-20200605153912-25a574ae18a3 h1:zMsHhfK9+Wdl1F7sIKLyx3wrOFofpb3rWFbA4HgcK5k=
github.com/vivint/infectious v0.0.0-20200605153912-25a574ae18a3/go.mod h1:R0Gbuw7ElaGSLOZUSwBm/GgVwMd30jWxBDdAyMOeTuc=
Expand Down Expand Up @@ -1267,7 +1262,6 @@ github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.1.32/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
gitlab.com/NebulousLabs/errors v0.0.0-20171229012116-7ead97ef90b8 h1:gZfMjx7Jr6N8b7iJO4eUjDsn6xJqoyXg8D+ogdoAfKY=
gitlab.com/NebulousLabs/errors v0.0.0-20171229012116-7ead97ef90b8/go.mod h1:ZkMZ0dpQyWwlENaeZVBiQRjhMEZvk6VTXquzl3FOFP8=
gitlab.com/NebulousLabs/errors v0.0.0-20200929122200-06c536cf6975 h1:L/ENs/Ar1bFzUeKx6m3XjlmBgIUlykX9dzvp5k9NGxc=
gitlab.com/NebulousLabs/errors v0.0.0-20200929122200-06c536cf6975/go.mod h1:ZkMZ0dpQyWwlENaeZVBiQRjhMEZvk6VTXquzl3FOFP8=
Expand Down Expand Up @@ -1359,6 +1353,7 @@ golang.org/x/exp v0.0.0-20191030013958-a1ab85dbe136/go.mod h1:JXzH8nQsPlswgeRAPE
golang.org/x/exp v0.0.0-20191129062945-2f5052295587/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM=
golang.org/x/exp v0.0.0-20200331195152-e8c3332aa8e5/go.mod h1:4M0jN8W1tt0AVLNr8HDosyJCDCDuyL9N9+3m7wDWgKw=
golang.org/x/image v0.0.0-20180708004352-c73c2afc3b81/go.mod h1:ux5Hcp/YLpHSI86hEcLt0YII63i6oz57MZXIpbrjZUs=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
Expand Down Expand Up @@ -1419,7 +1414,6 @@ golang.org/x/net v0.0.0-20200520182314-0ba52f642ac2/go.mod h1:qpuaurCH72eLCgpAm/
golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20200813134508-3edf25e44fcc/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20201021035429-f5854403a974 h1:IX6qOQeG5uLjB/hjjwjedwfjND0hgjPMMyO1RoIXQNI=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110 h1:qWPm9rbaAMKs8Bq/9LRpbMqxWRVUAQwMI9fVrssnTfw=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
Expand Down Expand Up @@ -1495,7 +1489,6 @@ golang.org/x/sys v0.0.0-20200615200032-f1bc736245b1/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200814200057-3d37ad5750ed/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201015000850-e3ed0017c211 h1:9UQO31fZ+0aKQOFldThf7BKPMJTiBfWycGh/u3UoO88=
golang.org/x/sys v0.0.0-20201015000850-e3ed0017c211/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210403161142-5e06dd20ab57 h1:F5Gozwx4I1xtr/sr/8CFbb57iKi3297KFs0QDbGN60A=
Expand Down Expand Up @@ -1561,9 +1554,7 @@ golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8T
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gonum.org/v1/gonum v0.0.0-20180816165407-929014505bf4/go.mod h1:Y+Yx5eoAFn32cQvJDxZx5Dpnq+c3wtXuadVZAcxbbBo=
gonum.org/v1/gonum v0.7.0 h1:Hdks0L0hgznZLG9nzXb8vZ0rRvqNvAcgAp84y7Mwkgw=
gonum.org/v1/gonum v0.7.0/go.mod h1:L02bwd0sqlsvRv41G7wGWFCsVNZFv/k1xzGIxeANHGM=
gonum.org/v1/netlib v0.0.0-20190313105609-8cb42192e0e0 h1:OE9mWmgKkjJyEmDAAtGMPjXu+YNeGvK9VTSHY6+Qihc=
gonum.org/v1/netlib v0.0.0-20190313105609-8cb42192e0e0/go.mod h1:wa6Ws7BG/ESfp6dHfk7C6KdzKA7wR7u/rKwOGE66zvw=
gonum.org/v1/plot v0.0.0-20190515093506-e2840ee46a6b/go.mod h1:Wt8AAjI+ypCyYX3nZBvf6cAIx93T+c/OS2HFAYskSZc=
google.golang.org/api v0.0.0-20180910000450-7ca32eb868bf/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0=
Expand Down Expand Up @@ -1663,7 +1654,6 @@ gopkg.in/yaml.v2 v2.2.5/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU=
gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b h1:h8qDotaEPuJATrMmW04NCwg7v22aHH28wwpauUhK9Oo=
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
Expand Down
5 changes: 3 additions & 2 deletions p2p/ipld/read.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,13 @@ import (
"github.com/ipfs/go-cid"
coreiface "github.com/ipfs/interface-go-ipfs-core"
"github.com/ipfs/interface-go-ipfs-core/path"
"github.com/lazyledger/rsmt2d"

"github.com/lazyledger/lazyledger-core/p2p/ipld/plugin/nodes"
"github.com/lazyledger/lazyledger-core/types"
"github.com/lazyledger/rsmt2d"
)

// //////////////////////////////////////
// ///////////////////////////////////////
// Retrieve Block Data
// ////////////////////////////////////

Expand Down
51 changes: 25 additions & 26 deletions p2p/ipld/read_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,19 @@ import (
"testing"
"time"

cid "github.com/ipfs/go-cid"
"github.com/ipfs/go-cid"
"github.com/ipfs/go-ipfs/core/coreapi"

coremock "github.com/ipfs/go-ipfs/core/mock"
format "github.com/ipfs/go-ipld-format"
"github.com/lazyledger/lazyledger-core/p2p/ipld/plugin/nodes"
"github.com/lazyledger/lazyledger-core/types"
iface "github.com/ipfs/interface-go-ipfs-core"
"github.com/lazyledger/nmt"
"github.com/lazyledger/nmt/namespace"
"github.com/lazyledger/rsmt2d"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"

"github.com/lazyledger/lazyledger-core/p2p/ipld/plugin/nodes"
"github.com/lazyledger/lazyledger-core/types"
)

var raceDetectorActive = false
Expand Down Expand Up @@ -98,20 +99,12 @@ func TestGetLeafData(t *testing.T) {
leaves [][]byte
}

// create a mock node
ipfsNode, err := coremock.NewMockNode()
if err != nil {
t.Error(err)
}
// create the context and batch needed for node collection from the tree
ctx, cancel := context.WithCancel(context.Background())
defer cancel()

// issue a new API object
ipfsAPI, err := coreapi.NewCoreAPI(ipfsNode)
if err != nil {
t.Error(err)
}

// create the context and batch needed for node collection from the tree
ctx := context.Background()
ipfsAPI := mockedIpfsAPI(t)
batch := format.NewBatch(ctx, ipfsAPI.Dag())

// generate random data for the nmt
Expand Down Expand Up @@ -238,17 +231,8 @@ func TestRetrieveBlockData(t *testing.T) {
errStr string
}

// create a mock node
ipfsNode, err := coremock.NewMockNode()
if err != nil {
t.Error(err)
}

// issue a new API object
ipfsAPI, err := coreapi.NewCoreAPI(ipfsNode)
if err != nil {
t.Error(err)
}
ipfsAPI := mockedIpfsAPI(t)

// the max size of messages that won't get split
adjustedMsgSize := types.MsgShareSize - 2
Expand Down Expand Up @@ -447,3 +431,18 @@ func generateRandomContiguousShares(count int) types.Txs {
}
return txs
}

func mockedIpfsAPI(t *testing.T) iface.CoreAPI {
ipfsNode, err := coremock.NewMockNode()
if err != nil {
t.Error(err)
}

// issue a new API object
ipfsAPI, err := coreapi.NewCoreAPI(ipfsNode)
if err != nil {
t.Error(err)
}

return ipfsAPI
}
108 changes: 108 additions & 0 deletions p2p/ipld/sample.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
package ipld

import (
crand "crypto/rand"
"math/big"

"github.com/ipfs/go-cid"
"github.com/lazyledger/nmt/namespace"

"github.com/lazyledger/lazyledger-core/p2p/ipld/plugin/nodes"
"github.com/lazyledger/lazyledger-core/types"
)

// Sample is a point in 2D space over square.
type Sample struct {
Row, Col uint32

// src defines the source for sampling, either from column(true) or row(false) root
src bool
}

// SampleSquare randomly picks *num* unique points from arbitrary *width* square
// and returns them as samples.
func SampleSquare(squareWidth uint32, num int) []Sample {
ss := newSquareSampler(squareWidth, num)
ss.sample(num)
return ss.samples()
}

// Leaf returns leaf info needed for retrieval using data provided with DAHeader.
func (s Sample) Leaf(dah *types.DataAvailabilityHeader) (cid.Cid, uint32, error) {
var (
leaf uint32
root namespace.IntervalDigest
)

// spread leaves retrieval from both Row and Column roots
if s.src {
root = dah.ColumnRoots[s.Col]
leaf = s.Row
} else {
root = dah.RowsRoots[s.Row]
leaf = s.Col
}

rootCid, err := nodes.CidFromNamespacedSha256(root.Bytes())
if err != nil {
return cid.Undef, 0, err
}

return rootCid, leaf, nil
}

// Equals check whenever to samples are equal.
func (s Sample) Equals(to Sample) bool {
return s.Row == to.Row && s.Col == to.Col
}
evan-forbes marked this conversation as resolved.
Show resolved Hide resolved

type squareSampler struct {
squareWidth uint32
smpls map[Sample]struct{}
}

func newSquareSampler(squareWidth uint32, expectedSamples int) *squareSampler {
return &squareSampler{
squareWidth: squareWidth,
smpls: make(map[Sample]struct{}, expectedSamples),
}
}

func (ss *squareSampler) sample(num int) {
Copy link
Member

@liamsi liamsi Apr 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: I think the name generateSamples or simply generate would be clearer. Otherwise, it sounds like this would already sample the network.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't agree here. Generate sound like producing something imho, also from the name squareSampler the method sample look more idiomatic

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generate sound like producing something

We are producing something: random numbers which comprise Sample objects. These are then used to actually sample the network (later, but not here). Doesn't really matter much as this is a private method though.

if uint32(num) > ss.squareWidth*ss.squareWidth {
panic("number of samples must be less than square width")
}

done := 0
for done < num {
s := Sample{
Row: randUint32(ss.squareWidth),
Col: randUint32(ss.squareWidth),
src: randUint32(2) == 0,
}

if _, ok := ss.smpls[s]; ok {
continue
}

done++
ss.smpls[s] = struct{}{}
}
}

func (ss *squareSampler) samples() []Sample {
samples := make([]Sample, 0, len(ss.smpls))
for s := range ss.smpls {
samples = append(samples, s)
}
return samples
}

func randUint32(max uint32) uint32 {
n, err := crand.Int(crand.Reader, big.NewInt(int64(max)))
if err != nil {
panic(err) // won't panic as rand.Reader is endless
}

return uint32(n.Int64())
}
35 changes: 35 additions & 0 deletions p2p/ipld/sample_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
package ipld

import (
"testing"

"github.com/stretchr/testify/assert"
)

func TestSampleSquare(t *testing.T) {
tests := []struct {
width uint32
samples int
}{
{width: 10, samples: 5},
{width: 500, samples: 90},
}

for _, tt := range tests {
ss := SampleSquare(tt.width, tt.samples)
assert.Len(t, ss, tt.samples)
// check points are within width
for _, s := range ss {
assert.Less(t, s.Row, tt.width)
assert.Less(t, s.Col, tt.width)
}
// checks samples are not equal
for i, s1 := range ss {
for j, s2 := range ss {
if i != j {
assert.False(t, s1.Equals(s2))
}
}
}
}
}
Loading