Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ParallelEnumerate and Enumerate #75

Merged
merged 75 commits into from
Dec 23, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
35374cb
copy code to try to fix ParallelEnumerate
nietras Nov 19, 2023
daeba21
copy rest of stuff from parallel trial
nietras Nov 19, 2023
156c7a6
add package assets Sep_MT
nietras Nov 19, 2023
06bccab
try fix MT run
nietras Nov 19, 2023
f6964a3
dead locked asparallel
nietras Nov 26, 2023
3cf12eb
dont dispose or clean up :|
nietras Nov 26, 2023
ebfa6dc
minor edits
nietras Nov 28, 2023
35faaf9
use charsIndex only in parser, as prep for parse multiple rows
nietras Nov 28, 2023
5852a15
refactor SepParserAvx2PackCmpOrMoveMaskTzcnt to use goto
nietras Dec 6, 2023
db5574b
sick quick edits to avx2 to try multi row parsing, failing
nietras Dec 6, 2023
1322231
fix many issues
nietras Dec 7, 2023
2fedbb7
remove LineNumberFrom from SepRowInfo
nietras Dec 8, 2023
2014d6c
initial parallel runs and parses tests, but slow
nietras Dec 8, 2023
1838bfd
Add thread safe SepToStringHashPoolPerColThreadSafe via ToStringThrea…
nietras Dec 9, 2023
c8bc971
use ConcurrentStack for state in AsParallel impl, ensure dispose
nietras Dec 9, 2023
ecd65be
Add ThreadLocal to cache last for ToStringThreadSafe
nietras Dec 9, 2023
80117cb
nit benchmark changes
nietras Dec 9, 2023
5255800
cleanup AsParallel impl based on SepReaderState
nietras Dec 9, 2023
9cba32d
REMOVE OLD PARALLEL ENUMERATE BRAINSTORMING CODE
nietras Dec 9, 2023
a314370
cleanup code to remove warnings
nietras Dec 9, 2023
86fe6c4
change ISepParser to no longer return rowLineEndingOffset, change Sep…
nietras Dec 10, 2023
95b6e07
increase initial col infos size to fix unescape perf diff
nietras Dec 10, 2023
3e6e23f
try different parse ways incl ForLoop instead of gotos
nietras Dec 10, 2023
b2338f8
cleanup SepParserAvx2 before duplicating
nietras Dec 10, 2023
176f09e
fix SepParseMaskTest_ParseSeparatorsLineEndingsMasks_Ordinary
nietras Dec 10, 2023
16fe19d
rename member field names to be more consistent, hacks to fix parser …
nietras Dec 10, 2023
bfb1953
MoveMask as separate method for easier duplicating
nietras Dec 11, 2023
1eed5a8
copy Avx2 changes to Sse2
nietras Dec 11, 2023
32491b1
copy Avx2 changes to Avx512
nietras Dec 11, 2023
d616da5
copy Avx512 changes to SepParserVector512NrwCmpExtMsbTzcnt
nietras Dec 11, 2023
9ba75d5
copy Vector512 changes to Vector256
nietras Dec 11, 2023
32309c6
copy to Vector128, Vector64
nietras Dec 11, 2023
059a32a
fix IndexOfAny
nietras Dec 11, 2023
e21097d
add ability to test all parsers via environment variable SEPFORCEPARS…
nietras Dec 11, 2023
98ff157
nit
nietras Dec 11, 2023
c79db72
Use ArrayPool for ParallelEnumerate
nietras Dec 15, 2023
6a57392
Add SepStringHashPoolThreadSafeFixedCapacity to improve assets bench …
nietras Dec 15, 2023
f9e2132
Fix SepStringHashPool use _count to entries length
nietras Dec 15, 2023
b482a81
Add TryRowFunc and enumerate overloads taking it
nietras Dec 15, 2023
3f69fe2
cleanup enumeration
nietras Dec 15, 2023
4a0c62a
remove using
nietras Dec 15, 2023
b7d0706
rename extensions file
nietras Dec 17, 2023
0d2815a
try parallel enumeration with separate parser Task.Run
nietras Dec 17, 2023
a0c6366
remove separate Task based parallel enumerate as only slower (more ov…
nietras Dec 17, 2023
f080509
cleanup usings
nietras Dec 17, 2023
f7c2e29
Sep__MT_Unescape
nietras Dec 17, 2023
84c7f9d
much faster PackageAssetsBench with <ServerGarbageCollection>true</Se…
nietras Dec 17, 2023
92db21d
cleanup reader code regarding chars buffer and reads, rename fixed ca…
nietras Dec 18, 2023
7d651d7
expand SepToStringTest
nietras Dec 18, 2023
d60c045
disable last cache for thread-safe in fixed capacity
nietras Dec 18, 2023
c0f7119
expand PackageAssetsTest to cover different ToStrings and fix OnePool…
nietras Dec 18, 2023
14ce997
cover ParallelEnumerate RowTryFunc
nietras Dec 18, 2023
733f8d8
cover Enumerate TryFunc
nietras Dec 18, 2023
93d2355
nits
nietras Dec 18, 2023
575b411
Merge branch 'main' into parallelenumerate
nietras Dec 18, 2023
8c48ed0
bump api validation to 0.3.0 and remove suppressions
nietras Dec 18, 2023
51a0c10
update README with new API additions
nietras Dec 18, 2023
291af44
refactor SepParserFactory and improve test coverage
nietras Dec 19, 2023
24b84e5
few code coverage nits
nietras Dec 19, 2023
0a365ca
add test-parsers to CI
nietras Dec 19, 2023
7abf8fe
cleanup usings
nietras Dec 19, 2023
e392ec9
interim bench update
nietras Dec 20, 2023
2b696ae
add long run (1m rows) for assets, rename MT
nietras Dec 20, 2023
211e00b
add GcServer true runs for package asset
nietras Dec 20, 2023
24e9824
remove gc server bench
nietras Dec 21, 2023
dfa70a8
Update 5950X benchmarks
nietras Dec 22, 2023
39a3fb5
add SepReader.Row.UnsafeToStringDelegate
nietras Dec 23, 2023
e3319e0
static delegate
nietras Dec 23, 2023
35e5338
expand README on Enumerate and ParallelEnumerate
nietras Dec 23, 2023
e159e5a
quick half fix to SepParserBench
nietras Dec 23, 2023
b03eeff
make SepParserBench workable
nietras Dec 23, 2023
e5b0256
REMOVE SepLockFreeReaderHashSet
nietras Dec 23, 2023
b5c8ae6
remove comment for internal UnsafeToStringDelegate
nietras Dec 23, 2023
e1e1954
rename internal CopyParsedRowsTo
nietras Dec 23, 2023
65e649b
cleanup test-parsers.ps1
nietras Dec 23, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .github/workflows/dotnet.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,28 @@ jobs:
with:
flags: ${{ matrix.os }},${{ matrix.configuration }}

test-parsers:
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
configuration: [Debug, Release]

runs-on: ${{ matrix.os }}

steps:
- uses: actions/checkout@v3
- name: Setup .NET
uses: actions/setup-dotnet@v3
with:
global-json-file: global.json
- name: Restore dependencies
run: dotnet restore
- name: Build
run: dotnet build -c ${{ matrix.configuration }} --no-restore
- name: Test Parsers
shell: pwsh
run: ./test-parsers.ps1

pack:
runs-on: windows-latest
steps:
Expand Down
253 changes: 176 additions & 77 deletions README.md

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions Sep.sln
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "root", "root", "{32021D88-4
publish-tester.ps1 = publish-tester.ps1
README.md = README.md
rename.ps1 = rename.ps1
test-parsers.ps1 = test-parsers.ps1
test-x64-x86.ps1 = test-x64-x86.ps1
EndProjectSection
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Sep.Tester", "src\Sep.Tester\Sep.Tester.csproj", "{28D41891-53C4-4FE9-8ED4-A8880E766E16}"
Expand Down
29 changes: 15 additions & 14 deletions benchmarks/AMD.Ryzen.9.5950X/FloatsReaderBench.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
```

BenchmarkDotNet v0.13.10, Windows 10 (10.0.19044.3086/21H2/November2021Update)
BenchmarkDotNet v0.13.11, Windows 10 (10.0.19044.3086/21H2/November2021Update)
AMD Ryzen 9 5950X, 1 CPU, 32 logical and 16 physical cores
.NET SDK 8.0.100
[Host] : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX2
Job-HCXKGU : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX2
Job-PQAZLF : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX2

Runtime=.NET 8.0 Toolchain=net80 InvocationCount=Default
IterationTime=350.0000 ms MaxIterationCount=15 MinIterationCount=5
Expand All @@ -13,17 +13,18 @@ WarmupCount=6 Reader=String
```
| Method | Scope | Rows | Mean | Ratio | MB | MB/s | ns/row | Allocated | Alloc Ratio |
|---------- |------- |------ |-----------:|------:|---:|--------:|-------:|------------:|------------:|
| Sep______ | Row | 25000 | 1.893 ms | 1.00 | 20 | 10733.4 | 75.7 | 1.15 KB | 1.00 |
| Sylvan___ | Row | 25000 | 2.212 ms | 1.17 | 20 | 9187.9 | 88.5 | 10.02 KB | 8.74 |
| ReadLine_ | Row | 25000 | 10.913 ms | 5.76 | 20 | 1862.0 | 436.5 | 73489.63 KB | 64,099.99 |
| CsvHelper | Row | 25000 | 25.079 ms | 13.25 | 20 | 810.2 | 1003.2 | 20.58 KB | 17.95 |
| Sep______ | Row | 25000 | 1.936 ms | 1.00 | 20 | 10498.4 | 77.4 | 1.18 KB | 1.00 |
| Sylvan___ | Row | 25000 | 2.259 ms | 1.17 | 20 | 8995.4 | 90.4 | 10.02 KB | 8.50 |
| ReadLine_ | Row | 25000 | 11.063 ms | 5.69 | 20 | 1836.8 | 442.5 | 73489.64 KB | 62,295.85 |
| CsvHelper | Row | 25000 | 25.135 ms | 12.99 | 20 | 808.4 | 1005.4 | 20.58 KB | 17.45 |
| | | | | | | | | | |
| Sep______ | Cols | 25000 | 2.591 ms | 1.00 | 20 | 7843.3 | 103.6 | 1.15 KB | 1.00 |
| Sylvan___ | Cols | 25000 | 3.738 ms | 1.44 | 20 | 5436.2 | 149.5 | 10.03 KB | 8.74 |
| ReadLine_ | Cols | 25000 | 11.010 ms | 4.25 | 20 | 1845.6 | 440.4 | 73489.64 KB | 64,045.44 |
| CsvHelper | Cols | 25000 | 27.201 ms | 10.43 | 20 | 747.0 | 1088.0 | 21340.99 KB | 18,598.45 |
| Sep______ | Cols | 25000 | 2.566 ms | 1.00 | 20 | 7918.8 | 102.6 | 1.18 KB | 1.00 |
| Sylvan___ | Cols | 25000 | 3.821 ms | 1.49 | 20 | 5318.1 | 152.8 | 10.03 KB | 8.48 |
| ReadLine_ | Cols | 25000 | 11.675 ms | 4.54 | 20 | 1740.5 | 467.0 | 73489.64 KB | 62,141.53 |
| CsvHelper | Cols | 25000 | 27.400 ms | 10.67 | 20 | 741.6 | 1096.0 | 21340.81 KB | 18,045.40 |
| | | | | | | | | | |
| Sep______ | Floats | 25000 | 21.397 ms | 1.00 | 20 | 949.7 | 855.9 | 7.87 KB | 1.00 |
| Sylvan___ | Floats | 25000 | 65.771 ms | 3.08 | 20 | 308.9 | 2630.8 | 18.2 KB | 2.31 |
| ReadLine_ | Floats | 25000 | 72.513 ms | 3.39 | 20 | 280.2 | 2900.5 | 73493.12 KB | 9,338.25 |
| CsvHelper | Floats | 25000 | 106.198 ms | 4.96 | 20 | 191.3 | 4247.9 | 22062.55 KB | 2,803.33 |
| Sep______ | Floats | 25000 | 21.852 ms | 1.00 | 20 | 929.9 | 874.1 | 7.93 KB | 1.00 |
| Sep_MT___ | Floats | 25000 | 3.916 ms | 0.18 | 20 | 5188.4 | 156.7 | 178.87 KB | 22.56 |
| Sylvan___ | Floats | 25000 | 71.861 ms | 3.29 | 20 | 282.8 | 2874.4 | 18.2 KB | 2.30 |
| ReadLine_ | Floats | 25000 | 73.981 ms | 3.39 | 20 | 274.7 | 2959.2 | 73493.12 KB | 9,271.52 |
| CsvHelper | Floats | 25000 | 102.224 ms | 4.68 | 20 | 198.8 | 4089.0 | 22062.55 KB | 2,783.30 |
50 changes: 28 additions & 22 deletions benchmarks/AMD.Ryzen.9.5950X/PackageAssetsBench.md
Original file line number Diff line number Diff line change
@@ -1,33 +1,39 @@
```

BenchmarkDotNet v0.13.10, Windows 10 (10.0.19044.3086/21H2/November2021Update)
BenchmarkDotNet v0.13.11, Windows 10 (10.0.19044.3086/21H2/November2021Update)
AMD Ryzen 9 5950X, 1 CPU, 32 logical and 16 physical cores
.NET SDK 8.0.100
[Host] : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX2
Job-TUSNIO : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX2
Job-PQAZLF : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX2

Job=Job-TUSNIO Runtime=.NET 8.0 Toolchain=net80
Job=Job-PQAZLF Runtime=.NET 8.0 Toolchain=net80
InvocationCount=Default IterationTime=350.0000 ms MaxIterationCount=15
MinIterationCount=5 WarmupCount=6 Quotes=False
Reader=String

```
| Method | Scope | Rows | Mean | Ratio | MB | MB/s | ns/row | Allocated | Alloc Ratio |
|------------- |------ |------ |-----------:|------:|---:|--------:|-------:|------------:|------------:|
| Sep______ | Row | 50000 | 2.409 ms | 1.00 | 29 | 12111.6 | 48.2 | 933 B | 1.00 |
| Sep_Unescape | Row | 50000 | 2.499 ms | 1.04 | 29 | 11675.7 | 50.0 | 934 B | 1.00 |
| Sylvan___ | Row | 50000 | 2.913 ms | 1.21 | 29 | 10016.5 | 58.3 | 7381 B | 7.91 |
| ReadLine_ | Row | 50000 | 12.198 ms | 5.04 | 29 | 2392.3 | 244.0 | 90734838 B | 97,250.63 |
| CsvHelper | Row | 50000 | 42.460 ms | 17.63 | 29 | 687.3 | 849.2 | 21074 B | 22.59 |
| | | | | | | | | | |
| Sep______ | Cols | 50000 | 3.480 ms | 1.00 | 29 | 8385.3 | 69.6 | 935 B | 1.00 |
| Sep_Unescape | Cols | 50000 | 3.959 ms | 1.14 | 29 | 7370.9 | 79.2 | 936 B | 1.00 |
| Sylvan___ | Cols | 50000 | 5.210 ms | 1.50 | 29 | 5600.5 | 104.2 | 7385 B | 7.90 |
| ReadLine_ | Cols | 50000 | 14.275 ms | 3.60 | 29 | 2044.3 | 285.5 | 90734826 B | 97,042.59 |
| CsvHelper | Cols | 50000 | 69.058 ms | 19.86 | 29 | 422.6 | 1381.2 | 457060 B | 488.83 |
| | | | | | | | | | |
| Sep______ | Asset | 50000 | 35.052 ms | 1.00 | 29 | 832.5 | 701.0 | 14131121 B | 1.00 |
| Sep_Unescape | Asset | 50000 | 34.813 ms | 1.00 | 29 | 838.2 | 696.3 | 14130768 B | 1.00 |
| Sylvan___ | Asset | 50000 | 38.875 ms | 1.12 | 29 | 750.6 | 777.5 | 14297351 B | 1.01 |
| ReadLine_ | Asset | 50000 | 127.866 ms | 3.76 | 29 | 228.2 | 2557.3 | 104584604 B | 7.40 |
| CsvHelper | Asset | 50000 | 86.415 ms | 2.47 | 29 | 337.7 | 1728.3 | 14306352 B | 1.01 |
| Method | Scope | Rows | Mean | Ratio | MB | MB/s | ns/row | Allocated | Alloc Ratio |
|------------- |------ |-------- |-------------:|------:|----:|--------:|-------:|-------------:|------------:|
| Sep______ | Row | 50000 | 2.319 ms | 1.00 | 29 | 12581.3 | 46.4 | 967 B | 1.00 |
| Sep_Unescape | Row | 50000 | 2.335 ms | 1.01 | 29 | 12497.4 | 46.7 | 967 B | 1.00 |
| Sylvan___ | Row | 50000 | 2.985 ms | 1.29 | 29 | 9776.4 | 59.7 | 7381 B | 7.63 |
| ReadLine_ | Row | 50000 | 12.779 ms | 5.55 | 29 | 2283.6 | 255.6 | 90734838 B | 93,831.27 |
| CsvHelper | Row | 50000 | 43.702 ms | 18.84 | 29 | 667.7 | 874.0 | 21074 B | 21.79 |
| | | | | | | | | | |
| Sep______ | Cols | 50000 | 3.127 ms | 1.00 | 29 | 9331.2 | 62.5 | 970 B | 1.00 |
| Sep_Unescape | Cols | 50000 | 3.757 ms | 1.20 | 29 | 7766.9 | 75.1 | 972 B | 1.00 |
| Sylvan___ | Cols | 50000 | 5.353 ms | 1.71 | 29 | 5451.1 | 107.1 | 7385 B | 7.61 |
| ReadLine_ | Cols | 50000 | 12.753 ms | 4.07 | 29 | 2288.1 | 255.1 | 90734839 B | 93,541.07 |
| CsvHelper | Cols | 50000 | 69.377 ms | 22.20 | 29 | 420.6 | 1387.5 | 457060 B | 471.20 |
| | | | | | | | | | |
| Sep______ | Asset | 50000 | 35.112 ms | 1.00 | 29 | 831.1 | 702.2 | 14133938 B | 1.00 |
| Sep_MT___ | Asset | 50000 | 22.928 ms | 0.66 | 29 | 1272.8 | 458.6 | 14327404 B | 1.01 |
| Sylvan___ | Asset | 50000 | 39.254 ms | 1.11 | 29 | 743.4 | 785.1 | 14297119 B | 1.01 |
| ReadLine_ | Asset | 50000 | 106.311 ms | 3.04 | 29 | 274.5 | 2126.2 | 104584704 B | 7.40 |
| CsvHelper | Asset | 50000 | 83.844 ms | 2.39 | 29 | 348.0 | 1676.9 | 14306424 B | 1.01 |
| | | | | | | | | | |
| Sep______ | Asset | 1000000 | 658.125 ms | 1.00 | 583 | 887.0 | 658.1 | 273067600 B | 1.00 |
| Sep_MT___ | Asset | 1000000 | 305.092 ms | 0.46 | 583 | 1913.5 | 305.1 | 274288576 B | 1.00 |
| Sylvan___ | Asset | 1000000 | 794.169 ms | 1.21 | 583 | 735.1 | 794.2 | 273227616 B | 1.00 |
| ReadLine_ | Asset | 1000000 | 1,842.849 ms | 2.80 | 583 | 316.8 | 1842.8 | 2087764672 B | 7.65 |
| CsvHelper | Asset | 1000000 | 1,740.008 ms | 2.64 | 583 | 335.5 | 1740.0 | 273238000 B | 1.00 |
50 changes: 28 additions & 22 deletions benchmarks/AMD.Ryzen.9.5950X/PackageAssetsBenchQuotes.md
Original file line number Diff line number Diff line change
@@ -1,33 +1,39 @@
```

BenchmarkDotNet v0.13.10, Windows 10 (10.0.19044.3086/21H2/November2021Update)
BenchmarkDotNet v0.13.11, Windows 10 (10.0.19044.3086/21H2/November2021Update)
AMD Ryzen 9 5950X, 1 CPU, 32 logical and 16 physical cores
.NET SDK 8.0.100
[Host] : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX2
Job-TUSNIO : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX2
Job-PQAZLF : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX2

Job=Job-TUSNIO Runtime=.NET 8.0 Toolchain=net80
Job=Job-PQAZLF Runtime=.NET 8.0 Toolchain=net80
InvocationCount=Default IterationTime=350.0000 ms MaxIterationCount=15
MinIterationCount=5 WarmupCount=6 Quotes=True
Reader=String

```
| Method | Scope | Rows | Mean | Ratio | MB | MB/s | ns/row | Allocated | Alloc Ratio |
|------------- |------ |------ |-----------:|------:|---:|-------:|-------:|------------:|------------:|
| Sep______ | Row | 50000 | 7.453 ms | 1.00 | 33 | 4478.2 | 149.1 | 944 B | 1.00 |
| Sep_Unescape | Row | 50000 | 7.381 ms | 0.99 | 33 | 4522.2 | 147.6 | 944 B | 1.00 |
| Sylvan___ | Row | 50000 | 18.035 ms | 2.40 | 33 | 1850.7 | 360.7 | 7390 B | 7.83 |
| ReadLine_ | Row | 50000 | 14.689 ms | 1.97 | 33 | 2272.3 | 293.8 | 111389433 B | 117,997.28 |
| CsvHelper | Row | 50000 | 52.212 ms | 7.01 | 33 | 639.3 | 1044.2 | 21081 B | 22.33 |
| | | | | | | | | | |
| Sep______ | Cols | 50000 | 7.780 ms | 1.00 | 33 | 4290.1 | 155.6 | 946 B | 1.00 |
| Sep_Unescape | Cols | 50000 | 8.337 ms | 1.07 | 33 | 4003.6 | 166.7 | 946 B | 1.00 |
| Sylvan___ | Cols | 50000 | 20.329 ms | 2.60 | 33 | 1641.9 | 406.6 | 7411 B | 7.83 |
| ReadLine_ | Cols | 50000 | 15.540 ms | 1.99 | 33 | 2147.8 | 310.8 | 111389433 B | 117,747.82 |
| CsvHelper | Cols | 50000 | 83.149 ms | 10.64 | 33 | 401.4 | 1663.0 | 457060 B | 483.15 |
| | | | | | | | | | |
| Sep______ | Asset | 50000 | 35.903 ms | 1.00 | 33 | 929.6 | 718.1 | 14139438 B | 1.00 |
| Sep_Unescape | Asset | 50000 | 34.316 ms | 0.93 | 33 | 972.7 | 686.3 | 14130926 B | 1.00 |
| Sylvan___ | Asset | 50000 | 51.465 ms | 1.43 | 33 | 648.5 | 1029.3 | 14296613 B | 1.01 |
| ReadLine_ | Asset | 50000 | 117.914 ms | 3.29 | 33 | 283.1 | 2358.3 | 125239144 B | 8.86 |
| CsvHelper | Asset | 50000 | 96.670 ms | 2.57 | 33 | 345.3 | 1933.4 | 14307304 B | 1.01 |
| Method | Scope | Rows | Mean | Ratio | MB | MB/s | ns/row | Allocated | Alloc Ratio |
|------------- |------ |-------- |-------------:|------:|----:|-------:|-------:|-------------:|------------:|
| Sep______ | Row | 50000 | 6.867 ms | 1.00 | 33 | 4860.2 | 137.3 | 983 B | 1.00 |
| Sep_Unescape | Row | 50000 | 6.749 ms | 0.98 | 33 | 4945.8 | 135.0 | 981 B | 1.00 |
| Sylvan___ | Row | 50000 | 17.676 ms | 2.57 | 33 | 1888.3 | 353.5 | 7406 B | 7.53 |
| ReadLine_ | Row | 50000 | 15.002 ms | 2.19 | 33 | 2224.9 | 300.0 | 111389433 B | 113,315.80 |
| CsvHelper | Row | 50000 | 52.772 ms | 7.68 | 33 | 632.5 | 1055.4 | 21081 B | 21.45 |
| | | | | | | | | | |
| Sep______ | Cols | 50000 | 7.753 ms | 1.00 | 33 | 4305.2 | 155.1 | 986 B | 1.00 |
| Sep_Unescape | Cols | 50000 | 8.778 ms | 1.13 | 33 | 3802.3 | 175.6 | 987 B | 1.00 |
| Sylvan___ | Cols | 50000 | 20.304 ms | 2.63 | 33 | 1643.9 | 406.1 | 7411 B | 7.52 |
| ReadLine_ | Cols | 50000 | 15.319 ms | 1.99 | 33 | 2178.8 | 306.4 | 111389433 B | 112,971.03 |
| CsvHelper | Cols | 50000 | 84.141 ms | 10.86 | 33 | 396.7 | 1682.8 | 457060 B | 463.55 |
| | | | | | | | | | |
| Sep______ | Asset | 50000 | 39.890 ms | 1.00 | 33 | 836.7 | 797.8 | 14134252 B | 1.00 |
| Sep_MT___ | Asset | 50000 | 24.580 ms | 0.61 | 33 | 1357.9 | 491.6 | 14320647 B | 1.01 |
| Sylvan___ | Asset | 50000 | 51.352 ms | 1.29 | 33 | 650.0 | 1027.0 | 14296619 B | 1.01 |
| ReadLine_ | Asset | 50000 | 118.667 ms | 2.91 | 33 | 281.3 | 2373.3 | 125239088 B | 8.86 |
| CsvHelper | Asset | 50000 | 96.789 ms | 2.42 | 33 | 344.8 | 1935.8 | 14307820 B | 1.01 |
| | | | | | | | | | |
| Sep______ | Asset | 1000000 | 752.314 ms | 1.00 | 667 | 887.5 | 752.3 | 273070272 B | 1.00 |
| Sep_MT___ | Asset | 1000000 | 403.912 ms | 0.54 | 667 | 1653.1 | 403.9 | 274538768 B | 1.01 |
| Sylvan___ | Asset | 1000000 | 1,095.609 ms | 1.46 | 667 | 609.4 | 1095.6 | 273233808 B | 1.00 |
| ReadLine_ | Asset | 1000000 | 2,550.179 ms | 3.38 | 667 | 261.8 | 2550.2 | 2500931696 B | 9.16 |
| CsvHelper | Asset | 1000000 | 2,031.309 ms | 2.70 | 667 | 328.7 | 2031.3 | 273236600 B | 1.00 |
2 changes: 1 addition & 1 deletion benchmarks/AMD.Ryzen.9.5950X/Versions.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
Sep 0.3.0.0, Sylvan 1.3.5.0, CsvHelper 30.0.1.0
Sep 0.4.0.0, Sylvan 1.3.5.0, CsvHelper 30.0.1.0
4 changes: 2 additions & 2 deletions src/Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -38,9 +38,9 @@
<!--<DefineConstants Condition="'$(Configuration)'=='Release'">$(DefineConstants);SEPBENCHSEPONLY</DefineConstants>-->
<DefineConstants Condition="'$(Configuration)'=='Release'">$(DefineConstants);SEPBENCHSLOWONES</DefineConstants>
<DefineConstants Condition="'$(Configuration)'=='Debug'">$(DefineConstants);SEPASSERT</DefineConstants>
<DefineConstants Condition="'$(Configuration)'=='Debug'">$(DefineConstants);SEPTRACE</DefineConstants>
<!--<DefineConstants Condition="'$(Configuration)'=='Debug'">$(DefineConstants);SEPTRACE</DefineConstants>-->
<DefineConstants Condition="'$(Configuration)'=='Debug'">$(DefineConstants);SEPREADERASSERT</DefineConstants>
<DefineConstants Condition="'$(Configuration)'=='Debug'">$(DefineConstants);SEPREADERTRACE</DefineConstants>
<!--<DefineConstants Condition="'$(Configuration)'=='Debug'">$(DefineConstants);SEPREADERTRACE</DefineConstants>-->
</PropertyGroup>

<ItemGroup>
Expand Down
2 changes: 1 addition & 1 deletion src/Sep.Benchmarks/Program.cs
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,6 @@
static long BytesFromCharsLength(IReadOnlyList<ParameterInstance> parameters)
{
return parameters.Where(p => p.Name == nameof(SepParserBench.Filler))
.Select(p => ((SepParserBench.FillerSpec)p.Value).Text.Length * sizeof(char))
.Select(p => ((SepParserBench.FillerSpec)p.Value).TotalLength * sizeof(char))
.Single()!;
}
6 changes: 3 additions & 3 deletions src/Sep.Benchmarks/SepParseSeparatorsMaskBench.cs
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ public readonly record struct MaskSpec(uint Mask)
MaskSpec _mask;
nuint _maskValue;
readonly int _dataIndex = 17;
readonly int* _colEnds = (int*)NativeMemory.Alloc((nuint)Unsafe.SizeOf<nuint>() * 8, sizeof(int));
readonly int* _colEndsOrColInfos = (int*)NativeMemory.Alloc((nuint)Unsafe.SizeOf<nuint>() * 8, sizeof(int));

public SepParseSeparatorsMaskBench()
{
Expand All @@ -44,13 +44,13 @@ public MaskSpec Mask
public unsafe ref int DummyForWarmup()
{
return ref SepParseMask.ParseSeparatorsMask(
_maskValue, _dataIndex, ref *_colEnds);
_maskValue, _dataIndex, ref *_colEndsOrColInfos);
}

[Benchmark(Baseline = true)]
public unsafe ref int ParseSeparatorsMask()
{
return ref SepParseMask.ParseSeparatorsMask(
_maskValue, _dataIndex, ref *_colEnds);
_maskValue, _dataIndex, ref *_colEndsOrColInfos);
}
}
Loading