Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reorganize repo to more standard format #5

Merged
merged 3 commits into from
Mar 25, 2024
Merged

reorganize repo to more standard format #5

merged 3 commits into from
Mar 25, 2024

Conversation

stevengj
Copy link
Member

Move the tests into test/runtests.jl and set up to run via @test in package testing, updated Project.toml with dependency info and removed obsolete dependency on StaticArrays, and set up github actions.

@stevengj
Copy link
Member Author

@rickbeeloo, any idea why the tests would be failing on 64-bit Windows (only)?

 Reading lines: Test Failed at D:\a\ViewReader.jl\ViewReader.jl\test\runtests.jl:115
  Expression: normalRead() == viewRead()
   Evaluated: 10000 == 0

@rickbeeloo
Copy link
Collaborator

@stevengj, you refer to the actual OS being 64 bit or the datatype you changed in 35818a7?

@rickbeeloo
Copy link
Collaborator

@stevengj, I see now, the first thing that comes to mind is that just Windows uses \r\n as newline delimiter. Since it still ends on \n I'm not sure why this directly fails. Let me switch to Windows and run it - will report back today.

@rickbeeloo
Copy link
Collaborator

@stevengj okay yeah it's the \r since the lines (in windows) end on \r\n. So the buffer slice has a trailing \r (which we don't see in StringViews) but when we then check == "Test" we miss them cause we actually have test\r in de underlaying UInt8 array.

Probably the clean fix would be checking the OS then adjusting some code to look for \n preceded by \r in windows. A bit more hassle as it would require checking two chars instead of one - which then also should be considered when flipping the buffer.

The ugly fix would just be stripping \r from lines before the StringView returns here and here.

I tested this by returning StringView(os_check(view(reader.arr, r)) ), state

function os_check(line_slice::SubArray{UInt8, 1})
     # Actually wanted to check the OS here but could not find the built-in function
    return (line_slice[end] == UInt8('\r')) ? view(line_slice, 1:length(line_slice) - 1) : line_slice
end

Perhaps you have a better idea :)

@stevengj
Copy link
Member Author

stevengj commented Mar 25, 2024

eachlineV really needs to treat \r as part of the line ending when it is followed by \n, in order to mimic the Base.eachline behavior.

So, basically, the iterate(::BufferedReader, state) function needs to be fixed in order to

  1. keep track (in the state) whether the previous chunk ended with \r, and if so don't write the \r until the next chunk is processed
  2. After find_newline, it should check whether the previous character is \r, and if so chop that off too. If the \n was at the beginning of a chunk, it will need to check the state to see if the previous chunk ended with \r

(Unfortunately, the logic is complicated by the need to handle the rare case where \r\n falls across a chunk boundary.)

@stevengj
Copy link
Member Author

I guess it's simpler than that because you assume that the buffers are large enough that a single line always fits within a single chunk.

@stevengj stevengj merged commit 9b4b385 into master Mar 25, 2024
9 checks passed
@stevengj stevengj deleted the reorg branch March 25, 2024 21:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants