Consider working around Git on Windows' use of CRLF vs. bash scripts

[bytetheripper]

When I try to run ./configure on the /john/src file in Cygwin on my Windows 10 device, I get the following error:

$ ./configure
./configure: line 16: $'\r': command not found
./configure: line 31: syntax error near unexpected token newline' '/configure: line 31: ;;

When I looked at the /src file I found this disclaimer starting on line 17211:

On cygwin, bash can eat \r inside `` if the user requested igncr.

But we know of no other shell where ac_cr would be empty at this

point, so we can use a bashism as a fallback.

Is there a workaround I can use?

[claudioandre-br]

I ran a build myself on May 11th [1] (on Windows 2016 using Cygwin 3.2.0) and it worked fine. Although this issue is worded as a bug report, not as a question, I'm afraid you're mistaken.

• Could you make your point more clear?
• if you need support as an end-user, ask for help on the mailing list john-users at lists.openwall.com (but first see the archives at http://www.openwall.com/lists/john-users/ and make sure you set a descriptive message Subject when you post).

[1]
https://dev.azure.com/claudioandre-br/JohnTheRipper/_build/results?buildId=320&view=logs&s=52e5c710-3f09-5ad6-0c38-decbe96b0346

[magnumripper]

https://www.openwall.com/lists/john-users/2021/05/26/1

[solardiz]

Searching for mentions of related problems seen by other projects finds e.g.:

reconquest/import.bash#2
https://community.perforce.com/s/article/2973

Reading and summarizing these and the like, it appears that Git on Windows defaults to using CRLF for text files ("On windows systems core.autocrlf is normally set to "true" globally.")

Some workarounds were mentioned in the thread on john-users. However, maybe as a project we should "add .gitattributes file [...] and force git to handle shell scripts eol=lf"? Re-opening for us to consider this.

This leaves me wondering why we didn't get this issue reported to us many times before, and why it didn't occur in @claudioandre-br's builds. Any ideas? Or maybe "is normally set to" doesn't mean it's the "default", but is something a Git user does, and most of those advanced users also worked around the consequences without needing our help? Anyway, we could want to provide the above workaround in our project.

[claudioandre-br]

add .gitattributes file [...] and force git to handle shell scripts eol=lf"

This is a nice "good first issue". If @HustleSec can test it, he will be able to check if it works and create a PR. Simple, straightforward.

[magnumripper]

We already have a .gitattributes file - it was added mostly for export-ignore but it also contains some sample eol features commented out, and a few active ones (eg. charset files marked as binary). I guess we need at least these:

diff --git a/.gitattributes b/.gitattributes
index da1ea8814..0fe5f07cc 100644
--- a/.gitattributes
+++ b/.gitattributes
@@ -9,6 +9,18 @@
 # Declare files that will always have CRLF line endings on checkout.
 run/oui.txt text eol=crlf
 
+# Shell scripts on the other hand must not have CRLF
+*.sh text eol=lf
+run/john.bash_completion text eol=lf
+run/mailer text eol=lf
+run/makechr text eol=lf
+src/compile text eol=lf
+src/config.guess text eol=lf
+src/config.status text eol=lf
+src/config.sub text eol=lf
+src/configure text eol=lf
+src/install-sh text eol=lf
+
 # Denote all files that are truly binary and should not be modified.
 run/*.chr binary
 src/john.com binary

This still leaves the first (sample) lines commented out, I'm not sure we should touch them and if we do, we need to test it with many possible settings of git environment (unless someone with knowledge can say for sure they will be fine). I mean these lines:

# Set the default behavior, in case people don't have core.autocrlf set.
#* text=auto

# Explicitly declare text files you want to always be normalized and converted
# to native line endings on checkout.
#*.c text
#*.h text