Configure Builds with Lua

[Lethja]

This pull request has been added in a draft state for feedback purposes...

This pull request adds a Lua script that can determine and address the differences between different types of build platforms. The script is inspired by GNU Autotools but thanks to Lua is completely platform agnostic.

The script aims to address in the current build system like:

• compiling on linux #72
• The need to run/emulating pre-compiled binaries to configure successfully
• Makefiles being generated with the wrong platform specific commands (rm vs del)
• Certain compiler flags not being available on older versions of the compiler (-fdiagnostics-color=never)
• Single source (configur.lua for everyone instead of configur.bat vs configur.sh)

Note: While FreeDOS has shipped Lua for some time on their bonus CD this version uses a DOS extender and could freeze if another DOS extender is started at the same time. https://github.com/Lethja/lua-watcom allows Lua to be built for real mode to avoid this problem. Luas official makefile is excellent at building non-DOS 32-bit targets.

[gvanem]

I do not see that lua configur.lua [ -h | --help ] is handled. Users would like to know what this script does before using it.

[Lethja]

I do not see that lua configur.lua [ -h | --help ] is handled. Users would like to know what this script does before using it.

Excellent point. b5eff3f has added this

$ lua configur.lua
Specify a makefile to generate, options are:
	cc	- Handle everything yourself with CC,CFLAGS,LD enviroment variables
	clang	- A GCC compatible alternative compiler
	djggp	- A port of GCC for 80386+ DOS systems
	gcc	- GNU Compiler Collection intended for Posix systems
	mingw	- A port of GCC for Win32 based systems
	watcom	- A C/C++ toolchain that can build 16-bit DOS programs

Use '/?' or '-h' to get full help

$ lua configur.lua /?
configur.lua [TARGET] [OPTIONS]
Configures and generates Watt-32 makefiles regardless of host enviroment using Lua
A platform agnostic alternative to `src/configur.sh` and `SRC\CONFIGUR.BAT`

[TARGET]:
	cc	- Handle everything yourself with CC,CFLAGS,LD enviroment variables
	clang	- A GCC compatible alternative compiler
	djggp	- A port of GCC for 80386+ DOS systems
	gcc	- GNU Compiler Collection intended for Posix systems
	mingw	- A port of GCC for Win32 based systems
	watcom	- A C/C++ toolchain that can build 16-bit DOS programs

[OPTIONS]:
	-h, --help, /? - Show this help
	-s, /s         - Skip compiler checks

[gvanem]

Does it need normal stock Lua 5.x or does it work with LuaJIT too?

[Lethja]

Does it need normal stock Lua 5.x or does it work with LuaJIT too?

Up until reading this comment this script has been tested using the official Lua 5.4.6 on x86_64 Linux and DOS Real Mode (via this build).

I just built and tested the script running under LuaJIT, initial results: looks like it works without issue

$ ~/Git/Repositories/luajit/src/luajit configur.lua gcc
Determining operating system family... Unix
Checking 'mkdir' works... Yes
Checking 'rm' works... Yes
Checking 'rm -R' works... Yes
Checking 'inc' contains required files... Yes
Checking 'src' contains required files... Yes
Checking 'CC' enviroment variable... None
Guessing compiler... GNU Compiler Collection or compatible
Checking 'gcc' is available... Yes
Checking if 'gcc' understands '-fdiagnostics-color=never'... Yes
Checking size of 'int' C type in bits... 32
Checking size of 'long' C type in bits... 64
Checking if Lua configuration script is finished and ready to use... No

[Lethja]

I'm not 100% on what cflagsbf.c output is suppose to look like. It'll take time to fully decipher and replicate all the moving parts of makefile.all and the util binaries.

For the time being I've pushed ef743bf which will generate a rough mockup (probably not working) of a Watcom makefile when running lua configur.lua wcc -s. Watcom doesn't need to be installed when -s is used

[jwt27]

This is a neat idea, I'm all for it. However, I don't see how it fully solves:

• compiling on linux #72
• The need to run/emulating pre-compiled binaries to configure successfully

Specifically, generating the error code macros and sys_errlist[]. This needs access to the target libc's <errno.h> to put them in the right order.

The solution, I think, is to preprocess this with the target compiler, and then make a native (build-time) binary that generates the headers. This implies that the configure script must distinguish between native and target toolchains, eg via autoconf-style variables such as CC_FOR_BUILD/CC_FOR_HOST. It gets complicated quick.

edit: Also, if your intention is to eventually merge #89 into this, the following may be worth considering beforehand:

• Ability to do out-of-tree builds (like autotools)
• Separate targets for each of Watcom's 16-bit memory models

[gvanem]

I'm not 100% on what cflagsbf.c output is suppose to look like.

It's cflagsbf.h.

E.g. build\clang\64bit\release\cflags.h is:

const char *w32_cflags = "-nologo -Zi -Zo -GS- -I. -I../inc -DWATT32_BUILD -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_OBSOLETE_NO_WARNINGS -D_WIN32_WINNT=0x0601 -D_WINSOCK_DEPRECATED_NO_WARNINGS -DBUILD_WINDOWS -Wno-#pragma-messages -MD -Ox -fms-compatibility";
const char *w32_cc     = ""f:\ProgramFiler\LLVM-17.0\win64/bin/clang-cl.exe"";

Hence those 3 \ (= 0x5C) above must be hex-ified into build\clang\64bit\release\cflagsbf.h.
A hex-view of build\clang\64bit\release\cflags.h:

00000000  63 6F 6E 73 74 20 63 68 61 72 20 2A 77 33 32 5F    const char *w32_
00000010  63 66 6C 61 67 73 20 3D 20 22 2D 6E 6F 6C 6F 67    cflags = "-nolog
00000020  6F 20 2D 5A 69 20 2D 5A 6F 20 2D 47 53 2D 20 2D    o -Zi -Zo -GS- -
00000030  49 2E 20 2D 49 2E 2E 2F 69 6E 63 20 2D 44 57 41    I. -I../inc -DWA
00000040  54 54 33 32 5F 42 55 49 4C 44 20 2D 44 5F 43 52    TT32_BUILD -D_CR
00000050  54 5F 53 45 43 55 52 45 5F 4E 4F 5F 57 41 52 4E    T_SECURE_NO_WARN
00000060  49 4E 47 53 20 2D 44 5F 43 52 54 5F 4E 4F 4E 53    INGS -D_CRT_NONS
00000070  54 44 43 5F 4E 4F 5F 57 41 52 4E 49 4E 47 53 20    TDC_NO_WARNINGS 
00000080  2D 44 5F 43 52 54 5F 4F 42 53 4F 4C 45 54 45 5F    -D_CRT_OBSOLETE_
00000090  4E 4F 5F 57 41 52 4E 49 4E 47 53 20 2D 44 5F 57    NO_WARNINGS -D_W
000000A0  49 4E 33 32 5F 57 49 4E 4E 54 3D 30 78 30 36 30    IN32_WINNT=0x060
000000B0  31 20 2D 44 5F 57 49 4E 53 4F 43 4B 5F 44 45 50    1 -D_WINSOCK_DEP
000000C0  52 45 43 41 54 45 44 5F 4E 4F 5F 57 41 52 4E 49    RECATED_NO_WARNI
000000D0  4E 47 53 20 2D 44 42 55 49 4C 44 5F 57 49 4E 44    NGS -DBUILD_WIND
000000E0  4F 57 53 20 2D 57 6E 6F 2D 23 70 72 61 67 6D 61    OWS -Wno-#pragma
000000F0  2D 6D 65 73 73 61 67 65 73 20 2D 4D 44 20 2D 4F    -messages -MD -O
00000100  78 20 2D 66 6D 73 2D 63 6F 6D 70 61 74 69 62 69    x -fms-compatibi
00000110  6C 69 74 79 22 3B 0A 63 6F 6E 73 74 20 63 68 61    lity";.const cha
00000120  72 20 2A 77 33 32 5F 63 63 20 20 20 20 20 3D 20    r *w32_cc     = 
00000130  22 22 66 3A 5C 50 72 6F 67 72 61 6D 46 69 6C 65    ""f:\ProgramFile
00000140  72 5C 4C 4C 56 4D 2D 31 37 2E 30 5C 77 69 6E 36    r\LLVM-17.0\win6
00000150  34 2F 62 69 6E 2F 63 6C 61 6E 67 2D 63 6C 2E 65    4/bin/clang-cl.e
00000160  78 65 22 22 3B 0A                                  xe"";.          ```

Otherwise the C pre-processor would complain. See version.c for it's purpose.
Was that what you was uncertain of?

[Lethja]

Specifically, generating the error code macros and sys_errlist[]. This needs access to the target libc's <errno.h> to put them in the right order.

The solution, I think, is to preprocess this with the target compiler, and then make a native (build-time) binary that generates the headers. This implies that the configure script must distinguish between native and target toolchains, eg via autoconf-style variables such as CC_FOR_BUILD/CC_FOR_HOST. It gets complicated quick.

Indeed! I had not thought of that, that's a tough problem to solve. Worst still CheckCompilerIntSize() and CheckCompilerLongSize() depend on target system execution to give the correct result. Unless I missed something a pre-processor check isn't good enough for CheckCompilerIntSize() since wcc pre-processor lies about how big int is and others might to.

The immediate workaround to these cross-compiling problems would be:

• Use the /s or -s flag (which will skip tests and fill in the values with guestimates)
• Setup Lua and a native C toolchain in a emulator/VM enviroment
  • Everywhere to DOS: DOSBox-X
  • Everywhere to everywhere: QEMU, Virtualbox
  • Linux to DOS: dosemu
  • Linux to Windows: Wine (wine cmd)
  • Windows to Linux: WSL2

It's cflagsbf.h.
Was that what you was uncertain of?

Yes, I assumed it was more complex then a direct string conversion. It should be trivial to write a function in the script that converts cflags.h into it's hex array counterpart for cflagsbf.h.

[Lethja]

As of 5debd9c configur.lua can generate makefiles which will build the library for DOS successfully.

Tested successfully

Makefile	Built from	Built for	Comment
DJGPP.MAK	DOSX	DOSX	NASM required
DJGPP.MAK	Linux	DOSX	DOSEMU & NASM required
WATCOM_L.MAK	DOSX	DOS
WATCOM_F.MAK	DOSX	DOSX

Untested but assumed to work

Makefile	Comment
WATCOM_3.MAK	Should work, identical to WATCOM_F.MAK besides CFLAGS
WATCOM_S.MAK	Should work, identical to WATCOM_L.MAK besides CFLAGS
WATCOM_W.MAK	Watcom tools specific WIN32 untested

Now that there's something worth testing I will re-open this PR