Merge pull request #1788 from Kazzz-S/0.29.4-mac1

Modified the macOS build system so that the debug build uses ASAN (Adress SANitizer) for clang++
KLayout · Jul 14, 2024 · a6c0718 · a6c0718
2 parents 4383802 + ae13152
commit a6c0718
Show file tree

Hide file tree

Showing 8 changed files with 267 additions and 143 deletions.
diff --git a/macbuild/ReadMe.md b/macbuild/ReadMe.md
@@ -1,6 +1,6 @@
-Relevant KLayout version: 0.29.2<br>
+Relevant KLayout version: 0.29.4<br>
 Author: Kazzz-S<br>
-Last modified: 2024-06-09<br>
+Last modified: 2024-07-10<br>
 
 # 1. Introduction
 This directory **`macbuild`** contains various files required for building KLayout (http://www.klayout.de/) version 0.29.0 or later for different 64-bit macOS, including:
@@ -61,7 +61,7 @@ You need to have the followings:
 
 # 5. Command-line options of **`build4mac.py`**
 
-**`build4mac.py`** is the top level Python script for for building KLayout for a macOS.
+**`build4mac.py`** is the top level Python script for building KLayout for a macOS.
 The operating system type is detected automatically.
 
 ```
@@ -102,7 +102,7 @@ $ [python] ./build4mac.py
    [-u|--noqtuitools]   : don't include uitools in Qt binding                            | disabled
    [-g|--nolibgit2]     : don't include libgit2 for Git package support                  | disabled
    [-m|--make <option>] : option passed to 'make'                                        | '--jobs=4'
-   [-d|--debug]         : enable debug mode build                                        | disabled
+   [-d|--debug]         : enable debug mode build; AddressSanitizer (ASAN) is linked     | disabled
    [-c|--checkcom]      : check command-line and exit without building                   | disabled
    [-y|--deploy]        : deploy executables and dylibs, including Qt's Frameworks       | disabled
    [-Y|--DEPLOY]        : deploy executables and dylibs for those who built KLayout      | disabled
@@ -189,10 +189,8 @@ $ ./build4mac.py -q qt5macports -r mp33 -p mp311 -Y
 4. Copy/move the generated application bundle **`klayout.app`** to your **`/Applications`** directory for installation.
 
 ### 6C. Fully Homebrew-flavored build with Homebrew Ruby 3.3 and Homebrew Python 3.11
-> [!CAUTION]
-> To build KLayout >= 0.29.0, you need "Qt6" to address [the compilation issue](https://github.com/KLayout/klayout/issues/1599).<br>
-> However, the current KLayout is not compliant with the latest Qt6.7.0, so you will get another compilation error.<br>
-> Therefore, this section will be disabled for the time being.
+> [!IMPORTANT]
+> To build KLayout >= 0.29.0, you need "Qt6" >= 6.7.0 to address [the compilation issue](https://github.com/KLayout/klayout/issues/1599).<br>
 
 0. Install Homebrew, then install Qt6, Ruby 3.3, Python 3.11, and libgit2 by
 ```
@@ -225,10 +223,8 @@ $ ./build4mac.py -q qt6brew -r hb33 -p hb311 -Y
 4. Copy/move the generated application bundle **`klayout.app`** to your **`/Applications`** directory for installation.
 
 ### 6D. Partially Homebrew-flavored build with System Ruby and Homebrew Python 3.11
-> [!CAUTION]
-> To build KLayout >= 0.29.0, you need "Qt6" to address [the compilation issue](https://github.com/KLayout/klayout/issues/1599).<br>
-> However, the current KLayout is not compliant with the latest Qt6.7.0, so you will get another compilation error.<br>
-> Therefore, this section will be disabled for the time being.
+> [!IMPORTANT]
+> To build KLayout >= 0.29.0, you need "Qt6" >= 6.7.0 to address [the compilation issue](https://github.com/KLayout/klayout/issues/1599).<br>
 
 0. Install Homebrew, then install Qt6, Python 3.11, and libgit2 by
 ```
@@ -263,7 +259,7 @@ $ ./build4mac.py -q qt6brew -r sys -p hb311 -y
 
 ### 6E. Heterogeneous combination of MacPorts Qt5, System Ruby, and Homebrew Python 3.11
 > [!IMPORTANT]
-> This is a practical solution for building an HW*.dmg package.
+> This is another practical solution for building a popular HW*.dmg package.
 
 0. Install MacPorts, then install Qt5 and libgit2 by
 ```
@@ -347,7 +343,37 @@ A sample content (`*.app.Bash`) of the script bundle can be found in `Resources/
 
 ----
 
-# 7. Making a DMG installer
+# 7. QA Tests
+You can optionally conduct QA tests using the `ut_runner` executable.<br>
+[This forum post](https://www.klayout.de/forum/discussion/comment/11012/#Comment_11012) provides information on the unit tests, mainly for Linux.
+In the macOS environment, the QA test working directory is `[ST|LW|HB]-build_directory.macQAT`, where you will find `macQAT.py`, a wrapper script for the `ut_runner` executable.<br>
+Some required environment variables including `TESTSRC`, `TESTTMP`, and `DYLD_LIBRARY_PATH` are set by `macQAT.py`.<br>
+
+1. Change directory to `[ST|LW|HB]-build_directory.macQAT`
+```
+cd [ST|LW|HB]-build_directory.macQAT
+```
+
+2. To print usage of `ut_runner`, run `macQAT.py` with '-u'
+```
+./macQAT.py -u
+```
+
+3. To start the KLayout main GUI window, run `macQAT.py` with '-k'
+```
+./macQAT.py -k
+```
+
+4. To normally run `ut_runner`, invoke `macQAT.py` with '-r'
+```
+./macQAT.py -r
+```
+
+If required, you can use the `-x <test>` option to skip some erroneous tests.
+
+----
+
+# 8. Making a DMG installer
 You can make a DMG installer using another Python script **`makeDMG4mac.py`**.<br>
 This script requires a directory generated by **`build4mac.py`** with the [-y|-Y] option (refer to 6B through 6E).
 

diff --git a/macbuild/build4mac.py b/macbuild/build4mac.py
@@ -80,7 +80,7 @@ def GenerateUsage(platform):
     usage += "   [-u|--noqtuitools]   : don't include uitools in Qt binding                            | disabled\n"
     usage += "   [-g|--nolibgit2]     : don't include libgit2 for Git package support                  | disabled\n"
     usage += "   [-m|--make <option>] : option passed to 'make'                                        | '--jobs=4'\n"
-    usage += "   [-d|--debug]         : enable debug mode build                                        | disabled\n"
+    usage += "   [-d|--debug]         : enable debug mode build; AddressSanitizer (ASAN) is linked     | disabled\n"
     usage += "   [-c|--checkcom]      : check command-line and exit without building                   | disabled\n"
     usage += "   [-y|--deploy]        : deploy executables and dylibs, including Qt's Frameworks       | disabled\n"
     usage += "   [-Y|--DEPLOY]        : deploy executables and dylibs for those who built KLayout      | disabled\n"
@@ -1007,6 +1007,7 @@ def Build_pymod_wheel(parameters):
 # @return 0 on success; non-zero (1), otherwise
 #------------------------------------------------------------------------------
 def Run_Build_Command(config, parameters):
+    DebugMode = config['DebugMode']
     ModuleQt  = config['ModuleQt']
     NoLibGit2 = config['NoLibGit2']
     ToolDebug = config['ToolDebug']
@@ -1015,8 +1016,22 @@ def Run_Build_Command(config, parameters):
     else:
         jump2pymod_wheel = True
 
+    #-----------------------------------------------------------------
+    # [1] Use the AddressSanitizer (ASan) in the debug build.
+    #     This environment variable is tested in ../src/klayout.pri.
+    #-----------------------------------------------------------------
+    try:
+        useAsan = os.environ['MAC_USE_ASAN']
+    except KeyError:
+        pass
+    else:
+        del os.environ['MAC_USE_ASAN']
+
+    if DebugMode:
+        os.environ['MAC_USE_ASAN'] = "1"
+
     #-----------------------------------------------------
-    # [1] Set two environment variables to use libgit2
+    # [2] Set two environment variables to use libgit2
     #-----------------------------------------------------
     if not NoLibGit2:
         # Using MacPorts
@@ -1048,7 +1063,7 @@ def Run_Build_Command(config, parameters):
 
     if not jump2pymod_wheel:
         #-----------------------------------------------------
-        # [2] Set parameters passed to the main Bash script
+        # [3] Set parameters passed to the main Bash script
         #-----------------------------------------------------
         cmd_args = ""
 
@@ -1106,7 +1121,7 @@ def Run_Build_Command(config, parameters):
             cmd_args += " \\\n  -nopython"
 
         #-----------------------------------------------------
-        # [3] Make the consolidated command line
+        # [4] Make the consolidated command line
         #-----------------------------------------------------
         command  = "time"
         command += " \\\n  %s" % parameters['build_cmd']
@@ -1119,7 +1134,7 @@ def Run_Build_Command(config, parameters):
             sys.exit(0)
 
         #-----------------------------------------------------
-        # [4] Invoke the main Bash script; takes time:-)
+        # [5] Invoke the main Bash script; takes time:-)
         #-----------------------------------------------------
         myscript = os.path.basename(__file__)
         ret = subprocess.call( command, shell=True )
@@ -1138,7 +1153,7 @@ def Run_Build_Command(config, parameters):
         print( "", file=sys.stderr )
 
         #------------------------------------------------------------------------
-        # [5] Prepare "*.macQAT/" directory for the QATest.
+        # [6] Prepare "*.macQAT/" directory for the QATest.
         #     Binaries under "*.macQAT/" such as *.dylib will be touched later.
         #------------------------------------------------------------------------
         print( "### Preparing <%s>" % MacBuildDirQAT )
@@ -1177,7 +1192,7 @@ def Run_Build_Command(config, parameters):
         print( "", file=sys.stderr )
 
     #------------------------------------------------------------------------
-    # [6] Build <pymod> for some predetermined environments on demand
+    # [7] Build <pymod> for some predetermined environments on demand
     #------------------------------------------------------------------------
     BuildPymodWhl = parameters['BuildPymodWhl']
     if BuildPymodWhl:

diff --git a/macbuild/build4mac_env.py b/macbuild/build4mac_env.py
@@ -182,7 +182,7 @@
 #  install with 'sudo port install ruby33'
 # [Key Type Name] = 'MP33'
 Ruby33MacPorts  = { 'exe': '/opt/local/bin/ruby3.3',
-                    'inc': '/opt/local/include/ruby-3.3.3',
+                    'inc': '/opt/local/include/ruby-3.3.4',
                     'lib': '/opt/local/lib/libruby.3.3.dylib'
                   }
 

diff --git a/macbuild/macQAT.py b/macbuild/macQAT.py
@@ -107,7 +107,7 @@ def GetTimeStamp():
 def ParseCommandLineArguments():
     global Usage
     global RunnerUsage
-    global StartKLayout 
+    global StartKLayout
     global Run
     global ContinueOnError
     global TestsExcluded
@@ -216,7 +216,9 @@ def ExportEnvVariables():
     MyEnviron[ 'TESTTMP' ] = WorkDir
     if System == "Darwin":
         MyEnviron[ 'DYLD_LIBRARY_PATH' ] = "%s:%s/db_plugins:%s/lay_plugins:%s/pymod" % (ProjectDir, ProjectDir, ProjectDir, ProjectDir)
-        for env in [ 'TESTSRC', 'TESTTMP', 'DYLD_LIBRARY_PATH' ]:
+        MyEnviron[ 'MallocNanoZone' ] = "0"
+        MyEnviron[ 'ASAN_OPTIONS' ] = "ast_unwind_on_malloc=0:verbosity=1:detect_leaks=0:abort_on_error=0:halt_on_error=0:symbolize=1"
+        for env in [ 'TESTSRC', 'TESTTMP', 'DYLD_LIBRARY_PATH', 'MallocNanoZone', 'ASAN_OPTIONS' ]:
             os.environ[env] = MyEnviron[env]
     else:
         MyEnviron[ 'LD_LIBRARY_PATH' ] = "%s:%s/db_plugins:%s/lay_plugins:%s/pymod" % (ProjectDir, ProjectDir, ProjectDir, ProjectDir)
@@ -283,7 +285,7 @@ def Main():
     #-------------------------------------------------------
     if StartKLayout:
         StartKLatyouGUIWindow()
-    
+
     #-------------------------------------------------------
     # [4] Run the unit tester
     #-------------------------------------------------------

diff --git a/macbuild/makeDMG4mac.py b/macbuild/makeDMG4mac.py
@@ -49,6 +49,7 @@ def SetGlobals():
     global DMGSerialNum       # the DMG serial number
     global PackagePrefix      # the package prefix: LW-', 'HW-', or 'EX-'
     global QtIdentification   # Qt identification
+    global BuildType          # build type ['release', 'debug']
     global RubyPythonID       # Ruby- and Python-identification
     global KLVersion          # KLayout's version
     global OccupiedDS         # approx. occupied disc space
@@ -144,6 +145,7 @@ def SetGlobals():
     DMGSerialNum      = 1
     PackagePrefix     = ""
     QtIdentification  = ""
+    BuildType         = ""
     RubyPythonID      = ""
     KLVersion         = GetKLayoutVersionFrom( "./version.sh" )
     OccupiedDS        = -1
@@ -236,6 +238,7 @@ def CheckPkgDirectory():
     global BundleName
     global PackagePrefix
     global QtIdentification
+    global BuildType
     global RubyPythonID
     global BackgroundPNG
     global LatestOSMacPorts
@@ -270,7 +273,8 @@ def CheckPkgDirectory():
     #     * ST-qt6MP.pkg.macos-Monterey-release-RsysPsys
     #     * LW-qt6MP.pkg.macos-Monterey-release-Rmp33Pmp311
     #-----------------------------------------------------------------------------------------------
-    patQRP = u'(ST|LW|HW|EX)([-])([qt5|qt6][0-9A-Za-z]+)([.]pkg[.])([A-Za-z]+[-][A-Za-z]+[-]release[-])([0-9A-Za-z]+)'
+    #                  0      1      2                      3                  4                    5         6        7
+    patQRP = u'(ST|LW|HW|EX)([-])([qt5|qt6][0-9A-Za-z]+)([.]pkg[.])([A-Za-z]+[-][A-Za-z]+[-])(release|debug)([-])([0-9A-Za-z]+)'
     regQRP = re.compile(patQRP)
     if not regQRP.match(PkgDir):
         print( "! Cannot identify (Qt, Ruby, Python) from the package directory name" )
@@ -283,14 +287,21 @@ def CheckPkgDirectory():
         pkgdirComponents = regQRP.match(PkgDir).groups()
         PackagePrefix    = pkgdirComponents[0]
         QtIdentification = pkgdirComponents[2]
-        RubyPythonID     = pkgdirComponents[5]
         if QtIdentification.find('qt5') == 0:
             BackgroundPNG = "KLayoutDMG-BackQt5.png"
         elif QtIdentification.find('qt6') == 0:
             BackgroundPNG = "KLayoutDMG-BackQt6.png"
         else:
             BackgroundPNG = None
             raise Exception( "! neither qt5 nor qt6" )
+        if pkgdirComponents[5] == 'release':
+            BuildType = 'release'
+        elif pkgdirComponents[5] == 'debug':
+            BuildType = 'debug'
+        else:
+            BuildType = None
+            raise Exception( "! neither release nor debug" )
+        RubyPythonID = pkgdirComponents[7]
 
         #-----------------------------------------------------------------------------
         # [3] Check if the "LatestOS" with MacPorts / Homebrew / Anaconda3
@@ -435,6 +446,7 @@ def ParseCommandLineArguments():
     global UnsafePkg
     global PackagePrefix
     global QtIdentification
+    global BuildType
     global RubyPythonID
     global KLVersion
     global OccupiedDS
@@ -532,6 +544,8 @@ def ParseCommandLineArguments():
                         % (PackagePrefix, KLVersion, GenOSName, Platform, DMGSerialNum, QtIdentification, RubyPythonID)
         if Machine == "arm64": # with an Apple Silicon Chip
             TargetDMG = Machine + TargetDMG
+        if BuildType == "debug": # in the case of 'debug' build
+            TargetDMG = "debug-" + TargetDMG
     return
 
 #------------------------------------------------------------------------------
@@ -607,7 +621,7 @@ def MakeTargetDMGFile(msg=""):
     #----------------------------------------------------
     if os.path.exists(WorkDMG):
         os.remove(WorkDMG)
-    dmgsize  = OccupiedDS + 20 # approx. occupied size plus 20[MB]
+    dmgsize  = OccupiedDS + int(0.2*OccupiedDS) # approx. occupied size plus 20[%]
     cmdline  = 'hdiutil create -srcfolder %s -volname %s -fs HFS+ -fsargs "-c c=64,a=16,e=16" '
     cmdline += '-format UDRW -size %dm  %s'
     command  = cmdline % (PkgDir, VolumeDMG, dmgsize, WorkDMG)