MSVS test fixes and minor changes to msvs tool

CHANGES.txt

@@ -13,8 +13,52 @@ NOTE: Python 3.6 support is deprecated and will be dropped in a future release.
 RELEASE  VERSION/DATE TO BE FILLED IN LATER
 
   From Joseph Brill:
-    - Added error handling when creating MS VC detection debug log file specified by
-      SCONS_MSCOMMON_DEBUG
+    - Added error handling when creating MSVC detection debug log file specified by
+      SCONS_MSCOMMON_DEBUG.
+    - MSVS: Added default HOST_ARCH values to sconstruct/sconscript environment for
+      select msvs test scripts to allow tests to run on platforms not recognized by
+      the msvs/msvc tool implementations.  Fixes #4608.
+    - MSVS: Fix early exit after the first msvc version loop execution in select msvs
+      test scripts.  Select msvs test scripts were being invoked for msvc version 8.0
+      only.  Fixes #4609.
+    - MSVS: Additional minor select msvs test script fixes as a result of the msvs
+      tests being invoked for all msvc versions: fix vs version number for vc version
+      14.3, fix expected platform toolset version, add and use a default known
+      project GUID for some select tests, add AdditionalOptions Condition to expected
+      vcx project file.
+    - MSVS: Additional minor changes to the msvs tool as a result of the msvs tests
+      being invoked for all msvc versions: use environment MSVS_PROJECT_GUID when
+      generating project files information, fix the visual studio string for VS2015,
+      add .vcxproj as an expected suffix for assigning the name to the file basename.
+    - MSVS: Add additional msvs tests for multi-project and solution builds.
+    - MSVS: Check for variant directory build of MSVSSolution and adjust the source
+      node similar to the handling for MSVSProject.  The solution was generated in
+      the build directory instead of the source directory.  The placeholder solution
+      file is not generated in the build directory and the solution file is generated
+      in the source directory similar to the handling for project files.
+      Fixes #4612.
+    - MSVS: Add project dsp nodes to the dsw source node list in the msvs tool.  This
+      appears to always cause the project files to be generated before the solution
+      files which is necessary to retrieve the project GUIDs for use in the solution
+      file.  This does change the behavior of clean for a project generated with
+      auto_build_solution disabled and explicit solution generation: when the
+      solution files are cleaned, the project files are also cleaned.  The tests for
+      vs 6.0-7.1 were changed accordingly.
+    - MSVS: Add an optional keyword argument, auto_filter_projects, to MSVSSolution.
+      Accepted values for auto_filter_projects are:
+      - None [default]: raise an exception when solution file names or nodes are
+        detected in the projects argument list.
+      - True or evaluates True: automatically remove solution file names and nodes
+        from the project argument list.
+      - False or evaluates False: leave solution file names and nodes in the project
+        argument list.  An exception is not raised.
+      Solution file names and/or nodes in the project argument list cause erroneous
+      Project records to be produced in the generated solution file.  As a
+      convenience, an end-user may elect to ignore solution file names and nodes in
+      the projects argument list rather than manually removing solution file names
+      and nodes from the MSVSProject return values.  Resolves #4613.
+    - MSVS: Remove the platform specification (i.e., platform = 'win32') from select
+      test script environments.  The platform specification appears superfluous.
 
   From Alex James:
     - On Darwin, PermissionErrors are now handled while trying to access

RELEASE.txt

@@ -36,6 +36,26 @@ CHANGED/ENHANCED EXISTING FUNCTIONALITY
 
 - Added support for tracking beamer themes in the LaTeX scanner.
 
+- MSVS: msvs project files are always generated before the corresponding
+  msvs solution files. This changes the behavior of clean for a project
+  generated with auto_build_solution disabled and explicit solution
+  generation: when the solution files are cleaned, the project files are
+  also cleaned.  The tests for vs 6.0-7.1 were updated accordingly.
+
+- MSVS: Add an optional keyword argument, auto_filter_projects, to
+  MSVSSolution. Accepted values for auto_filter_projects are:
+  - None [default]: raise an exception when solution file names or nodes
+    are detected in the projects argument list.
+  - True or evaluates True: automatically remove solution file names and
+    nodes from the project argument list.
+  - False or evaluates False: leave solution file names and nodes in the
+    project argument list.  An exception is not raised.
+  Solution file names and/or nodes in the project argument list cause
+  erroneous Project records to be produced in the generated solution file.
+  As a convenience, a user may elect to ignore solution file names and nodes
+  in the projects argument list rather than manually removing solution file
+  names and nodes from the MSVSProject return values.
+
 - Add a tag to each CacheDir to let systems ignore backing it up
   (per https://bford.info/cachedir/). Update the way a CacheDir
   is created, since it now has to create two files.
@@ -58,11 +78,31 @@ FIXES
 
 - Fix a problem with compilation_db component initialization - the
   entries for assembler files were not being set up correctly.
+
 - On Darwin, PermissionErrors are now handled while trying to access
   /etc/paths.d. This may occur if SCons is invoked in a sandboxed environment
   (such as Nix).
-- Added error handling when creating MS VC detection debug log file specified by
-  SCONS_MSCOMMON_DEBUG
+
+- Added error handling when creating MSVC detection debug log file specified
+  by SCONS_MSCOMMON_DEBUG.
+
+- MSVS: Modify select msvs test scripts to run on platforms not supported by
+  the msvs/msvc tool implementation via a default host architecture for
+  unsupported platforms.
+
+- MSVS: Fixed early loop exit in select msvs test scripts.  Select msvs test
+  scripts were being invoked for msvc version 8.0 only.  Additional msvs
+  tool and test changes due to the msvs test scripts being run for all msvc
+  versions (i.e., minor test and tool issues went undetected).
+
+- MSVS: for variant build configurations, msvs solution files are
+  generated in the source directory and a placeholder file is generated in
+  the variant build directory.  This mirrors the behavior of generated
+  msvs project files.
+
+- MSVS: msvs project files are generated before the corresponding msvs
+  solution file.  User-specified project GUIDs should now be correctly
+  written to the solution file.
 
 - Fix nasm test for missing include file, cleanup.
 

SCons/Tool/msvs.py

@@ -124,6 +124,21 @@ def _generateGUID(slnfile, name, namespace=external_makefile_guid):
     return '{' + str(solution).upper() + '}'
 
 
+def _projectGUID(env, dspfile):
+    """Generates a GUID for the project file to use.
+
+    In order of preference:
+    * MSVS_PROJECT_GUID in environment
+    * Generated from the project file name (dspfile)
+
+    Returns (str)
+    """
+    project_guid = env.get('MSVS_PROJECT_GUID')
+    if not project_guid:
+        project_guid = _generateGUID(dspfile, '')
+    return project_guid
+
+
 version_re = re.compile(r'(\d+\.\d+)(.*)')
 
 
@@ -436,6 +451,10 @@ def __init__(self, dspfile, source, env) -> None:
         else:
             self.dspabs = get_abspath()
 
+        self.project_guid = _projectGUID(env, self.dspfile)
+        dspnode = env.File(self.dspfile)
+        dspnode.Tag('project_guid', self.project_guid)
+
         if 'variant' not in env:
             raise SCons.Errors.InternalError("You must specify a 'variant' argument (i.e. 'Debug' or " +\
                   "'Release') to create an MSVSProject.")
@@ -571,7 +590,7 @@ def GetKeyFromEnv(env, key, variants):
         self.configs = {}
 
         self.nokeep = 0
-        if 'nokeep' in env and env['variant'] != 0:
+        if 'nokeep' in env and env['nokeep'] != 0:
             self.nokeep = 1
 
         if self.nokeep == 0 and os.path.exists(self.dspabs):
@@ -912,8 +931,9 @@ def __init__(self, dspfile, source, env) -> None:
 
     def PrintHeader(self) -> None:
         env = self.env
-        versionstr = self.versionstr
         name = self.name
+        versionstr = self.versionstr
+        project_guid = self.project_guid
         encoding = self.env.subst('$MSVSENCODING')
         scc_provider = env.get('MSVS_SCC_PROVIDER', '')
         scc_project_name = env.get('MSVS_SCC_PROJECT_NAME', '')
@@ -923,9 +943,6 @@ def PrintHeader(self) -> None:
         scc_local_path_legacy = env.get('MSVS_SCC_LOCAL_PATH', '')
         scc_connection_root = env.get('MSVS_SCC_CONNECTION_ROOT', os.curdir)
         scc_local_path = os.path.relpath(scc_connection_root, os.path.dirname(self.dspabs))
-        project_guid = env.get('MSVS_PROJECT_GUID', '')
-        if not project_guid:
-            project_guid = _generateGUID(self.dspfile, '')
         if scc_provider != '':
             scc_attrs = '\tSccProjectName="%s"\n' % scc_project_name
             if scc_aux_path != '':
@@ -1209,8 +1226,8 @@ def PrintHeader(self) -> None:
         env = self.env
         name = self.name
         versionstr = self.versionstr
+        project_guid = self.project_guid
         encoding = env.subst('$MSVSENCODING')
-        project_guid = env.get('MSVS_PROJECT_GUID', '')
         scc_provider = env.get('MSVS_SCC_PROVIDER', '')
         scc_project_name = env.get('MSVS_SCC_PROJECT_NAME', '')
         scc_aux_path = env.get('MSVS_SCC_AUX_PATH', '')
@@ -1219,8 +1236,6 @@ def PrintHeader(self) -> None:
         scc_local_path_legacy = env.get('MSVS_SCC_LOCAL_PATH', '')
         scc_connection_root = env.get('MSVS_SCC_CONNECTION_ROOT', os.curdir)
         scc_local_path = os.path.relpath(scc_connection_root, os.path.dirname(self.dspabs))
-        if not project_guid:
-            project_guid = _generateGUID(self.dspfile, '')
         if scc_provider != '':
             scc_attrs = '\t\t<SccProjectName>%s</SccProjectName>\n' % scc_project_name
             if scc_aux_path != '':
@@ -1465,22 +1480,51 @@ def Build(self):
 
         _GenerateV10User.Build(self)
 
+def _projectDSPNodes(env):
+    auto_filter_projects = env.get('auto_filter_projects', None)
+    if 'projects' not in env:
+        raise SCons.Errors.UserError("You must specify a 'projects' argument to create an MSVSSolution.")
+    projects = env['projects']
+    if not SCons.Util.is_List(projects):
+        raise SCons.Errors.InternalError("The 'projects' argument must be a list of nodes.")
+    projects = SCons.Util.flatten(projects)
+    if len(projects) < 1:
+        raise SCons.Errors.UserError("You must specify at least one project to create an MSVSSolution.")
+    sln_suffix = os.path.normcase(env.subst('$MSVSSOLUTIONSUFFIX'))
+    dspnodes = []
+    for p in projects:
+        node = env.File(p)
+        if os.path.normcase(str(node)).endswith(sln_suffix):
+            # solution node detected (possible issues when opening generated solution file with VS IDE)
+            if auto_filter_projects is None:
+                nodestr = str(node)
+                errmsg = (
+                    f"An msvs solution file was detected in the MSVSSolution 'projects' argument: {nodestr!r}.\n"
+                    "  Add MSVSSolution argument 'auto_filter_projects=True' to suppress project solution nodes.\n"
+                    "  Add MSVSSolution argument 'auto_filter_projects=False' to keep project solution nodes.\n"
+                    "  Refer to the MSVSSolution documentation for more information."
+                )
+                raise SCons.Errors.UserError(errmsg)
+            elif auto_filter_projects:
+                # skip solution node
+                continue
+            else:
+                # keep solution node
+                pass
+        dspnodes.append(node)
+    if len(dspnodes) < 1:
+        raise SCons.Errors.UserError("You must specify at least one project node to create an MSVSSolution.")
+    return dspnodes
+
 class _DSWGenerator:
     """ Base class for DSW generators """
     def __init__(self, dswfile, source, env) -> None:
         self.dswfile = os.path.normpath(str(dswfile))
         self.dsw_folder_path = os.path.dirname(os.path.abspath(self.dswfile))
         self.env = env
 
-        if 'projects' not in env:
-            raise SCons.Errors.UserError("You must specify a 'projects' argument to create an MSVSSolution.")
-        projects = env['projects']
-        if not SCons.Util.is_List(projects):
-            raise SCons.Errors.InternalError("The 'projects' argument must be a list of nodes.")
-        projects = SCons.Util.flatten(projects)
-        if len(projects) < 1:
-            raise SCons.Errors.UserError("You must specify at least one project to create an MSVSSolution.")
-        self.dspfiles = list(map(str, projects))
+        dspnodes = _projectDSPNodes(env)
+        self.dsp_srcnodes = [dspnode.srcnode() for dspnode in dspnodes]
 
         if 'name' in self.env:
             self.name = self.env['name']
@@ -1519,7 +1563,7 @@ def __init__(self, dswfile, source, env) -> None:
         self.configs = {}
 
         self.nokeep = 0
-        if 'nokeep' in env and env['variant'] != 0:
+        if 'nokeep' in env and env['nokeep'] != 0:
             self.nokeep = 1
 
         if self.nokeep == 0 and os.path.exists(self.dswfile):
@@ -1554,7 +1598,9 @@ def AddConfig(self, variant, dswfile=dswfile) -> None:
                           if not (p.platform in seen or seen.add(p.platform))]
 
         def GenerateProjectFilesInfo(self) -> None:
-            for dspfile in self.dspfiles:
+            for dspnode in self.dsp_srcnodes:
+                project_guid = dspnode.GetTag('project_guid')
+                dspfile = str(dspnode)
                 dsp_folder_path, name = os.path.split(dspfile)
                 dsp_folder_path = os.path.abspath(dsp_folder_path)
                 if SCons.Util.splitext(name)[1] == '.filters':
@@ -1565,8 +1611,10 @@ def GenerateProjectFilesInfo(self) -> None:
                     dsp_relative_file_path = name
                 else:
                     dsp_relative_file_path = os.path.join(dsp_relative_folder_path, name)
+                if not project_guid:
+                    project_guid = _generateGUID(dspfile, '')
                 dspfile_info = {'NAME': name,
-                                'GUID': _generateGUID(dspfile, ''),
+                                'GUID': project_guid,
                                 'FOLDER_PATH': dsp_folder_path,
                                 'FILE_PATH': dspfile,
                                 'SLN_RELATIVE_FOLDER_PATH': dsp_relative_folder_path,
@@ -1615,7 +1663,7 @@ def PrintSolution(self) -> None:
         elif self.version_num >= 14.2:
             # Visual Studio 2019 is considered to be version 16.
             self.file.write('# Visual Studio 16\n')
-        elif self.version_num > 14.0:
+        elif self.version_num >= 14.0:
             # Visual Studio 2015 and 2017 are both considered to be version 15.
             self.file.write('# Visual Studio 15\n')
         elif self.version_num >= 12.0:
@@ -1632,7 +1680,7 @@ def PrintSolution(self) -> None:
         for dspinfo in self.dspfiles_info:
             name = dspinfo['NAME']
             base, suffix = SCons.Util.splitext(name)
-            if suffix == '.vcproj':
+            if suffix in ('.vcxproj', '.vcproj'):
                 name = base
             self.file.write('Project("%s") = "%s", "%s", "%s"\n'
                             % (external_makefile_guid, name, dspinfo['SLN_RELATIVE_FILE_PATH'], dspinfo['GUID']))
@@ -1645,7 +1693,7 @@ def PrintSolution(self) -> None:
 
         env = self.env
         if 'MSVS_SCC_PROVIDER' in env:
-            scc_number_of_projects = len(self.dspfiles) + 1
+            scc_number_of_projects = len(self.dsp_srcnodes) + 1
             slnguid = self.slnguid
             scc_provider = env.get('MSVS_SCC_PROVIDER', '').replace(' ', r'\u0020')
             scc_project_name = env.get('MSVS_SCC_PROJECT_NAME', '').replace(' ', r'\u0020')
@@ -1777,7 +1825,7 @@ class _GenerateV6DSW(_DSWGenerator):
     def PrintWorkspace(self) -> None:
         """ writes a DSW file """
         name = self.name
-        dspfile = os.path.relpath(self.dspfiles[0], self.dsw_folder_path)
+        dspfile = os.path.relpath(str(self.dsp_srcnodes[0]), self.dsw_folder_path)
         self.file.write(V6DSWHeader % locals())
 
     def Build(self):
@@ -1867,7 +1915,22 @@ def GenerateProject(target, source, env):
         GenerateDSW(dswfile, source, env)
 
 def GenerateSolution(target, source, env) -> None:
-    GenerateDSW(target[0], source, env)
+
+    builddswfile = target[0]
+    dswfile = builddswfile.srcnode()
+
+    if dswfile is not builddswfile:
+
+        try:
+            bdsw = open(str(builddswfile), "w+")
+        except OSError as detail:
+            print('Unable to open "' + str(dspfile) + '" for writing:',detail,'\n')
+            raise
+
+        bdsw.write("This is just a placeholder file.\nThe real workspace file is here:\n%s\n" % dswfile.get_abspath())
+        bdsw.close()
+
+    GenerateDSW(dswfile, source, env)
 
 def projectEmitter(target, source, env):
     """Sets up the DSP dependencies."""
@@ -1961,7 +2024,7 @@ def projectEmitter(target, source, env):
     sourcelist = source
 
     if env.get('auto_build_solution', 1):
-        env['projects'] = [env.File(t).srcnode() for t in targetlist]
+        env['projects'] = [env.File(t) for t in targetlist]
         t, s = solutionEmitter(target, target, env)
         targetlist = targetlist + t
 
@@ -2028,6 +2091,9 @@ def solutionEmitter(target, source, env):
         source = source + ' "%s"' % str(target[0])
         source = [SCons.Node.Python.Value(source)]
 
+    dsp_nodes = _projectDSPNodes(env)
+    source.extend(dsp_nodes)
+
     return ([target[0]], source)
 
 projectAction = SCons.Action.Action(GenerateProject, None)