Merge pull request #4470 from mwichmann/doc/scanner-examples

Fix scanner examples in User Guide

CHANGES.txt

@@ -72,6 +72,8 @@ RELEASE  VERSION/DATE TO BE FILLED IN LATER
     - Be more cautious about encodings fetching command output on Windows.
       Problem occurs in piped-spawn scenario, used by Configure tests.
       Fixes #3529.
+    - Clarify/fix documentation of Scanners in User Guide and Manpage.
+      Fixes #4468.
 
 
 RELEASE 4.6.0 -  Sun, 19 Nov 2023 17:22:20 -0700

RELEASE.txt

@@ -73,9 +73,9 @@ PACKAGING
 DOCUMENTATION
 -------------
 
-- List any significant changes to the documentation (not individual
-  typo fixes, even if they're mentioned in src/CHANGES.txt to give
-  the contributor credit)
+- Fixed the Scanner examples in the User Guide to be runnable and added
+  some more explantion. Clarified discussion of the scanner function in
+  the Scanner Objects section of the manpage.
 
 DEVELOPMENT
 -----------

doc/generated/examples/scanners_builders_1.xml

@@ -0,0 +1,5 @@
+<screen xmlns="http://www.scons.org/dbxsd/v1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.scons.org/dbxsd/v1.0 http://www.scons.org/dbxsd/v1.0/scons.xsd">% <userinput>scons -Q</userinput>
+DEBUG: scan of 'file.input' found ['other_file']
+DEBUG: scanned dependencies found: ['inc/other_file']
+build_function(["file.k"], ["file.input"])
+</screen>

doc/generated/examples/scanners_scan_1.xml

@@ -0,0 +1,3 @@
+<screen xmlns="http://www.scons.org/dbxsd/v1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.scons.org/dbxsd/v1.0 http://www.scons.org/dbxsd/v1.0/scons.xsd">% <userinput>scons -Q</userinput>
+scons: *** [foo] Implicit dependency `other_file' not found, needed by target `foo'.
+</screen>

doc/generated/examples/scanners_scan_foo.k

@@ -0,0 +1,5 @@
+
+some initial text
+include other_file
+some other text
+

doc/man/scons.xml

@@ -5634,7 +5634,7 @@ may be a string or a list of strings.</para>
   </listitem>
   </varlistentry>
 
-  <varlistentry>
+  <varlistentry id="target_scanner">
   <term><parameter>target_scanner</parameter></term>
   <listitem>
 <para>A Scanner object that
@@ -5652,7 +5652,7 @@ for information about creating Scanner objects.</para>
   </listitem>
   </varlistentry>
 
-  <varlistentry>
+  <varlistentry id="source_scanner">
   <term><parameter>source_scanner</parameter></term>
   <listitem>
 <para>A Scanner object that
@@ -7243,11 +7243,12 @@ the rest are optional:
   <term><parameter>function</parameter></term>
   <listitem>
 <para>
-A scanner function to call to process
+A function which can process ("scan")
 a given Node (usually a file)
 and return a list of Nodes
-representing the implicit
-dependencies (usually files) found in the contents.
+representing any implicit
+dependencies (usually files) which will be tracked
+for the Node.
 The function must accept three required arguments,
 <parameter>node</parameter>,
 <parameter>env</parameter> and
@@ -7260,59 +7261,98 @@ the internal &SCons; node representing the file to scan,
 the scan, and <parameter>path</parameter> is a tuple
 of directories that can be searched for files,
 as generated by the optional scanner
-<parameter>path_function</parameter> (see below).
-If <parameter>argument</parameter> was supplied when the Scanner
-object was created, it is given as <parameter>arg</parameter>
-when the scanner function is called; since <parameter>argument</parameter>
-is optional, the default is no <parameter>arg</parameter>.
+<xref linkend="path_function"/>.
+If the <xref linkend="scanner-argument"/>
+parameter was supplied when the Scanner object was created,
+it is passed as the <parameter>arg</parameter> parameter
+to the scanner function when it is called.
+Since <parameter>argument</parameter> is optional,
+the scanner function <emphasis>may</emphasis> be
+called without an <parameter>arg</parameter> parameter.
 </para>
+
 <para>
-The function can use use
+The scanner function can make use of
 <function>str</function>(<parameter>node</parameter>)
 to fetch the name of the file,
-<replaceable>node</replaceable>.<function>dir</function>
+<parameter>node</parameter>.<methodname>dir</methodname>
 to fetch the directory the file is in,
-<replaceable>node</replaceable>.<function>get_contents</function>()
+<parameter>node</parameter>.<methodname>get_contents</methodname>()
 to fetch the contents of the file as bytes or
-<replaceable>node</replaceable>.<function>get_text_contents</function>()
+<parameter>node</parameter>.<methodname>get_text_contents</methodname>()
 to fetch the contents of the file as text.
 </para>
+
 <para>
-The function must take into account the <parameter>path</parameter>
-directories when generating the dependency Nodes. To illustrate this,
-a C language source file may contain a line like
-<literal>#include "foo.h"</literal>.  However, there is no guarantee
-that <filename>foo.h</filename> exists in the current directory:
-the contents of &cv-link-CPPPATH; is passed to the C preprocessor which
-will look in those places for the header,
-so the scanner function needs to look in those places as well
-in order to build Nodes with correct paths.
-Using &f-link-FindPathDirs; with an argument of <literal>CPPPATH</literal>
-as the <parameter>path_function</parameter> in the &f-Scanner; call
-means the scanner function will be called with the paths extracted
-from &cv-CPPPATH; in the environment <parameter>env</parameter>
-passed as the <parameter>paths</parameter> parameter.
-</para>
-<para>
-Note that the file to scan is
-<emphasis>not</emphasis>
-guaranteed to exist at the time the scanner is called -
-it could be a generated file which has not been generated yet -
-so the scanner function must be tolerant of that.
+The scanner function should account for any directories
+listed in the <parameter>path</parameter> parameter
+when determining the existence of possible dependencies.
+External tools such as the C/C++ preprocessor are given
+lists of directories to search for source file inclusion directives
+(e.g.  <literal>#include "myheader.h"</literal>).
+That list is generated from the relevant path variable
+(e.g. &cv-link-CPPPATH; for C/C++). The Scanner can be
+directed to pass the same list on to the scanner function
+via the <parameter>path</parameter> parameter so it can
+search in the same places.
+The Scanner is enabled to pass this list via the
+<xref linkend="path_function"/> argument at Scanner creation time.
 </para>
+
 <para>
-Alternatively, you can supply a dictionary as the
-<parameter>function</parameter> parameter,
-to map keys (such as file suffixes) to other Scanner objects.
+Instead of a scanner function, you can supply a dictionary as the
+<parameter>function</parameter> parameter.
+The dictionary must map keys (such as file suffixes)
+to other Scanner objects.
 A Scanner created this way serves as a dispatcher:
-the Scanner's <parameter>skeys</parameter> parameter is
+the Scanner's <xref linkend="skeys"/> parameter is
 automatically populated with the dictionary's keys,
 indicating that the Scanner handles Nodes which would be
 selected by those keys; the mapping is then used to pass
 the file on to a different Scanner that would not have been
 selected to handle that Node based on its
 own <parameter>skeys</parameter>.
 </para>
+
+<para>
+Note that the file to scan is
+<emphasis>not</emphasis>
+guaranteed to exist at the time the scanner is called -
+it could be a generated file which has not been generated yet -
+so the scanner function must be tolerant of that.
+</para>
+
+<para>
+While many scanner functions operate on source code files by
+looking for known patterns in the code, they can really
+do anything they need to.
+For example, the &b-link-Program; Builder is assigned a
+<xref linkend="target_scanner"/> which examines the
+list of libraries supplied for the build (&cv-link-LIBS;)
+and decides whether to add them as dependencies,
+it does not look <emphasis>inside</emphasis> the built binary.
+</para>
+
+<para>
+It is up to the scanner function to decide whether or not to
+generate an &SCons; dependency for candidates identified by scanning.
+Dependencies are a key part of &SCons; operation,
+enabling both rebuild determination and correct ordering of builds.
+It is particularly important that generated files which are
+dependencies are added into the Node graph,
+or use-before-create failures are likely.
+However, not everything may need to be tracked as a dependency.
+In some cases, implementation-provided header files change
+infrequently but are included very widely,
+so tracking them in the &SCons; node graph could become quite
+expensive for limited benefit -
+consider for example the C standard header file
+<filename>string.h</filename>.
+The scanner function is not passed any special information
+to help make this choice, so the decision making encoded
+in the scanner function must be carefully considered.
+</para>
+
   </listitem>
   </varlistentry>
 
@@ -7325,7 +7365,7 @@ The default value is <literal>"NONE"</literal>.</para>
   </listitem>
   </varlistentry>
 
-  <varlistentry>
+  <varlistentry id="scanner-argument">
   <term><parameter>argument</parameter></term>
   <listitem>
 <para>If specified,
@@ -7339,7 +7379,7 @@ as the optional parameter each of those functions takes.
   </listitem>
   </varlistentry>
 
-  <varlistentry>
+  <varlistentry id="skeys">
   <term><parameter>skeys</parameter></term>
   <listitem>
 <para>Scanner key(s) indicating the file types
@@ -7355,27 +7395,42 @@ it will be expanded into a list by the current environment.
   </listitem>
   </varlistentry>
 
-  <varlistentry>
+  <varlistentry id="path_function">
   <term><parameter>path_function</parameter></term>
   <listitem>
-<para>A Python function that takes four or five arguments:
+<para>
+If specified, a function to generate paths to pass to
+the scanner function to search while generating dependencies.
+The function must take five arguments:
 a &consenv;,
 a Node for the directory containing
 the &SConscript; file in which
 the first target was defined,
 a list of target nodes,
 a list of source nodes,
 and the value of <parameter>argument</parameter>
-if it was supplied when the Scanner was created.
+if it was supplied when the Scanner was created
+(since <parameter>argument</parameter> is optional,
+the function may be called without this argument,
+the <parameter>path_function</parameter>
+should be prepared for this).
 Must return a tuple of directories
 that can be searched for files to be returned
 by this Scanner object.
-(Note that the
-&f-link-FindPathDirs;
-function can be used to return a ready-made
+</para>
+
+<para>
+The &f-link-FindPathDirs;
+function can be called to return a ready-made
 <parameter>path_function</parameter>
 for a given &consvar; name,
-instead of having to write your own function from scratch.)
+which is often easier than writing your own function from scratch.
+For example,
+<userinput>path_function=FindPathDirs('CPPPATH')</userinput>
+means the scanner function will be called with the paths extracted
+from &cv-CPPPATH; in the &consenv; <parameter>env</parameter>,
+and passed as the <parameter>path</parameter> parameter
+to the scanner function.
 </para>
   </listitem>
   </varlistentry>