Merge commit 'upstream/1.2.0.d20091224'

1 files changed, 898 insertions, 0 deletions

diff --git a/doc/python10/design.xml b/doc/python10/design.xml
new file mode 100644
index 0000000..cb58af9
--- /dev/null
+++ b/doc/python10/design.xml
@@ -0,0 +1,898 @@
+<para>
+
+  The &SCons; architecture consists of three layers:
+
+</para>
+
+<mediaobject>
+  <imageobject>
+    <imagedata fileref="arch" format="eps" align="center">
+  </imageobject>
+  <imageobject>
+    <imagedata fileref="arch.jpg" format="jpg" align="center">
+  </imageobject>
+  <!--  PDF files?
+  <imageobject>
+    <imagedata fileref="arch.pdf" align="center">
+  </imageobject>
+  -->
+</mediaobject>
+
+<itemizedlist>
+
+  <listitem>
+    <para>
+
+      The &SCons; <emphasis>Build Engine</emphasis>, a package of Python
+      modules that handle dependency management and updating out-of-date
+      objects.
+
+    </para>
+  </listitem>
+
+  <listitem>
+    <para>
+
+      The &SCons; <emphasis>API</emphasis> (applications programming
+      interface) between the Build Engine
+      and the user interface.
+
+    </para>
+  </listitem>
+
+  <listitem>
+    <para>
+
+      The &scons; <emphasis>script</emphasis> itself (note lower case
+      <emphasis>sc</emphasis>), which is the pre-provided interface to
+      the Build Engine.
+
+    </para>
+  </listitem>
+
+</itemizedlist>
+
+<para>
+
+  Notice that this architecture separates the internal workings of
+  &SCons; (the Build Engine) from the
+  external user interface.  The benefit is that the &SCons; Build Engine
+  can be imported into any other software package written in Python
+  to support a variety of user interfaces&mdash;or, to look at it
+  in reverse, other software interfaces can use the &SCons; Build
+  Engine to manage dependencies between their objects.
+
+</para>
+
+<para>
+
+  Because the
+  &SCons; package itself is modular, only those parts of the package
+  relevant to the embedding interface need be imported; for example,
+  a utility that wants to use only file timestamps for checking
+  whether a file is up-to-date
+  need not import the MD5 signature module.
+
+</para>
+
+<section>
+  <title>The &SCons; Build Engine</title>
+
+  <para>
+
+    The Build Engine is a package of Python modules that
+    form the heart of &SCons;.
+
+    The Build Engine can be broadly divided into five
+    architectural subsystems, each responsible
+    for a crucial part of &SCons; functionality:
+
+  </para>
+
+  <itemizedlist>
+
+    <listitem>
+      <para>
+
+        A <emphasis>node</emphasis> subsystem, responsible for managing
+        the files (or other objects) to be built, and the dependency
+        relationships between them.
+
+      </para>
+    </listitem>
+
+    <listitem>
+      <para>
+
+        A <emphasis>scanner</emphasis> subsystem, responsible for
+	scanning various file types for implicit dependencies.
+
+      </para>
+    </listitem>
+
+    <listitem>
+      <para>
+
+        A <emphasis>signature</emphasis> subsystem, responsible for
+        deciding whether a given file (or other object) requires
+        rebuilding.
+
+      </para>
+    </listitem>
+
+    <listitem>
+      <para>
+
+        A <emphasis>builder</emphasis> subsystem, responsible for
+        actually executing the necessary command or function to
+        build a file (or other object).
+
+      </para>
+    </listitem>
+
+    <listitem>
+      <para>
+
+        A <emphasis>job/task</emphasis> subsystem, responsible for
+        handling parallelization of builds.
+
+      </para>
+    </listitem>
+
+  </itemizedlist>
+
+  <para>
+
+    The rest of this section will provide a high-level overview of the
+    class structure of each of these Build Engine subsystems.
+
+  </para>
+
+  <section>
+    <title>Node Subsystem</title>
+
+    <para>
+
+      The node subsystem of the Build Engine is
+      responsible for managing the knowledge in &SCons; of
+      the relationships among the external objects
+      (files) it is responsible for updating.
+      The most important of these relationships is
+      the dependency relationship between various &Node; objects,
+      which &SCons; uses to determine the order
+      in which builds should be performed.
+
+    </para>
+
+    <mediaobject>
+      <imageobject>
+        <imagedata fileref="node" format="eps" align="center">
+      </imageobject>
+      <imageobject>
+        <imagedata fileref="node.jpg" format="jpg" align="center">
+      </imageobject>
+      <!--  PDF files?
+      <imageobject>
+        <imagedata fileref="node.pdf" align="center">
+      </imageobject>
+      -->
+    </mediaobject>
+
+    <para>
+
+      The &scons; script (or other
+      user interface)
+      tells the Build Engine
+      about dependencies
+      through its &consenv; API.
+      The Build Engine also discovers
+      dependencies automatically through the use of &Scanner; objects.
+
+    </para>
+
+    <para>
+
+      Subclasses of the &Node; class maintain additional
+      relationships that reflect the real-world
+      existence of these objects.
+      For example, the &Node_FS; subclass
+      is responsible for managing a
+      representation of the directory hierarchy
+      of a file system.
+
+    </para>
+
+    <para>
+
+      A &Walker; class is used by other subsystems
+      to walk the dependency tree maintained by the &Node; class.
+      The &Walker; class maintains a stack of &Node; objects
+      visited during its depth-first traversal of the
+      dependency tree,
+      and uses an intermediate node &Wrapper; class
+      to maintain state information about a
+      &Node; object's dependencies.
+
+    </para>
+
+  </section>
+
+  <section>
+    <title>Scanner Subsystem</title>
+
+    <para>
+
+      The scanner subsystem is responsible for maintaining
+      objects that can scan the contents of a &Node;'s
+      for implicit dependencies.
+
+    </para>
+
+    <mediaobject>
+      <imageobject>
+        <imagedata fileref="scanner" format="eps" align="center">
+      </imageobject>
+      <imageobject>
+        <imagedata fileref="scanner.jpg" format="jpg" align="center">
+      </imageobject>
+      <!--  PDF files?
+      <imageobject>
+        <imagedata fileref="scanner.pdf" align="center">
+      </imageobject>
+      -->
+    </mediaobject>
+
+    <para>
+
+      In practice, a given &Scanner; subclass object
+      functions as a prototype,
+      returning clones of itself
+      depending on the &consenv;
+      values governing how the &Node;
+      should be scanned.
+
+    </para>
+
+  </section>
+
+  <section>
+    <title>Signature Subsystem</title>
+
+    <para>
+
+      The signature subsystem is responsible for computing
+      signature information for &Node; objects.
+      The signature subsystem in &SCons;
+      supports multiple ways to
+      determine whether a &Node is up-to-date
+      by using an abstract &Sig; class
+      as a strategy wrapper:
+
+    </para>
+
+    <mediaobject>
+      <imageobject>
+        <imagedata fileref="sig" format="eps" align="center">
+      </imageobject>
+      <imageobject>
+        <imagedata fileref="sig.jpg" format="jpg" align="center">
+      </imageobject>
+      <!--  PDF files?
+      <imageobject>
+        <imagedata fileref="sig.pdf" align="center">
+      </imageobject>
+      -->
+    </mediaobject>
+
+    <para>
+
+      By default, &SCons; tracks dependencies by computing and
+      maintaining MD5 signatures for the contents of each source file
+      (or other object). The signature of a <emphasis>derived</emphasis>
+      file consists of the aggregate of the signatures of all the source
+      files <emphasis>plus</emphasis> the command-line string used to
+      build the file. These signatures are stored in a &sconsign; file
+      in each directory.
+
+    </para>
+
+    <para>
+
+      If the contents of any of the source files changes, the change to its
+      MD5 signature is propogated to the signature of the derived file(s). The
+      simple fact that the new signature does not match the stored signature
+      indicates that the derived file is not up to date and must be rebuilt.
+
+    </para>
+
+    <para>
+
+      A separate &TimeStamp; subclass of the &Sig; class supports
+      the use of traditional file timestamps for
+      deciding whether files are up-to-date.
+
+    </para>
+
+  </section>
+
+  <section>
+    <title>Builder Subsystem</title>
+
+    <para>
+
+      The &SCons; Build Engine records how out-of-date files
+      (or other objects) should be rebuilt in &Builder; objects,
+      maintained by the builder subsystem:
+
+    </para>
+
+    <mediaobject>
+      <imageobject>
+        <imagedata fileref="builder" format="eps" align="center">
+      </imageobject>
+      <imageobject>
+        <imagedata fileref="builder.jpg" format="jpg" align="center">
+      </imageobject>
+      <!--  PDF files?
+      <imageobject>
+        <imagedata fileref="builder.pdf" align="center">
+      </imageobject>
+      -->
+    </mediaobject>
+
+    <para>
+      
+      The actual underlying class name is &BuilderBase;,
+      and there are subclasses that can encapsulate
+      multiple &Builder; objects for special purposes.
+      One subclass
+      (&CompositeBuilder;)
+      selects an appropriate encapsulated &Builder;
+      based on the file suffix of the target object.
+      The other
+      (&MultiStepBuilder;).
+      can chain together multiple
+      &Builder; objects,
+      for example,
+      to build an executable program from a source file
+      through an implicit intermediate object file.
+
+    </para>
+
+    <para>
+
+      A &BuilderBase; object has an associated
+      &ActionBase; object
+      responsible for actually executing
+      the appropriate steps
+      to update the target file.
+      There are three subclasses,
+      one for externally executable commands
+      (&CommandAction;),
+      one for Python functions
+      (&FunctionAction;),
+      and one for lists of
+      multiple &Action; objects
+      (&ListAction;).
+
+    </para>
+
+  </section>
+
+  <section>
+    <title>Job/Task Subsystem</title>
+
+    <para>
+
+      &SCons; supports parallel builds with a thread-based tasking
+      model, managed by the job/task subsystem.
+
+    </para>
+
+    <mediaobject>
+      <imageobject>
+        <imagedata fileref="job-task" format="eps" align="center">
+      </imageobject>
+      <imageobject>
+        <imagedata fileref="job-task.jpg" format="jpg" align="center">
+      </imageobject>
+      <!--  PDF files?
+      <imageobject>
+        <imagedata fileref="job-task.pdf" align="center">
+      </imageobject>
+      -->
+    </mediaobject>
+
+    <para>
+      
+      Instead of performing an outer-loop recursive descent
+      of the dependency tree and then forking a task when it finds a
+      file that needs updating, &SCons; starts as many threads as are
+      requested, each thread managed by the &Jobs; class.
+      As a performance optimization,
+      the &Jobs; class maintains an internal
+      distinction between
+      &Serial; and &Parallel;
+      build jobs,
+      so that serial builds
+      don't pay any performance penalty
+      by using a multi-threaded implementation
+      written for &Parallel; builds.
+
+    </para>
+
+    <para>
+      
+      Each &Jobs; object, running in its own thread,
+      then requests a &Task; from a central &Taskmaster;,
+      which is responsible
+      for handing out available &Task; objects for (re-)building
+      out-of-date nodes. A condition variable
+      makes sure that the &Jobs; objects
+      query the &Taskmaster; one at a time.
+
+    </para>
+
+    <para>
+
+      The &Taskmaster uses the node subsystem's
+      &Walker; class to walk the dependency tree,
+      and the &Sig; class to use the
+      appropriate method
+      of deciding if a &Node; is up-to-date.
+
+    </para>
+
+    <para>
+
+      This scheme has many advantages over the standard &Make;
+      implementation of <option>-j</option>.
+      Effective use of <option>-j</option> is difficult
+      with the usual recursive use of Make,
+      because the number of jobs started by <option>-j</option> multiply
+      at each level of the source tree.
+      This makes the actual number of jobs
+      executed at any moment very dependent on the size and layout of
+      the tree. &SCons;, in contrast, starts only as many jobs as are
+      requested, and keeps them constantly busy (excepting jobs that
+      block waiting for their dependency files to finish building).
+
+    </para>
+
+  </section>
+
+</section>
+
+<section>
+  <title>The &SCons; API</title>
+
+  <para>
+
+    This section provides an overview of the &SCons; interface. The
+    complete interface specification is both more detailed and flexible
+    than this overview.
+
+  </para>
+
+  <section>
+    <title>&ConsVars;</title>
+
+    <para>
+
+      In &SCons;, a &consenv; is an object through which an external
+      interface (such as the &scons; script) communicates dependency
+      information to the &SCons; Build Engine.
+
+    </para>
+
+    <para>
+
+      A construction environment is implemented as a dictionary
+      containing:
+
+    </para>
+
+    <itemizedlist>
+
+      <listitem>
+        <para>
+
+          construction variables, string values that are substituted
+          into command lines or used by builder functions;
+
+        </para>
+      </listitem>
+
+      <listitem>
+        <para>
+
+          one or more &Builder; objects that can be invoked to update a
+          file or other object;
+
+        </para>
+      </listitem>
+
+      <listitem>
+        <para>
+
+          one or more &Scanner; objects that can be used to
+          scan a file automatically for dependencies (such as
+          files specified on <literal>#include</literal> lines).
+
+        </para>
+      </listitem>
+
+    </itemizedlist>
+
+    <para>
+
+      &Consenvs; are instantiated as follows:
+
+    </para>
+
+    <programlisting>
+      env = Environment()
+      env_debug = Environment(CCFLAGS = '-g')
+    </programlisting>
+
+  </section>
+
+  <section>
+    <title>&Builder; Objects</title>
+
+    <para>
+
+      An &SCons; &Builder; object encapsulates information about how to
+      build a specific type of file: an executable program, an object
+      file, a library, etc. A &Builder; object is associated with a
+      file through an associated &consenv; method and later invoked to
+      actually build the file. The &Builder; object will typically use
+      construction variables (such as &CCFLAGS;, &LIBPATH;) to influence
+      the specific build execution.
+
+    </para>
+
+    <para>
+
+      &Builder; objects are instantiated as follows:
+
+    </para>
+
+    <programlisting>
+      bld = Builder(name = 'Program', action = "$CC -o $TARGET $SOURCES")
+    </programlisting>
+
+    <para>
+
+      In the above example, the <literal>action</literal> is a
+      command-line string in which the Build Engine will
+      interpolate the values of construction
+      variables before execution. The actual
+      <literal>action</literal> specified, though,
+      may be a function:
+
+    </para>
+
+    <programlisting>
+      def update(dest):
+          # [code to update the object]
+          return 0
+
+      bld = Builder(name = 'Program', function = update)
+    </programlisting>
+
+    <para>
+
+      Or a callable Python object (or class):
+
+    </para>
+
+    <programlisting>
+      class class_a:
+          def __call__(self, kw):
+              # build the desired object
+          return 0
+
+      builder = SCons.Builder.Builder(action = class_a())
+    </programlisting>
+
+    <para>
+
+      A &Builder; object may have the <literal>prefix</literal> and
+      <literal>suffix</literal> of its target file type specified
+      as keyword arguments at instantiation.  Additionally, the
+      suffix of the <emphasis>source files</emphasis> used by this
+      &Builder; to build its target files may be specified using the
+      <literal>src_suffix</literal> keyword argument:
+
+    </para>
+
+    <programlisting>
+      bld_lib = Builder(name = 'Library', action = "$AR r $TARGET $SOURCES",
+                        prefix = 'lib', suffix = '.a', src_suffix = '.o')
+    </programlisting>
+
+    <para>
+
+      The specified <literal>prefix</literal> and
+      <literal>suffix</literal> will be appended to the name of any
+      target file built by this &Builder; object, if they are not
+      already part of the file name.  The <literal>src_suffix</literal>
+      is used by the &SCons; Build Engine to chain together
+      multiple &Builder; objects to create,
+      for example, a library from the original source
+      files without having to specify the
+      intermediate <literal>.o</literal> files.
+
+    </para>
+
+    <para>
+
+      &Builder; objects are associated with a &consenv; through a
+      &consvar; named &BUILDERS;, a list of the &Builder objects that
+      will be available for execution through the &consenv:
+
+    </para>
+
+    <programlisting>
+      env = Environment(BUILDERS = [ Object, Library, WebPage, Program ])
+    </programlisting>
+
+  </section>
+
+  <section>
+    <title>&Scanner; Objects</title>
+
+    <para>
+
+      &Scanner; objects perform automatic checking for dependencies
+      by scanning the contents of files. The canonical
+      example is scanning a C source file or header file for
+      files specified on <literal>#include</literal> lines.
+
+    </para>
+
+    <para>
+
+      A &Scanner; object is instantiated as follows:
+
+    </para>
+
+    <programlisting>
+      def c_scan(contents):
+           # scan contents of file
+           return # list of files found
+
+      c_scanner = Scanner(name = 'CScan', function = c_scan,
+                          argument = None,
+                          skeys = ['.c', '.C', '.h', '.H')
+    </programlisting>
+
+    <para>
+
+      The <literal>skeys</literal> argument specifies a list of file
+      suffixes for file types that this &Scanner; knows how to scan.
+
+    </para>
+
+    <para>
+
+      &Scanner; objects are associated with a &consenv; through a
+      &consvar; named &SCANNERS;, a list of the &Scanner; objects that
+      will be available through the &consenv:
+
+    </para>
+
+    <programlisting>
+      env = Environment(SCANNERS = [ CScan, M4Scan ])
+    </programlisting>
+
+    <para>
+
+      For utilities that will build files with a variety of file
+      suffixes, or which require unusual scanning rules, a &Scanner;
+      object may be associated explicitly with a &Builder; object as
+      follows:
+
+    </para>
+
+    <programlisting>
+      def tool_scan(contents):
+          # scan contents of file
+          return # list of files found
+
+      tool_scanner = Scanner(name = 'TScan', function = tool_scan)
+
+      bld = Builder(name = 'Tool', scanner = tool_scanner)
+    </programlisting>
+
+  </section>
+
+  <section>
+    <title>&BuildDir;</title>
+
+    <para>
+
+      &SCons; supports a flexible mechanism for building target
+      files in a separate build directory from the source files.
+      The &BuildDir; syntax is straightforward:
+
+    </para>
+
+    <programlisting>
+      BuildDir(source = 'src', build = 'bld')
+    </programlisting>
+
+    <para>
+
+      By
+      default, source files are linked or copied into the build
+      directory, because exactly replicating the source directory
+      is sometimes necessary for certain combinations of use of
+      <literal>#include "..."</literal> and <option>-I</option> search
+      paths.
+
+      An option exists to specify that only output files should be placed in
+      the build directory:
+
+    </para>
+
+    <programlisting>
+      BuildDir(source = 'src', build = 'bld', no_sources = 1)
+    </programlisting>
+
+  </section>
+
+  <section>
+    <title>&Repository;</title>
+
+    <para>
+
+      &SCons; supports the ability to search a list of code repositories
+      for source files and derived files. This works much like
+      &Make;'s <varname>VPATH</varname> feature, as implemented in
+      recent versions of GNU &Make;.
+      (The POSIX standard for &Make; specifies slightly
+      different behavior for <varname>VPATH</varname>.)
+      The syntax is:
+
+    </para>
+
+    <programlisting>
+      Repository('/home/source/1.1', '/home/source/1.0')
+    </programlisting>
+
+    <para>
+
+      A command-line <option>-Y</option> option exists to allow
+      repositories to be specified on the command line, or in the
+      &SCONSFLAGS; environment variable (not construction variable!).
+      This avoids a chicken-and-egg situation and allows the top-level
+      &SConstruct; file to be found in a repository as well.
+
+    </para>
+
+  </section>
+
+  <section>
+    <title>&Cache;</title>
+
+    <para>
+
+      &SCons; supports a way for developers to share derived files. Again, the
+      syntax is straightforward:
+
+    </para>
+
+    <programlisting>
+      Cache('/var/build.cache/i386')
+    </programlisting>
+
+    <para>
+
+      Copies of any derived files built will be placed in the specified
+      directory with their MD5 signature. If another build results in an
+      out-of-date derived file with the same signature, the derived file
+      will be copied from the cache instead of being rebuilt.
+
+    </para>
+
+  </section>
+
+</section>
+
+<section>
+  <title>The &scons; Script</title>
+
+  <para>
+
+    The &scons; script provides an interface
+    that looks roughly equivalent to the
+    classic &Make; utility&mdash;that is, execution from the command
+    line, and dependency information read from configuration files.
+
+  </para>
+
+  <para>
+
+    The most noticeable difference between &scons; and &Make;, or most
+    other build tools, is that the configuration files are actually
+    Python scripts, generically called "SConscripts" (although the
+    top-level "Makefile" is named &SConstruct). Users do not have to
+    learn a new language syntax, but instead configure dependency
+    information by making direct calls to the Python API of the
+    &SCons; Build Engine. Here is an example &SConstruct file which
+    builds a program in side-by-side normal and debug versions:
+
+  </para>
+
+  <programlisting>
+    env = Environment()
+    debug = env.Copy(CCFLAGS = '-g')
+
+    source_files = ['f1.c', 'f2.c', 'f3.c']
+
+    env.Program(target = 'foo', sources = source_files)
+    debug.Program(target = 'foo-debug', sources = source_files)
+  </programlisting>
+
+  <para>
+
+    Notice the fact that this file is a Python script, which allows us
+    to define and re-use an array that lists the source files.
+
+  </para>
+
+  <para>
+
+    Because quoting individul strings in long
+    lists of files can get tedious and error-prone, the &SCons;
+    methods support a short-cut of listing multiple files in a single
+    string, separated by white space.
+    This would change
+    the assignment in the above example to a more easily-readable:
+
+  </para>
+
+  <programlisting>
+    source_files = 'f1.c f2.c f3.c'
+  </programlisting>
+
+  <para>
+
+    The mechanism to establish hierarchical builds is to "include" any
+    subsidiary configuration files in the build by listing them explicitly
+    in a call to the &SConscript; function:
+
+  </para>
+
+  <programlisting>
+    SConscript('src/SConscript', 'lib/SConscript')
+  </programlisting>
+
+  <para>
+
+    By convention, configuration files in subdirectories are named
+    &SConscript;.
+
+  </para>
+
+  <para>
+
+    The &scons; script has intentionally been made to look, from
+    the outside, as much like &Make; as is practical. To this
+    end, the &scons; script supports all of the same command-line
+    options supported by GNU &Make;: <option>-f</option> FILE,
+    <option>-j</option>, <option>-k</option>, <option>-s</option>,
+    etc. For compatibility, &scons; ignores those GNU &Make; options
+    that don't make sense for the &SCons; architecture, such as
+    <option>-b</option>, <option>-m</option>, <option>-S</option>,
+    and <option>-t</option>.  The
+    intention is that, given an equivalent &SConstruct; file for a
+    &Makefile;, a user could use &SCons; as a drop-in replacement for
+    &Make;. Additional command-line options are, where possible, taken
+    from the Perl &Cons; utility on which the &SCons; design is based.
+
+  </para>
+
+</section>