summaryrefslogtreecommitdiff
path: root/doc/user/caching.in
diff options
context:
space:
mode:
Diffstat (limited to 'doc/user/caching.in')
-rw-r--r--doc/user/caching.in502
1 files changed, 502 insertions, 0 deletions
diff --git a/doc/user/caching.in b/doc/user/caching.in
new file mode 100644
index 0000000..31b1103
--- /dev/null
+++ b/doc/user/caching.in
@@ -0,0 +1,502 @@
+<!--
+
+ Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009 The SCons Foundation
+
+ Permission is hereby granted, free of charge, to any person obtaining
+ a copy of this software and associated documentation files (the
+ "Software"), to deal in the Software without restriction, including
+ without limitation the rights to use, copy, modify, merge, publish,
+ distribute, sublicense, and/or sell copies of the Software, and to
+ permit persons to whom the Software is furnished to do so, subject to
+ the following conditions:
+
+ The above copyright notice and this permission notice shall be included
+ in all copies or substantial portions of the Software.
+
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
+ KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
+ WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+-->
+
+ <para>
+
+ On multi-developer software projects,
+ you can sometimes speed up every developer's builds a lot by
+ allowing them to share the derived files that they build.
+ &SCons; makes this easy, as well as reliable.
+
+ </para>
+
+ <section>
+ <title>Specifying the Shared Cache Directory</title>
+
+ <para>
+
+ To enable sharing of derived files,
+ use the &CacheDir; function
+ in any &SConscript; file:
+
+ </para>
+
+ <scons_example name="ex1">
+ <file name="SConstruct">
+ env = Environment()
+ env.Program('hello.c')
+ CacheDir('cache')
+ </file>
+ <file name="hello.c">
+ hello.c
+ </file>
+ <directory name="cache">
+ </directory>
+ <file name="not_used" printme="1">
+ CacheDir('/usr/local/build_cache')
+ </file>
+ </scons_example>
+
+ <para>
+
+ Note that the directory you specify must already exist
+ and be readable and writable by all developers
+ who will be sharing derived files.
+ It should also be in some central location
+ that all builds will be able to access.
+ In environments where developers are using separate systems
+ (like individual workstations) for builds,
+ this directory would typically be
+ on a shared or NFS-mounted file system.
+
+ </para>
+
+ <para>
+
+ Here's what happens:
+ When a build has a &CacheDir; specified,
+ every time a file is built,
+ it is stored in the shared cache directory
+ along with its MD5 build signature.
+ <footnote>
+ <para>
+ Actually, the MD5 signature is used as the name of the file
+ in the shared cache directory in which the contents are stored.
+ </para>
+ </footnote>
+ On subsequent builds,
+ before an action is invoked to build a file,
+ &SCons; will check the shared cache directory
+ to see if a file with the exact same build
+ signature already exists.
+ If so, the derived file will not be built locally,
+ but will be copied into the local build directory
+ from the shared cache directory,
+ like so:
+
+ </para>
+
+ <scons_output example="ex1">
+ <scons_output_command>scons -Q</scons_output_command>
+ <scons_output_command>scons -Q -c</scons_output_command>
+ <scons_output_command>scons -Q</scons_output_command>
+ </scons_output>
+
+ <para>
+
+ Note that the &CacheDir; feature still calculates
+ MD5 build sigantures for the shared cache file names
+ even if you configure &SCons; to use timestamps
+ to decide if files are up to date.
+ (See the <xref linkend="chap-depends"></xref>
+ chapter for information about the &Decider; function.)
+ Consequently, using &CacheDir; may reduce or eliminate any
+ potential performance improvements
+ from using timestamps for up-to-date decisions.
+
+ </para>
+
+ </section>
+
+ <section>
+ <title>Keeping Build Output Consistent</title>
+
+ <para>
+
+ One potential drawback to using a shared cache
+ is that the output printed by &SCons;
+ can be inconsistent from invocation to invocation,
+ because any given file may be rebuilt one time
+ and retrieved from the shared cache the next time.
+ This can make analyzing build output more difficult,
+ especially for automated scripts that
+ expect consistent output each time.
+
+ </para>
+
+ <para>
+
+ If, however, you use the <literal>--cache-show</literal> option,
+ &SCons; will print the command line that it
+ <emphasis>would</emphasis> have executed
+ to build the file,
+ even when it is retrieving the file from the shared cache.
+ This makes the build output consistent
+ every time the build is run:
+
+ </para>
+
+ <scons_output example="ex1">
+ <scons_output_command>scons -Q</scons_output_command>
+ <scons_output_command>scons -Q -c</scons_output_command>
+ <scons_output_command>scons -Q --cache-show</scons_output_command>
+ </scons_output>
+
+ <para>
+
+ The trade-off, of course, is that you no longer
+ know whether or not &SCons;
+ has retrieved a derived file from cache
+ or has rebuilt it locally.
+
+ </para>
+
+ </section>
+
+ <section>
+ <title>Not Using the Shared Cache for Specific Files</title>
+
+ <para>
+
+ You may want to disable caching for certain
+ specific files in your configuration.
+ For example, if you only want to put
+ executable files in a central cache,
+ but not the intermediate object files,
+ you can use the &NoCache;
+ function to specify that the
+ object files should not be cached:
+
+ </para>
+
+ <scons_example name="ex-NoCache">
+ <file name="SConstruct" printme="1">
+ env = Environment()
+ obj = env.Object('hello.c')
+ env.Program('hello.c')
+ CacheDir('cache')
+ NoCache('hello.o')
+ </file>
+ <file name="hello.c">
+ hello.c
+ </file>
+ <directory name="cache">
+ </directory>
+ </scons_example>
+
+ <para>
+
+ Then when you run &scons; after cleaning
+ the built targets,
+ it will recompile the object file locally
+ (since it doesn't exist in the shared cache directory),
+ but still realize that the shared cache directory
+ contains an up-to-date executable program
+ that can be retrieved instead of re-linking:
+
+ </para>
+
+ <!--
+
+ <scons_output example="ex1">
+ <scons_output_command>scons -Q</scons_output_command>
+ <scons_output_command>scons -Q -c</scons_output_command>
+ <scons_output_command>scons -Q</scons_output_command>
+ </scons_output>
+
+ -->
+
+ <screen>
+ % <userinput>scons -Q</userinput>
+ cc -o hello.o -c hello.c
+ cc -o hello hello.o
+ % <userinput>scons -Q -c</userinput>
+ Removed hello.o
+ Removed hello
+ % <userinput>scons -Q</userinput>
+ cc -o hello.o -c hello.c
+ Retrieved `hello' from cache
+ </screen>
+
+ </section>
+
+ <section>
+ <title>Disabling the Shared Cache</title>
+
+ <para>
+
+ Retrieving an already-built file
+ from the shared cache
+ is usually a significant time-savings
+ over rebuilding the file,
+ but how much of a savings
+ (or even whether it saves time at all)
+ can depend a great deal on your
+ system or network configuration.
+ For example, retrieving cached files
+ from a busy server over a busy network
+ might end up being slower than
+ rebuilding the files locally.
+
+ </para>
+
+ <para>
+
+ In these cases, you can specify
+ the <literal>--cache-disable</literal>
+ command-line option to tell &SCons;
+ to not retrieve already-built files from the
+ shared cache directory:
+
+ </para>
+
+ <scons_output example="ex1">
+ <scons_output_command>scons -Q</scons_output_command>
+ <scons_output_command>scons -Q -c</scons_output_command>
+ <scons_output_command>scons -Q</scons_output_command>
+ <scons_output_command>scons -Q -c</scons_output_command>
+ <scons_output_command>scons -Q --cache-disable</scons_output_command>
+ </scons_output>
+
+ </section>
+
+ <section>
+ <title>Populating a Shared Cache With Already-Built Files</title>
+
+ <para>
+
+ Sometimes, you may have one or more derived files
+ already built in your local build tree
+ that you wish to make available to other people doing builds.
+ For example, you may find it more effective to perform
+ integration builds with the cache disabled
+ (per the previous section)
+ and only populate the shared cache directory
+ with the built files after the integration build
+ has completed successfully.
+ This way, the cache will only get filled up
+ with derived files that are part of a complete, successful build
+ not with files that might be later overwritten
+ while you debug integration problems.
+
+ </para>
+
+ <para>
+
+ In this case, you can use the
+ the <literal>--cache-force</literal> option
+ to tell &SCons; to put all derived files in the cache,
+ even if the files already exist in your local tree
+ from having been built by a previous invocation:
+
+ </para>
+
+ <scons_output example="ex1">
+ <scons_output_command>scons -Q --cache-disable</scons_output_command>
+ <scons_output_command>scons -Q -c</scons_output_command>
+ <scons_output_command>scons -Q --cache-disable</scons_output_command>
+ <scons_output_command>scons -Q --cache-force</scons_output_command>
+ <scons_output_command>scons -Q</scons_output_command>
+ </scons_output>
+
+ <para>
+
+ Notice how the above sample run
+ demonstrates that the <literal>--cache-disable</literal>
+ option avoids putting the built
+ <filename>hello.o</filename>
+ and
+ <filename>hello</filename> files in the cache,
+ but after using the <literal>--cache-force</literal> option,
+ the files have been put in the cache
+ for the next invocation to retrieve.
+
+ </para>
+
+ </section>
+
+ <section>
+ <title>Minimizing Cache Contention: the <literal>--random</literal> Option</title>
+
+ <para>
+
+ If you allow multiple builds to update the
+ shared cache directory simultaneously,
+ two builds that occur at the same time
+ can sometimes start "racing"
+ with one another to build the same files
+ in the same order.
+ If, for example,
+ you are linking multiple files into an executable program:
+
+ </para>
+
+ <scons_example name="ex-random">
+ <file name="SConstruct" printme="1">
+ Program('prog',
+ ['f1.c', 'f2.c', 'f3.c', 'f4.c', 'f5.c'])
+ </file>
+ <file name="f1.c">f1.c</file>
+ <file name="f2.c">f2.c</file>
+ <file name="f3.c">f3.c</file>
+ <file name="f4.c">f4.c</file>
+ <file name="f5.c">f5.c</file>
+ <file name="f6.c">f6.c</file>
+ </scons_example>
+
+ <para>
+
+ &SCons; will normally build the input object files
+ on which the program depends in their normal, sorted order:
+
+ </para>
+
+ <scons_output example="ex-random">
+ <scons_output_command>scons -Q</scons_output_command>
+ </scons_output>
+
+ <para>
+
+ But if two such builds take place simultaneously,
+ they may each look in the cache at nearly the same
+ time and both decide that <filename>f1.o</filename>
+ must be rebuilt and pushed into the shared cache directory,
+ then both decide that <filename>f2.o</filename>
+ must be rebuilt (and pushed into the shared cache directory),
+ then both decide that <filename>f3.o</filename>
+ must be rebuilt...
+ This won't cause any actual build problems--both
+ builds will succeed,
+ generate correct output files,
+ and populate the cache--but
+ it does represent wasted effort.
+
+ </para>
+
+ <para>
+
+ To alleviate such contention for the cache,
+ you can use the <literal>--random</literal> command-line option
+ to tell &SCons; to build dependencies
+ in a random order:
+
+ </para>
+
+ <!--
+
+ The following <screen> output was generated by this:
+
+ <scons_output example="ex-random">
+ <scons_output_command>scons -Q - -random</scons_output_command>
+ </scons_output>
+
+ We captured it directly here to guarantee a "random" order,
+ guarding against the potential for - -random to happen
+ to return things in the original sorted order.
+
+ -->
+
+ <screen>
+ % <userinput>scons -Q --random</userinput>
+ cc -o f3.o -c f3.c
+ cc -o f1.o -c f1.c
+ cc -o f5.o -c f5.c
+ cc -o f2.o -c f2.c
+ cc -o f4.o -c f4.c
+ cc -o prog f1.o f2.o f3.o f4.o f5.o
+ </screen>
+
+ <para>
+
+ Multiple builds using the <literal>--random</literal> option
+ will usually build their dependencies in different,
+ random orders,
+ which minimizes the chances for a lot of
+ contention for same-named files
+ in the shared cache directory.
+ Multiple simultaneous builds might still race to try to build
+ the same target file on occasion,
+ but long sequences of inefficient contention
+ should be rare.
+
+ </para>
+
+ <para>
+
+ Note, of course,
+ the <literal>--random</literal> option
+ will cause the output that &SCons; prints
+ to be inconsistent from invocation to invocation,
+ which may be an issue when
+ trying to compare output from different build runs.
+
+ </para>
+
+ <para>
+
+ If you want to make sure dependencies will be built
+ in a random order without having to specify
+ the <literal>--random</literal> on very command line,
+ you can use the &SetOption; function to
+ set the <literal>random</literal> option
+ within any &SConscript; file:
+
+ </para>
+
+ <scons_example name="ex-random">
+ <file name="SConstruct" printme="1">
+ SetOption('random', 1)
+ Program('prog',
+ ['f1.c', 'f2.c', 'f3.c', 'f4.c', 'f5.c'])
+ </file>
+ <file name="f1.c">f1.c</file>
+ <file name="f2.c">f2.c</file>
+ <file name="f3.c">f3.c</file>
+ <file name="f4.c">f4.c</file>
+ <file name="f5.c">f5.c</file>
+ <file name="f6.c">f6.c</file>
+ </scons_example>
+
+ </section>
+
+ <!--
+
+ <section>
+ <title>Troubleshooting Shared Caching: the &cache-debug; Option</title>
+
+ <para>
+
+ XXX describe the - - cache-debug option
+ XXX maybe point to the troubleshooting appendix?
+
+ </para>
+
+ </section>
+
+ -->
+
+ <!--
+
+ <section>
+
+ <para>
+
+ XXX describe CacheDir management: monitoring, deleting, etc.
+
+ </para>
+
+ </section>
+
+ -->