From 72c578fd4b0b4a5a43e18594339ac4ff26c376dc Mon Sep 17 00:00:00 2001 From: Luca Falavigna Date: Sat, 2 Jan 2010 20:56:27 +0100 Subject: Imported Upstream version 1.2.0.d20091224 --- doc/user/caching.xml | 506 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 506 insertions(+) create mode 100644 doc/user/caching.xml (limited to 'doc/user/caching.xml') diff --git a/doc/user/caching.xml b/doc/user/caching.xml new file mode 100644 index 0000000..073539c --- /dev/null +++ b/doc/user/caching.xml @@ -0,0 +1,506 @@ + + + + + On multi-developer software projects, + you can sometimes speed up every developer's builds a lot by + allowing them to share the derived files that they build. + &SCons; makes this easy, as well as reliable. + + + +
+ Specifying the Shared Cache Directory + + + + To enable sharing of derived files, + use the &CacheDir; function + in any &SConscript; file: + + + + + CacheDir('/usr/local/build_cache') + + + + + Note that the directory you specify must already exist + and be readable and writable by all developers + who will be sharing derived files. + It should also be in some central location + that all builds will be able to access. + In environments where developers are using separate systems + (like individual workstations) for builds, + this directory would typically be + on a shared or NFS-mounted file system. + + + + + + Here's what happens: + When a build has a &CacheDir; specified, + every time a file is built, + it is stored in the shared cache directory + along with its MD5 build signature. + + + Actually, the MD5 signature is used as the name of the file + in the shared cache directory in which the contents are stored. + + + On subsequent builds, + before an action is invoked to build a file, + &SCons; will check the shared cache directory + to see if a file with the exact same build + signature already exists. + If so, the derived file will not be built locally, + but will be copied into the local build directory + from the shared cache directory, + like so: + + + + + % scons -Q + cc -o hello.o -c hello.c + cc -o hello hello.o + % scons -Q -c + Removed hello.o + Removed hello + % scons -Q + Retrieved `hello.o' from cache + Retrieved `hello' from cache + + + + + Note that the &CacheDir; feature still calculates + MD5 build sigantures for the shared cache file names + even if you configure &SCons; to use timestamps + to decide if files are up to date. + (See the + chapter for information about the &Decider; function.) + Consequently, using &CacheDir; may reduce or eliminate any + potential performance improvements + from using timestamps for up-to-date decisions. + + + +
+ +
+ Keeping Build Output Consistent + + + + One potential drawback to using a shared cache + is that the output printed by &SCons; + can be inconsistent from invocation to invocation, + because any given file may be rebuilt one time + and retrieved from the shared cache the next time. + This can make analyzing build output more difficult, + especially for automated scripts that + expect consistent output each time. + + + + + + If, however, you use the --cache-show option, + &SCons; will print the command line that it + would have executed + to build the file, + even when it is retrieving the file from the shared cache. + This makes the build output consistent + every time the build is run: + + + + + % scons -Q + cc -o hello.o -c hello.c + cc -o hello hello.o + % scons -Q -c + Removed hello.o + Removed hello + % scons -Q --cache-show + cc -o hello.o -c hello.c + cc -o hello hello.o + + + + + The trade-off, of course, is that you no longer + know whether or not &SCons; + has retrieved a derived file from cache + or has rebuilt it locally. + + + +
+ +
+ Not Using the Shared Cache for Specific Files + + + + You may want to disable caching for certain + specific files in your configuration. + For example, if you only want to put + executable files in a central cache, + but not the intermediate object files, + you can use the &NoCache; + function to specify that the + object files should not be cached: + + + + + env = Environment() + obj = env.Object('hello.c') + env.Program('hello.c') + CacheDir('cache') + NoCache('hello.o') + + + + + Then when you run &scons; after cleaning + the built targets, + it will recompile the object file locally + (since it doesn't exist in the shared cache directory), + but still realize that the shared cache directory + contains an up-to-date executable program + that can be retrieved instead of re-linking: + + + + + + + % scons -Q + cc -o hello.o -c hello.c + cc -o hello hello.o + % scons -Q -c + Removed hello.o + Removed hello + % scons -Q + cc -o hello.o -c hello.c + Retrieved `hello' from cache + + +
+ +
+ Disabling the Shared Cache + + + + Retrieving an already-built file + from the shared cache + is usually a significant time-savings + over rebuilding the file, + but how much of a savings + (or even whether it saves time at all) + can depend a great deal on your + system or network configuration. + For example, retrieving cached files + from a busy server over a busy network + might end up being slower than + rebuilding the files locally. + + + + + + In these cases, you can specify + the --cache-disable + command-line option to tell &SCons; + to not retrieve already-built files from the + shared cache directory: + + + + + % scons -Q + cc -o hello.o -c hello.c + cc -o hello hello.o + % scons -Q -c + Removed hello.o + Removed hello + % scons -Q + Retrieved `hello.o' from cache + Retrieved `hello' from cache + % scons -Q -c + Removed hello.o + Removed hello + % scons -Q --cache-disable + cc -o hello.o -c hello.c + cc -o hello hello.o + + +
+ +
+ Populating a Shared Cache With Already-Built Files + + + + Sometimes, you may have one or more derived files + already built in your local build tree + that you wish to make available to other people doing builds. + For example, you may find it more effective to perform + integration builds with the cache disabled + (per the previous section) + and only populate the shared cache directory + with the built files after the integration build + has completed successfully. + This way, the cache will only get filled up + with derived files that are part of a complete, successful build + not with files that might be later overwritten + while you debug integration problems. + + + + + + In this case, you can use the + the --cache-force option + to tell &SCons; to put all derived files in the cache, + even if the files already exist in your local tree + from having been built by a previous invocation: + + + + + % scons -Q --cache-disable + cc -o hello.o -c hello.c + cc -o hello hello.o + % scons -Q -c + Removed hello.o + Removed hello + % scons -Q --cache-disable + cc -o hello.o -c hello.c + cc -o hello hello.o + % scons -Q --cache-force + scons: `.' is up to date. + % scons -Q + scons: `.' is up to date. + + + + + Notice how the above sample run + demonstrates that the --cache-disable + option avoids putting the built + hello.o + and + hello files in the cache, + but after using the --cache-force option, + the files have been put in the cache + for the next invocation to retrieve. + + + +
+ +
+ Minimizing Cache Contention: the <literal>--random</literal> Option + + + + If you allow multiple builds to update the + shared cache directory simultaneously, + two builds that occur at the same time + can sometimes start "racing" + with one another to build the same files + in the same order. + If, for example, + you are linking multiple files into an executable program: + + + + + Program('prog', + ['f1.c', 'f2.c', 'f3.c', 'f4.c', 'f5.c']) + + + + + &SCons; will normally build the input object files + on which the program depends in their normal, sorted order: + + + + + % scons -Q + cc -o f1.o -c f1.c + cc -o f2.o -c f2.c + cc -o f3.o -c f3.c + cc -o f4.o -c f4.c + cc -o f5.o -c f5.c + cc -o prog f1.o f2.o f3.o f4.o f5.o + + + + + But if two such builds take place simultaneously, + they may each look in the cache at nearly the same + time and both decide that f1.o + must be rebuilt and pushed into the shared cache directory, + then both decide that f2.o + must be rebuilt (and pushed into the shared cache directory), + then both decide that f3.o + must be rebuilt... + This won't cause any actual build problems--both + builds will succeed, + generate correct output files, + and populate the cache--but + it does represent wasted effort. + + + + + + To alleviate such contention for the cache, + you can use the --random command-line option + to tell &SCons; to build dependencies + in a random order: + + + + + + + % scons -Q --random + cc -o f3.o -c f3.c + cc -o f1.o -c f1.c + cc -o f5.o -c f5.c + cc -o f2.o -c f2.c + cc -o f4.o -c f4.c + cc -o prog f1.o f2.o f3.o f4.o f5.o + + + + + Multiple builds using the --random option + will usually build their dependencies in different, + random orders, + which minimizes the chances for a lot of + contention for same-named files + in the shared cache directory. + Multiple simultaneous builds might still race to try to build + the same target file on occasion, + but long sequences of inefficient contention + should be rare. + + + + + + Note, of course, + the --random option + will cause the output that &SCons; prints + to be inconsistent from invocation to invocation, + which may be an issue when + trying to compare output from different build runs. + + + + + + If you want to make sure dependencies will be built + in a random order without having to specify + the --random on very command line, + you can use the &SetOption; function to + set the random option + within any &SConscript; file: + + + + + Program('prog', + ['f1.c', 'f2.c', 'f3.c', 'f4.c', 'f5.c']) + + SetOption('random', 1) + Program('prog', + ['f1.c', 'f2.c', 'f3.c', 'f4.c', 'f5.c']) + + +
+ + + + -- cgit v1.2.3