GNU Make Unleashed release

For 4 years I've written the Ask Mr Make column over at CM Crossroads (and I continue to write it). Since there's been great interest in the column, I've put together all 4 years of columns plus additional unpublished material as a book and ebook.

All the material has been rechecked for accuracy, errata have been incorporated and the text re-edited. The result is a 230 page book covering everything from basics of GNU Make to advanced topics like eliminating recursive make, doing arithmetic in GNU Make or dealing with spaces in file names.





The book contains 43 separate articles about GNU Make, plus a complete reference to the GNU Make Standard Library.

You can buy a copy in either form here.

A big thank you to everyone who's commented, emailed, or made suggestions on my GNU Make articles over the years.

Escaping comma and space in GNU Make

Sometimes you need to hide a comma or a space from GNU Make's parser because GNU Make might strip it (if it's a space) or interpret it as an argument separator (for example, in a function invocation).

First the problem. If you wanted to change every , into a ; in a string in GNU Make you'd probably head for the $(subst) function and do the following:

    $(subst ,,;,$(string))

See the problem? The argument separator for functions in GNU Make is , and hence the first , (the search text) is considered to be separator. Hence the search text in the above is actually the empty string, the replacement text is also the empty string and the ;, is just preprended to whatever is in $(string).

A similar problem occurs with spaces. Suppose you want to replace all spaces with ; in a string. You get a similar problem with $(subst), this time because the leading space is stripped:

    $(subst  ,;,$(string))

That extra space isn't an argument it's just extraneous whitespace and hence it is ignored. GNU Make just ends up appending ; to the $(string).

So, how do you solve this?

The answer is define variables that contain just a comma and just a space and use them. Because the argument splitting is done before variable expansion it's possible to have an argument that's a comma or a space.

For a comma you just do:

    comma := ,
    $(subst $(comma),;,$(string))

And everything works.

For a space you need to get a space into a string, I find the easiest way is like this:

    space :=
    space +=
    $(subst $(space),;,$(string))

That works because += always space separates the value of the variable with the appended text.

Now, GNU Make has really liberal variable naming rules. Pretty much anything goes, so it's possible to define a variable with the name , or even having the name consisting of a space character.

First, here's how to define them:

    , := ,
    space :=
    space +=
    $(space) := 
    $(space) +=

The first line is clear, it does an immediate define of a , to the variable named ,. The second one is a little more complex. First, I define a variable called space which contains a space character and then I use it to define a variable whose name is that space character.

You can verify that these work using $(warning) (I like to wrap the variable being printed in square brackets for absolute certainty of the content):

    $(warning [$(,)])
    $(warning [$( )])

$ make
Makefile:1: [,]
Makefile:2: [ ]

Yes, that's pretty odd looking, but it gets stranger. Since GNU Make will interpret $ followed by a single character as a variable expansion you can drop the braces and write:

    $(warning [$,])
    $(warning [$ ])

Now that starts to look like escaping. In the examples above you can use these variables to make things a little clearer:

    $(subst $(,),;,$(string))
    $(subst $ ,;,$(string))

Note that you have to use the $(,) form because function argument splitting occurs before the expansion and GNU Make gets confused. In the second line the space is 'escaped' with the $ sign.

You might be wondering about other crazy variable names: here are a few that's possible with GNU Make:

# Defining the $= or $(=) variable which has the value =
equals := =
$(equals) := =

# Define the $# or $(#) variable which has the value #
hash := \#
$(hash) := \#

# Define the $: or $(:) variable which has the value :
colon := :
$(colon) := :

# Define the $($$) variable which has the value $
dollar := $$
$(dollar) := $$

; := ;
% := %

You probably don't need any of those, but you never know...

Introducing Usman's Law

Back at Electric Cloud I worked with a smart guy named Usman Muzaffar. As part of his job he spent a lot of time dealing with our customers, many of whom used GNU Make other other Make tools to build their software.

One of the constant problems that Usman encountered was that most people had no way to get back to a truly clean build. No matter what they'd put in place for doing a totally scratch, clean build it was hard for everyone because their process often accidentally ommitted to delete something.

I've observed this problem in my own code. Like many people I have a 'make clean' option which deletes all the output files: in my case by rm -rfing an obj directory:

.PHONY: clean
clean:
 @rm -rf $(OUT)/*
 @$(MAKE_DIRECTORIES)

And I make sure that generated things only go under $(OUT). But it's easy to screw up. Consider a program like yacc or bison which'll create temporary source code files in the same place as the source code being analyzed. The truth is you have to be very careful to ensure that everything goes in one deletable place. (Not to mention the difficulties involved if the Makefile output different versions of objects for, say, different processor targets or platforms).

That leads me to Usman's Law: make clean doesn't.

Live by it and you'll be on the look out for poorly coded Makefiles that leave generated files in places they should not.

Electric Cloud wins a Jolt Productivity Award

Back in 2005 POPFile (which is now in desperate need of an updated version) won a Productivity Award at the 15th Annual Jolt awards. This week the company I co-founded, Electric Cloud, won the exact same award for its product ElectricCommander.

OK, I should stop bragging now.

And show a little humility.

Truth be told, the glow from the second award is strictly reflected... I didn't design, code, or do anything to make ElectricCommander :-) But being a company founder is a good thing; you get to pretend you had all the smart ideas.

What Makefile am I in?

A common request when using GNU Make is: "Is there a way to find the name and path of the current Makefile?". By 'current' people usually mean that Makefile that GNU Make is currently parsing. There's no built-in way to quickly get the answer, but there is a way using the GNU Make variable MAKEFILE_LIST.

MAKEFILE_LIST (documented in the manual here) is the list of Makefiles currently loaded or included. Each time a Makefile is loaded or included the variable is appended. The paths and names in the variable are relative to the current working directory (where GNU Make was started or where it moved to with the -C or --directory option). The current working directory is stored in the CURDIR variable.

So you can quite easily define a GNU Make function (let's call it where-am-i) that will return the current Makefile (it uses $(word) to get the last Makefile name from the list):

where-am-i = $(CURDIR)/$(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST))

then whenever you want to find out the full path to the current Makefile write the following at the top of the Makefile (the 'at the top' part is important because any include statement in the Makefile will change the value of MAKEFILE_LIST so you want to grab the location of the current Makefile right at the top):

THIS_MAKEFILE := $(call where-am-i)

Example:

Here's Makefile

where-am-i = $(CURDIR)/$(word ($words $(MAKEFILE_LIST)),$(MAKEFILE_LIST)

include foo/Makefile

foo/Makefile contains:

THIS_MAKEFILE := $(call where-am-i)
$(warning $(THIS_MAKEFILE))

include foo/bar/Makefile

foo/bar/Makefile contains:

THIS_MAKEFILE := $(call where-am-i)
$(warning $(THIS_MAKEFILE))

Running this on my machine (with the first Makefile in /tmp) gives the output:

foo/Makefile:2: /tmp/foo/Makefile
foo/bar/Makefile:2: /tmp/foo/bar/Makefile

Bug fix: 12 Tasty Make Recipes Part II

Back in May, 2006 I gave a talk called 12 Tasty Make Recipes Part II in which I talked about a user-defined GNU Make function to recursively search from a directory for a file or set of files.

The function was written like this:

search = $(foreach d,$(wildcard $1/*),$(call search,$d)$(filter $(subst *,%,$2),$d))

Unfortunately, there was a small mistake in the code that appeared in the presentation. However, two bugs collided to cause the mistake to have no effect. In the above function I wrote $(call search,$d) when I should have written $(call search,$d,$2). But since GNU Make had a bug that caused it to not reset $2 (or any other arguments in the form $n) on a nested $(call) and since I was reusing $2 as the second argument, my function worked.

That is, until GNU Make 3.81 was released and fixed bug 1744.

The correct version of search which will work with all versions of GNU Make is:

search = $(foreach d,$(wildcard $1/*),$(call search,$d,$2)$(filter $(subst *,%,$2),$d))

Thanks for Lou Iacoponi for writing in and pointing out my error.

Double quote bug fix for signature

A while ago I wrote about Shoehorening Keep State into GNU Make. A reader has identified a bug in the code that I presented that causes the code to keep regenerating a target even though the commands have not changed.

This bug occurs if the commands for a target have a double-quote in them. For example, if you have something like:

foo.o:
@compiler -D_DEBUG="YES" foo.c

foo.o will be regenerated every time the Makefile is run. This is because the keep state code echoes the current state of the command to a file and echo strips the double-quotes around YES.

The fix is simple: escape ". Here's the updated code for signature with the fix in it:

include gmsl

last_target :=

dump_var = \$$(eval $1 := $($1))

define new_rule
@echo "$(call map,dump_var,@ % < ? ^ + *)" > $S
@$(if $(wildcard $F),,touch $F)
@echo $@: $F >> $S
endef

define do
$(eval S := $*.sig)$(eval F := $*.force)$(eval C := $(strip $1))
$(if $(call sne,$@,$(last_target)),$(call new_rule),$(eval last_target := $@))
@echo "$(subst ",\",$(subst $$,\$$,$$(if $$(call sne,$(strip $1),$C),$$(shell touch $F))))" >> $S
$C
endef

Free GNU Make documentation

If, like me, you use GNU Make a lot then you should be aware of two really important pieces of documentation for GNU Make that are totally free (speech and beer):

1. The GNU Make Manual. This is the standard manual that comes with GNU Make and is available on the web here: http://www.gnu.org/software/make/manual/make.html

2. Robert Mecklenburg's "Managing Projects with GNU Make". This is the book published by O'Reilly but released under the Free Documentation License. PDFs of each of the sections are available here: http://www.oreilly.com/catalog/make3/book/index.csp

It's great that these two resources are freely available, but don't let that stop you buying them. Supporting the FSF and Mecklenburg with a little cash is a good way of keeping free documents free.

Would you buy a "GNU Make Cookbook" e-book?

I've been thinking about taking all the recipes for GNU Make things that I've written over the years as articles, or blog entries, or answers to people's questions on help-make and writing them up as an e-book for purchase and download from this web site.

Here's a sample recipe in PDF format so that you can see what I'm taking about.



So, the critical questions:

1. Would you buy such a book?
2. If so, how much would you pay for it?
3. What format would be best? PDF?

John.

A small bug fix to my keep state shoehorning

A while back I wrote about a way to shoehorn Sun Make's "keep state" functionality into GNU Make. With a fairly simple Makefile it's possible to get GNU Make to rebuild targets when the targets' commands have changed. I blogged this here and wrote it up for Ask Mr Make here.

One reader was having trouble with the system because every single Make he did caused a certain target to be built. It turned out this was because he'd done something like:

target:
    $(call do, commands)

The space after the , and before the commands was messing up my signature system's comparison and causing it to think that the commands changed every time. This is easily fixed by stripping the commands.

Here's the updated signature file (with the changed parts highlighted in blue):

include gmsl

last_target := 

dump_var = \$$(eval $1 := $($1))

define new_rule
@echo "$(call map,dump_var,@ % < ? ^ + *)" > $S
@$(if $(wildcard $F),,touch $F)
@echo $@: $F >> $S
endef

define do
$(eval S := $*.sig)$(eval F := $*.force)$(eval C := $(strip $1))
$(if $(call sne,$@,$(last_target)),$(call new_rule),$(eval last_target := $@))
@echo "$(subst $$,\$$,$$(if $$(call sne,$(strip $1),$C),$$(shell touch $F)))" >> $S
$C
endef

Another common thing people have asked for is that the signature system rebuild targets when the Makefile has changed. Currently the signature system cannot spot an edit to the Makefile that changes the commands. It's pretty simple to Make this happen (although this will cause all targets to be built when the Makefile is updated) by adding the following line in new_rule above:

define new_rule
@echo "$(call map,dump_var,@ % < ? ^ + *)" > $S
@$(if $(wildcard $F),,touch $F)
@echo $@: $F >> $S
@echo $F: Makefile >> $S
endef

It's an exercise or the reader to replace Makefile with the actual name of the Makefile that is including active when new_rule is called.

Rebuilding when the hash has changed, not the timestamp

GNU Make decides whether to rebuild a file based on whether any of its prerequisites are newer or if the file is missing. But sometimes this isn't desirable: when using GNU Make with a source code control system the time on a prerequisite might be updated by the source code system when the files are checked out, even though the file itself hasn't changed.

It's desirable, in fact, to change GNU Make to check a hash of the file contents and only rebuild if the file has actually changed (and ignore the timestamp).

You can hack this into GNU Make using md5sum (I'm assuming you're on a system with Unix-like commands). Here's a little example that builds foo.o from foo.c but only updates foo.o when foo.h has changed... and changed means that its checksum has changed:

   .PHONY: all

   to-md5 = $(patsubst %,%.md5,$1)
   from-md5 = $(patsubst %.md5,%,$1)

   all: foo.o

   foo.o: foo.c
   foo.o: $(call to-md5,foo.h)

   %.md5: FORCE
       @$(if $(filter-out $(shell cat $@ 2>/dev/null),
       $(shell md5sum $*)),md5sum $* > $@)

   FORCE:

This works because when foo.h was mentioned in the prerequisite list of foo.o it was changed to foo.h.md5 by the to-md5 function. So GNU Make sees the prerequisites of foo.o to be foo.c and foo.h.md5.

Then there's a pattern rule to build foo.h.md5 (the %.md5 rule) that will only update the .md5 file if the checksum has changed. Thus if and only if the checksum has changed does the .md5 file get changed and foo.o rebuilt.

The %.md5 rule is forced to run by having a dummy prereq called FORCE so that every MD5 hash is checked for every prerequisite that GNU Make needs to examine.

First the rule uses a filter-out/if combination to check to see if the MD5 hash has changed. If it has then the %.md5 rule will run md5sum $* > $@ (in the example md5sum foo.h > foo.h.md5). This will both update the hash in the file and change the .md5 file's timestamp and force foo.o to build.

If within the rule for foo.o $?, $^ or other automatics that work on the prerequisite list were used these need to be passed through from-md5 to remove the .md5 extension so that the real prerequisite is used in the commands to build foo.o.

In the example this isn't necessary.

If the foo.h.md5 file does not exist then the %.md5 rule will create it and force foo.o to get built.

You can also adapt this tip to work with different definitions of 'changed'. For example, the .md5 file could store the version number of a file from the source control system and rebuilds would only happen when the version had changed.

Shoehorning Keep State into GNU Make

Sun Make has a lovely feature called Keep State: if the commands used to build a target change from build to build the target is rebuilt, even if looking at file time stamps shows that the target is "up to date". Why is this a lovely feature? Because it means that make followed by make DEBUG=1 will do that right thing. In Make's that only check time stamps the make DEBUG=1 would probably report that there was no work to do.

Of course, you can get round these problems if you really try (e.g. for the DEBUG case you could encode the fact that the objects are debug objects in either the name or path and then Make would do the right thing).

A recent post on the GNU Make mailing list got me thinking about this problem again and I've come up with a very simple solution that shoehorns Keep State into GNU Make. There's no code change to GNU Make at all; it's all done with existing GNU Make functions.

Here's an example Makefile that I've modified to rebuild foo.o and bar.o if their commands change

    include signature

    all: foo.o bar.o

    FOO=$(FOOEY)

    foo.o: foo.c
        $(call do,$$(COMPILE.C) -DFOO=$$(FOO)$$(@F) -o $$@ $$<)

    bar.o: bar.c
        $(call do,$$(COMPILE.C) -DBAR=$$(BAR) -o $$@ $$<)

    -include foo.sig bar.sig

There are three modifications from a standard Makefile: firstly there's 'include signature' at the start. (You'll see the definition of signature below), then the commands for each rule have been wrapped in $(call do,...) and any $'s in the commands have been quoted with an extra $. Lastly the Makefile includes a .sig file for each .o being created (if the .sig exists, hence the -include instead of include).

The .sig file is generated by code in signature when a rule is run and is used to perform the 'command has changed' checking that you need. Here, for example, is the contents of bar.sig after make has been run for the first time:

    $(eval @ := bar.o)
    $(eval % := )
    $(eval < := bar.c)
    $(eval ? := bar.c)
    $(eval ^ := bar.c)
    $(eval + := bar.c)
    $(eval * := bar)

    bar.o: bar.force

    $(if $(call sne,$(COMPILE.C) -DBAR=$(BAR) -o $@ $<,
    g++    -c -DBAR= -o bar.o bar.c),$(shell
    touch bar.force))

The first set of lines captures the state of the automatic variables within the rule to make bar.o, the next line says that bar.o depends on a special file called bar.force and lastly there's a rather complex $(if ...) that uses the GMSL (see GNU Make Standard Library) string-not-equal (sne) function to check the current expansion of the commands to make bar.o against the previous expansion. It's this $(if ...) that can detect a change in the commands to run a rule. If such a change is detected bar.force is touched and hence bar.o will be rebuilt because bar.force is newer.

The signature include is where the work is done:

    include gmsl

    last_target :=

    dump_var = \$$(eval $1 := $($1))

    define new_rule
    @echo "$(call map,dump_var,@ % < ? ^ + *)" > $S
    @$(if $(wildcard $F),,touch $F)
    @echo $@: $F >> $S
    endef

    define do
    $(eval S := $*.sig)$(eval F := $*.force)$(eval C := $1)
    $(if $(call sne,$@,$(last_target)),$(call new_rule),$(eval
    last_target := $@))
    @echo "$(subst $$,\$$,$$(if $$(call sne,$1,$C),
    $$(shell touch $F)))" >> $S
    $C
    endef

I won't go into all the details of how signature works, but essentially the do macro is responsible for updating the .sig files as needed. I'll write this up for my column on CM Crossroads in March, but you can play around with the code (you need the GMSL and GNU Make 3.80 for this to work) and you'll see that changing a parameter does work.

Here's an example of starting from scratch and then changing the values of FOO and BAR in the Makefile above:

    $ make
    g++    -c -DFOO=foo.o -o foo.o foo.c
    g++    -c -DBAR= -o bar.o bar.c
    $ make
    make: Nothing to be done for `all'.
    $ make BAR=bar
    g++    -c -DBAR=bar -o bar.o bar.c
    $ make BAR=bar
    make: Nothing to be done for `all'.
    $ make BAR=baz
    g++    -c -DBAR=baz -o bar.o bar.c
    $ make BAR=baz FOO=foo
    g++    -c -DFOO=foofoo.o -o foo.o foo.c
    $ make BAR=bar FOO=foo
    g++    -c -DBAR=bar -o bar.o bar.c
    $ make
    g++    -c -DFOO=foo.o -o foo.o foo.c
    g++    -c -DBAR= -o bar.o bar.c

The only limitation of this scheme is that if you change the commands in a rule by editing the Makefile you need to do a clean build or at least delete the corresponding .sig file so that it gets remade. (Of course, even that could be worked around by making foo.o and bar.o depend on Makefile)

GMSL gets set functions

In preparing for some changes to the GNU Make Debugger I decided that it would be helpful to have sets as a fundamental type and so I've added set manipulation functions to the GNU Make Standard Library; those functions will be released with the next version (1.0.3) of the library.

Adding the functions turned out to be pretty trivial with the right representation: a set in the GMSL is a deduplicated, sorted list and GNU Make's existing $(sort) function deduplicates and sorts a list. The library now has the empty set, functions for set union and intersection and tests to determine if one set is the subset of another or if an element is present in a set.

As with the rest of the GMSL there are tests to cover all the newly added functions and updated documentation is on the SourceForge site.

GNU Make Debugger released

As part of the consulting work I've been doing for Electric Cloud I created a simple interactive debugger for GNU Make Makefiles. It's written using GNU Make's internal functions and supports breakpoints and interactive querying of variables.

I originally wrote about the debugger for CM Crossroads in this article, and then did a follow up webinar for Electric Cloud.

But now the debugger has been released under the General Public License on SourceForge. Anyone interestedin getting or modifying the debugger should visit the SourceForge GMD site.

In its current state the debugger is pretty simple. You can set breakpoints manually by inserting $(__BREAKPOINT) in a Makefile or in a rule, but what's really needed are breakpoints that can be set interactively.

It's possible to do this by maintaining a set of target names that have breakpoints enabled on them and then by modifying SHELL the debugger could watch for the execution of those rules. There are a number of usability challenges though: there's no way to query the set of rules that currently exist in a Makefile and hence no way to offer target name completion. That means that entering the target name could be really tricy; the best option is probably to allow wildcards so that only a partial name needs to be entered.