Dienstag, 15. Oktober 2013

Build specific targets of a sub-makefile (inspired by Gruntjs)

I recently had a great idea:
When I was using gruntjs a couple of months ago, I was introduced to the possibility of sub-targets there. Those were specified using colons (":") to seperate parent target and subtarget. Since I am using dedicated sub-makefiles for single modules in my project, I just had the brilliant idea: realize something similar to access goals from those. This saves me the trouble of specifying the file used and to define necessary environment variables by hand.

So what I came up with initially and what worked out quite well, was this:
# idea from http://blog.jgc.org/2007/06/escaping-comma-and-space-in-gnu-make.html 
# I put this in my dedicated defs.mk file for general usage
 ,             := ,
 space         :=
 space         +=
 $(space)      := 
 $(space)      +=
   
 $(PROJECT)\:%:  
      $(eval SUBGOALS := $(subst $(,),$( ),$*))  
      $(PRINT) making $(PROJECT) with goal\(s\) $(SUBGOALS)...  
      $(MAKE) -f "$(abspath $(ROOTDIR)/Make/$(PROJECT).mk)" $(SUBGOALS)  
      $(PRINT) done making $(PROJECT) with goal\(s\) $(SUBGOALS)  

Here is an example, of what the benefit of this is:

$: make modulexy:libxy,binxy
making modulexy with goal(s) libxy binxy...

There is one problem, though: While you CAN run makefile goals in any recursion level
(e.g. $: make goal1:goal2:goal3: ...)
only the first level allows you to pass the goals together to a single instance of make. Things like
"$: make goal1:goal2:goal3.1,goal3.2"
will be executed as
"make -f goal1file goal2:goal3.1 goal3.2"
That means, the command tries to run goal3.2 in goal1file (the file belonging to goal1).
The workaround would be
"make goal1:goal2:goal3.1,goal2:goal3.2"
which is then causing two instances of make being run on the makefile belonging to goal2: One for goal3.1 and one for goal 3.2.

So far, I did not feel the need to improve this (by adding a way to evaluate brackets, for example. Imagine "$: make goal1:goal2:(goal3.1,goal3.2)"). But if you are in a desperate need of this, it will be hard to achieve with make-internal tools only. sed will probably be your friend, here.
Contact me, if you come to a solution, so I can add it here.

See you soon!

Donnerstag, 10. Oktober 2013

GNU Make as templating engine (???)

Hello visitors!

So this is already my second post about GNU Make I am writing within two weeks! My intention this time is to describe how you can get all the power of the Makefile-parser into any usual, build-related file.

To give you a better image of what I wanted to achieve originally and why, let me give you this scenario:
Imagine you are starting to work on yet another c/c++ program for GNU and you are trying to use as much basic project stuff as possible from your previous project. You are copying over makefiles, licenses, other textfiles and launching scripts. After that, you alter the name of the last project everywhere to match your new one's - which is pretty inconvenient if done by hand. You know: These project specific strings occuring in standard files all over the place should be kept at only one place, namely your build system's configuration file.

In my last article I was calling this build system configuration file for make "defs.mk". Now the question is: How will we get the definitions from there into our text files?

With the help of the standard *nix tools "cat", "sed" and "printf" I managed to write a small copy-over routine for make which performs make's variable expansion on the text files we want to reuse in the future. Here is the code:

 DOCFILES        = $(DOCSRCDIR)/LICENSE.txt $(DOCSRCDIR)/README.txt  
 DOCDESTS    = $(subst $(DOCSRCDIR),$(DISTDIR),$(DOCFILES))  
 .PHONY    = all  
 all: $(DOCDESTS)  
 $(DISTDIR)/%: $(DOCSRCDIR)/%  
 # Get the file's contents into "DOCCONTENT". Don't forget to escape "#"  
 # by using sed -e 's/\#/\\&/g' and also get rid of troublesome newlines  
 # (replaced with \n which is re-translated by printf later)  
 # http://stackoverflow.com/questions/1251999/sed-how-can-i-replace-a-newline-n  
     @$(eval DOCCONTENT := $(shell cat $^ | sed -e ':a;N;$$!ba;{s/\#/\\&/g;s/\n/\\n/g}'))  
     @printf "$(DOCCONTENT)" > $@  

By reading the document into the makefile variable "DOCCONTENT" all currently set variables are being expanded (replaced) in the script before we write it back to its destination. DISTDIR is hereby our distributiable package folder.

Inside the document we are also able to use make's inbuilt functions, such as $(wildcard and even $(shell. This is a bit risky though, since almost everything can be done this way. At least noone will complain about lacking features, though :)

I hope this can be helpful for you!
Regards

Dienstag, 1. Oktober 2013

My idea of how make should be done

Welcome dear reader!

When I start spending time with a new programming language, I usually want to get a clear picture first of how my workflow should look like. I simply want to do it right from the beginning, using state-of-the-art tools and profiting from others' experiences, so I will be able to work efficiently right away and so that my project evolves around a good structure.

Currently, I am trying out C++ (not for the first time, but this time for real), denying myself using an IDE which would get me started quickly but rendering me entirely planless of what is going on behind the scenes.
Naturally, I was confronted right at the beginning with the decision which build system to use. I started out using the best known one, gnu make, and got comfortable with it. Actually it was even so good, that when I decided to go for cmake instead (which seems to be next to autotools the only  relativley widespread build tool out there), I couldn't tell any improvements over make.

So here I want to give you an overview of the conclusions I've drawn about how to use makefiles in a project-independent, yet absolutely clean manner:

  1. of all: The project's top-level structure:
    I wanted a clean top-level project directory with nothing in it but introductory documentation (license, readme...), one folder for source (src) and one to build into (build). Lateron, looking into other projects, I also saw that being able to offer support for several other (maybe ide-specific) build systems would be great, too, so I decided to give each one of them a separate directory - "Make" doing the first step here.
    That leads us getting something like:
    • project root
      • build
      • src
      • Make
      • ...
      • XCode/Eclipse etc.
    This way we can also strictly keep the build-tool related files from our code, as another positive sideeffect.

  2. how to even make it work:
    Now that we have decided to put all our makefiles into the Make directory, how will make get to see the files it has to build together? It is not even in the project's root directory. That's why we need to get this root directory first - the parent of the folder, our makefile resides in:
     ROOTDIR            = $(realpath $(dir $(realpath $(dir $(lastword $(MAKEFILE_LIST))))))  
    
    I am using here, that calling "realpath" on a directory name will cause the original path to be stripped of trailing slashes. Therefore, another call of "dir" on it will give me the parent directory. I know, this is somewhat hacky, but we just won't mind.
    So next thing is to export our ROOTDIR variable, so that all other makefiles can use it et voilà: Accessing specific directories far away from our "Make" directory won't be a problem any more.

  3. how to reuse our makefiles:
    I far as I could see, it is very common to extend a build system as the actual project grows with the time. The downside of this procedure is obvious: Makefiles written this way tend to be project specific and can hardly be reused. So we need ways to separate project specific attributes from the reoccuring parts of a makefile.
    First thing I'd recommend here is a dedicated defs.mk makefile containing all or most of your definitions. This way you will have all your project-dependent stuff just there and this will be easy to understand for external collaborators. Note that my make_boilerplate contains a nice draft of such a defs.mk file which you can use and modify as you like to.
    In the future, we will just include our definition file to gain access to our whole project's configuration:
     include $(ROOTDIR)/Make/defs.mk  
    
    Yes, I do know about the "MAKEFILES" environment variable, but I think, this way it is just more transparent. So let's just do it this way.

  4. recursive makefiles(?):
    Coming from Java WITH IDEs, I had a pretty detailed vision of how I wanted my project to be structured. Over all, folders play a central role in dividing pieces of code into single units of related functionality. But how to tell make that those source files, distributed over several hierarchies of folders, have to be built into one single binary?

    Recursive makefiles - while seeming obvious at first (icu, for example, is doing it this way) - didn't comply with my ideas stated above, since they had to lie in the source directory. Additionally, it relatively quickly turned out, that recursive use of makefiles is actually not desireable. Click here for details.

    So I went for good old "find" to get my .cpp files which I would then turn into a list of .o files:
     CXXFILES       = $(shell find $(SRCDIR) -type f -name '*.cpp')  
     CXXOBJECTS     = $(patsubst $(SRCDIR)/%.cpp,$(OBJDIR)/module/%.o,$(CXXFILES))  
    
    Those, as you can see, I put into their respective subdirectory in my object build directory.

  5. where to divide makefiles:
    When working with makefiles in a project, you often have them grow according to your needs while you are concentrating on your code. Therefore one often ends up having huge and unreadable makefiles that are likely to break on minimal changes. I suggest the following here: Have one makefile for each big part of your project. This will not contradict our earlier statement "recursive make considered harmful", since the makefile is still as self-contained as possible and should not call any sub-makes either. We just use the modularity of our project to have clean cuts. I suggest one makefile for every lib or binary of your own project, maybe one makefile for each of your dependencies if they have to be built from source or one single makefile together for all your prebuilt libraries.
    Standard targets that can be "outsourced" include:
    • deps.mk (for dependencies)
    • tests.mk
    • <projectname>.mk for your project

  6. handling indirect dependencies:
    We know that make works using a file's dependencies to create the file itself, if those are newer. Therefore we have to specify the dependencies, which we already automated. But we completely left out the fact, that header files included in our .cpp files are also dependencies.
    In order to be able to respond to changes made to only those included files, we will use a trick which I borrowed from here: We exploit the -M option (for gcc and similar), which gives us all files included by a codefile and write them into a seperate makefile, which we will include from now on. These secondary makefiles we will be giving the suffix ".d", as in "dependency". So the rule for .o files will be looking like 
     $(OBJDIR)/%.o: $(SRCDIR)/%.cpp  
         @mkdir -p $(dir $@)  
         @$(COMPILE.cxx) $@ $^  
         @# Create the .d file using gcc's -M or -MM option  
         $(CXX) $(CPPFLAGS) $(CXXFLAGS) -MM $(lastword $^) > $(patsubst %.o,%.d,$@)  
    
    Additionally, we will have to retrieve the existing .d files at the beginning of our makefile and include them:
     DEPFILES        = $(CXXOBJECTS:.o=.d)  
     -include $(DEPFILES)  
    
    The "-" in front of include turns off errors on file-not-found. And that's it!
    Make sure though that you include the DEPFILES only after the "all" target, so some object file won't become your default target.

  7. coping with different levels of verbosity:
    In general it has to be said, that make's philosophy can be a bit obstructive when it comes to communicating what is going on. First of all, there is no parse-time option to output messages to the console. Trying it with approaches like "$(shell echo some message)" will fail because make' s goal here is it to grab the shell command's output. So you will not be able to just dump all variable values of interest on make startup before a target is built. You will rather have to have an own "info" target doing that, which is prerequisite of your "all" target or you are calling yourself.

    The next problem is, that when you have a simple target that just aggregates other tasks, and you want to announce its start and its end, at least the first of those two cannot be achieved. For example, imagine you want to make sure, all build directories are set up correctly in target "directories: dir1 dir2 dir3". Where will you put the "Start creating directories" message? You will have to live with that limitation or invent a smart way to avoid this.

    To have control over what is printed and when, I suggest setting up variables like "PRINT" (echoing always) and "VPRINT" (echo if in verbose mode, otherwise just /bin/true). This can be extended to "VVPRINT" (VERBOSE = 2) etc. if necessary.
    Additionally, the make variable "MAKECMDGOALS" can be helpful to detect if the user ran make with the intention of getting more information as usual about the build process. I, for example, did the following right at the beginning of my main Makefile: 
     ifeq ($(filter info, $(MAKECMDGOALS)),info)
         VERBOSE = YES
         export VERBOSE # for all sub-makes
     endif
    
On these thoughts as a basis, I set up a small github repository I called "make_boilerplate". You will find it by following https://github.com/suluke/make_boilerplate . Maybe things will also be clearer if you take a look into the makefiles themselves.

I've been reading lots and lots of articles and stackoverflow questions on the internet to get all this done in a way I personally can live with. I don't guarantee you, though, that it is bugfree or even trapless. I see a high chance that some professional with 40 years of experience shows up and tells me about a bunch of flaws my makefile design comes with. Personally I don't see any problems so far though - and this is why I posted this. I hope, you will find it helpful, too.

Regards

suluke


Sources: