manual.asciidoc

Ninja

Introduction

Ninja is yet another build system. It takes as input the interdependencies of files (typically source code and output executables) and orchestrates building them, quickly.

Ninja joins a sea of other build systems. Its distinguishing goal is to be fast. It is born from my work on the Chromium browser project, which has over 30,000 source files and whose other build systems (including one built from custom non-recursive Makefiles) can take ten seconds to start building after changing one file. Ninja is under a second.

Philosophical overview

Build systems get slow when they need to make decisions. When you are in a edit-compile cycle you want it to be as fast as possible — you want the build system to do the minimum work necessary to figure out what needs to be built immediately.

Ninja contains the barest functionality necessary to describe arbitrary dependency graphs. Its lack of syntax makes it impossible to express complex decisions. Instead, Ninja is intended to be used with a separate program generating its input files. The generator program (like the ./configure found in autotools projects) can analyze system dependencies and make as many decisions as possible up front so that incremental builds stay fast. Going beyond autotools, even build-time decisions like "which compiler flags should I use?" or "should I build a debug or release-mode binary?" belong in the .ninja file generator.

Conceptual overview

Ninja evaluates a graph of dependencies between files, and runs whichever commands are necessary to make your build target up to date. If you are familiar with Make, Ninja is very similar.

A build file (default name: build.ninja) provides a list of rules — short names for longer commands, like how to run the compiler — along with a list of build statements saying how to build files using the rules — which rule to apply to which inputs to produce which outputs.

Conceptually, build statements describe the dependency graph of your project, while rule statements describe how to generate the files along a given edge of the graph.

Design goals

Here are some of the design goals of Ninja:

• very fast (i.e., instant) incremental builds, even for very large projects.

• very little policy about how code is built; "explicit is better than implicit".

• get dependencies correct, and in particular situations that are difficult to get right with Makefiles (e.g. outputs need an implicit dependency on the command line used to generate them; to build C source code you need to use gcc’s -M flags for header dependencies).

• when convenience and speed are in conflict, prefer speed.

Some explicit non-goals:

• convenient syntax for writing build files by hand. You should generate your ninja files using another program. This is how we can sidestep many policy decisions.

• built-in rules. Out of the box, Ninja has no rules for e.g. compiling C code.

• build-time customization of the build. Options belong in the program that generates the ninja files.

• build-time decision-making ability such as conditionals or search paths. Making decisions is slow.

To restate, Ninja is faster than other build systems because it is painfully simple. You must tell Ninja exactly what to do when you create your project’s .ninja files.

Comparison to GNU make

Ninja is closest in spirit and functionality to make. But fundamentally, make has a lot of features: suffix rules, functions, built-in rules that e.g. search for RCS files when building source. Many projects find make alone adequate for their build problems. In contrast, Ninja has almost no features; just those necessary to get builds correct while punting most complexity to generation of the ninja input files. Ninja by itself is unlikely to be useful for most projects.

Here are some of the features ninja adds to make. (These sorts of features can often be implemented using more complicated Makefiles, but they are not part of make itself.)

• A Ninja rule may point at a path for extra implicit dependency information. This makes it easy to get header dependencies correct for C/C++ code.

• A build edge may have multiple outputs.

• Outputs implicitly depend on the command line that was used to generate them, which means that changing e.g. compilation flags will cause the outputs to rebuild.

• Output directories are always implicitly created before running the command that relies on them.

• Rules can provide shorter descriptions of the command being run, so you can print e.g. CC foo.o instead of a long command line while building.

• Command output is always buffered. This means commands running in parallel don’t interleave their output, and when a command fails we can print its failure output next to the full command line that produced the failure.

Getting started

The included bootstrap.sh should hopefully produce a working ninja binary, by first blindly compiling all non-test files together then re-building Ninja using itself.

Usage is currently just

ninja target

where target is a known output described by build.ninja in the current directory.

There is no installation step; the only files of interest to a user are the resulting binary and this manual.

Creating .ninja files

Here’s a basic .ninja file that demonstrates most of the syntax. It will be used as an example for the following sections.

cflags = -Wall

rule cc
  command = gcc $cflags -c $in -o $out

build foo.o: cc foo.c

Variables

Despite the non-goal of being convenient to write by hand, to keep build files readable (debuggable), Ninja supports declaring shorter reusable names for strings. A declaration like the following

cflags = -g

can be used on the right side of an equals sign, dereferencing it with a dollar sign, like this:

rule cc
  command = gcc $cflags -c $in -o $out

Variables can also be referenced using curly braces like ${in}.

Variables might better be called "bindings", in that a given variable cannot be changed, only shadowed. There is more on how shadowing works later in this document.

Rules

Rules declare a short name for a command line. They begin with a line consisting of the rule keyword and a name for the rule. Then follows an indented set of variable = value lines.

The basic example above declares a new rule named cc, along with the command to run. (In the context of a rule, the command variable is special and defines the command to run. A full list of special variables is provided in the reference.)

Within the context of a rule, two additional special variables are available: $in expands to the list of input files (foo.c) and $out to the output file (foo.o) for the command.

Build statements

Build statements declare a relationship between input and output files. They begin with the build keyword, and have the format build outputs: rulename inputs. Such a declaration says that all of the output files are derived from the input files. When the output files are missing or when the inputs change, Ninja will run the rule to regenerate the outputs.

The basic example above describes how to build foo.o, using the cc rule.

In the scope of a build block (including in the evaluation of its associated rule), the variable $in is the list of inputs and the variable $out is the list of outputs.

A build statement may be followed by an indented set of key = value pairs, much like a rule. These variables will shadow any variables when evaluating the variables in the command. For example:

cflags = -Wall -Werror
rule cc
  command = gcc $cflags -c $in -o $out

# If left unspecified, builds get the outer $cflags.
build foo.o: cc foo.c

# But you can can shadow variables like cflags for a particular build.
build special.o: cc special.c
  cflags = -Wall

# The variable was only shadowed for the scope of special.o;
# Subsequent build lines get the outer (original) cflags.
build bar.o: cc bar.c

For more discussion of how scoping works, consult the reference.

If you need more complicated information passed from the build statement to the rule (for example, if the rule needs "the file extension of the first input"), pass that through as an extra variable, like how cflags is passed above.

The phony rule

The special rule name phony can be used to create aliases for other targets. For example:

build all: phony some/file/in/a/faraway/subdir

This makes ninja all build the other files. Semantically, the phony rule is equivalent to a plain rule where the command does nothing, but phony rules are handled specially in that they aren’t printed when run, logged (see below), nor do they contribute to the command count printed as part of the build process.

The Ninja log

For each built file, Ninja keeps a log of the command used to build it. Using this log Ninja can know when an existing output was built with a different command line than the build files specify (i.e., the command line changed) and knows to rebuild the file.

The log file is kept in the build root in a file called .ninja_log. If you provide a variable named builddir in the outermost scope, .ninja_log will be kept in that directory instead.

Generating Ninja files

A work-in-progress patch to gyp, the system used to generate build files for the Chromium browser to generate ninja files for Linux is included in the source distribution.

Conceptually, you could coax Automake into producing ninja files as well, but I haven’t tried it. It may very well be the case that most projects use too much Makefile syntax in their .am files for this to work.

Extra tools

The -t flag on the Ninja command line runs some tools that I have found useful during Ninja’s development. The current tools are:

query

dump the inputs and outputs of a given target.

browse

browse the dependency graph in a web browser. Clicking a file focuses the view on that file, showing inputs and outputs. This feature requires a Python installation.

graph

output a file in the syntax used by graphviz, a automatic graph layout tool. Use it like: ninja -t graph target | dot -Tpng -ograph.png /dev/stdin . In the Ninja source tree, ninja graph generates an image for Ninja itself. If no target is given generate a graph for all root targets.

targets

output a list of targets either by rule or by depth. If used like this ninja -t targets rule name it prints the list of targets using the given rule to be built. If no rule is given, it prints the source files (the leaf of the graph). If used like this ninja -t targets depth digit it prints the list of targets in a depth-first manner starting by the root targets (the ones with no outputs). Indentation is used to mark dependencies. If the depth is zero it prints all targets. If no arguments are provided ninja -t targets depth 1 is assumed. In this mode targets may be listed several times. If used like this ninja -t targets all it prints all the targets available without indentation and it is way faster than the depth mode. It returns non-zero if an error occurs.

rules

output the list of all rules with their description if they have one. It can be used to know which rule name to pass to ninja -t targets rule name.

clean

remove built files. If used like this ninja -t clean it removes all the built files. If used like this ninja -t clean targets…​ or like this ninja -t clean target targets…​ it removes the given targets and recursively all files built for it. If used like this ninja -t clean rule rules it removes all files built using the given rules. The depfiles are not removed. Files created but not referenced in the graph are not removed. This tool takes in account the -v and the -n options (note that -n implies -v). It returns non-zero if an error occurs.

Ninja file reference

A file is a series of declarations. A declaration can be one of:

1. A rule declaration, which begins with rule rulename, and then has a series of indented lines defining variables.

2. A build edge, which looks like build output1 output2: rulename input1 input2.
   Implicit dependencies may be tacked on the end with | dependency1 dependency2. Order-only dependencies may be tacked on the end with || dependency1 dependency2.

3. Variable declarations, which look like variable = value.

4. References to more files, which look like subninja path or include path. The difference between these is explained below in the discussion about scoping.

Comments begin with # and extend to the end of the line.

Newlines are significant, but they can be escaped by putting a \ before them.

Other whitespace is only significant if it’s at the beginning of a line. If a line is intended more than the previous one, it’s considered part of its parent’s scope; if it is indented less than the previous one, it closes the previous scope.

Rule variables

A rule block contains a list of key = value declarations that affect the processing of the rule. Here is a full list of special keys.

command (required)

the command line to run. This string (after $variables are expanded) is passed directly to sh -c without interpretation by Ninja.

depfile

path to an optional Makefile that contains extra implicit dependencies (see the the reference on dependency types). This is explicitly to support gcc and its -M family of flags, which output the list of headers a given .c file depends on.

Use it like in the following example:

rule cc
  depfile = $out.d
  command = gcc -MMD -MF $out.d [other gcc flags here]

description

a short description of the command, used to pretty-print the command as it’s running. The -v flag controls whether to print the full command or its description; if a command fails, the full command line will always be printed before the command’s output.

Additionally, the special $in and $out variables expand to the space-separated list of files provided to the build line referencing this rule.

Build dependencies

There are three types of build dependencies which are subtly different.

1. Explicit dependencies, as listed in a build line. These are available as the $in variable in the rule. Changes in these files cause the output to be rebuilt; if these file are missing and Ninja doesn’t know how to build them, the build is aborted.

   This is the standard form of dependency to be used for e.g. the source file of a compile command.

2. Implicit dependencies, either as picked up from a depfile attribute on a rule or from the syntax | dep1 dep2 on the end of a build line. Changes in these files cause the output to be rebuilt; if they are missing, they are just skipped.

   This is for expressing dependencies that don’t show up on the command line of the command; for example, for a rule that runs a script, the script itself should be an implicit dependency, as changes to the script should cause the output to rebuild.

3. Order-only dependencies, expressed with the syntax || dep1 dep2 on the end of a build line. When these are missing, the output is not rebuilt until they are built, but once they are available further changes to the files do not affect the output.

   Order-only dependencies can be useful for bootstrapping dependencies that are only discovered during build time: for example, to generate a header file before starting a subsequent compilation step. (Once the header is used in compilation, a generated dependency file will then express the implicit dependency.)

Evaluation and scoping

Top-level variable declarations are scoped to the file they occur in.

The subninja keyword, used to include another .ninja file, introduces a new scope. The included subninja file may use the variables from the parent file, and shadow their values for the file’s scope, but it won’t affect values of the variables in the parent.

To include another .ninja file in the current scope, much like a C #include statement, use include instead of subninja.

Variable declarations indented in a build block are scoped to the build block. This scope is inherited by the rule. The full lookup order for a variable referenced in a rule is:

1. Rule-level variables (i.e. $in, $command).

2. Build-level variables from the build that references this rule.

3. File-level variables from the file that the build line was in.

4. Variables from the file that included that file using the subninja keyword.

Future work

Some pieces I’d like to add:

inotify. I had originally intended to make Ninja be memory-resident and to use inotify to keep the build state hot at all times. But upon writing the code I found it was fast enough to run from scratch each time. Perhaps a slower computer would still benefit from inotify; the data structures are set up such that using inotify shouldn’t be hard.

build estimation and analysis. As part of build correctness, Ninja keeps a log of the time spent on each build statement. This log format could be adjusted to instead store timing information across multiple runs. From that, the total time necessary to build could be estimated, allowing Ninja to print status like "3 minutes until complete" when building. Additionally, a tool could output which commands are the slowest or which directories take the most time to build.

many others. See the todo file included in the distribution.