path: root/doc/concepts/rules.md

User-defined Rules
==================

Targets are defined in terms of high-level concepts like "libraries",
"binaries", etc. In order to translate these high-level definitions
into actionable tasks, the user defines rules, explaining at a single
point how all targets of a given type are built.

Rules files
-----------

Rules are defined in rules files (by default named `RULES`). Those
contain a JSON object mapping rule names to their rule definition. For
rules, the same naming scheme as for targets applies. However, built-in
rules (always named by a single string) take precedence in naming; to
explicitly refer to a rule defined in the current module, the module has
to be specified, possibly by a relative path, e.g.,
`["./", ".", "install"]`.

Basic components of a rule
--------------------------

A rule is defined through a JSON object with various keys. The only
mandatory key is `"expression"` containing the defining expression of
the rule.

### `"config_fields"`, `"string_fields"` and `"target_fields"`

These keys specify the fields that a target defined by that rule can
have. In particular, those have to be disjoint lists of strings.

For `"config_fields"` and `"string_fields"` the respective field has to
evaluate to a list of strings, whereas `"target_fields"` have to
evaluate to a list of target references. Those references are evaluated
immediately, and in the name context of the target they occur in.

The difference between `"config_fields"` and `"string_fields"` is that
`"config_fields"` are evaluated before the target fields and hence can
be used by the rule to specify config transitions for the target fields.
`"string_fields"` on the other hand are evaluated *after*
the target fields; hence the rule cannot use them to specify a
configuration transition, however the target definition in those fields
may use the `"outs"` and `"runfiles"` functions to have access to the
names of the artifacts or runfiles of a target specified in one of the
target fields.

### `"implicit"`

This key specifies a map of implicit dependencies. The keys of the map
are additional target fields, the values are the fixed list of targets
for those fields. If a short-form name of a target is used (e.g., only a
string instead of a module-target pair), it is interpreted relative to
the repository and module the rule is defined in, not the one the rule
is used in. Other than this, those fields are evaluated the same way as
target fields settable on invocation of the rule.

### `"config_vars"`

This is a list of strings specifying which parts of the configuration
the rule uses. The defining expression of the rule is evaluated in an
environment that is the configuration restricted to those variables; if
one of those variables is not specified in the configuration the value
in the restriction is `null`.

### `"config_transitions"`

This key specifies a map of (some of) the target fields (whether
declared as `"target_fields"` or as `"implicit"`) to a configuration
expression. Here, a configuration expression is any expression in our
language. It has access to the `"config_vars"` and the `"config_fields"`
and has to evaluate to a list of maps. Each map specifies a transition
to the current configuration by amending it on the domain of that map to
the given value.

### `"imports"`

This specifies a map of expressions that can later be used by
`CALL_EXPRESSION`. In this way, duplication of (rule) code can be
avoided. For each key, we have to have a name of an expression;
expressions are named following the same naming scheme as targets and
rules. The names are resolved in the context of the rule. Expressions
themselves are defined in expression files, the default name being
`EXPRESSIONS`.

Each expression is a JSON object. The only mandatory key is
`"expression"` which has to be an expression in our language. It
optionally can have a key `"vars"` where the value has to be a list of
strings (and the default is the empty list). Additionally, it can have
another optional key `"imports"` following the same scheme as the
`"imports"` key of a rule; in the `"imports"` key of an expression,
names are resolved in the context of that expression. It is a
requirement that the `"imports"` graph be cycle free.

### `"expression"`

This specifies the defining expression of the rule. The value has to be
an expression of our expression language (basically, an abstract syntax
tree serialized as JSON). It has access to the following extra functions
and, when evaluated, has to return a result value.

#### `FIELD`

The field function takes one argument, `name` which has to evaluate
to the name of a field. For string fields, the given list of strings
is returned; for target fields, the list of abstract names for the
given target is returned. These abstract names are opaque within the
rule language (but meaningful when reported in error messages) and
should only be used to be passed on to other functions that expect
names as inputs.

#### `DEP_ARTIFACTS` and `DEP_RUNFILES`

These functions give access to the artifacts, or runfiles,
respectively, of one of the targets depended upon. It takes two
(evaluated) arguments, the mandatory `"dep"` and the optional
`"transition"`.

The argument `"dep"` has to evaluate to an abstract name (as can be
obtained from the `FIELD` function) of some target specified in one
of the target fields. The `"transition"` argument has to evaluate to
a configuration transition (i.e., a map) and the empty transition is
taken as default. It is an error to request a target-transition pair
for a target that was not requested in the given transition through
one of the target fields.

#### `DEP_PROVIDES`

This function gives access to a particular entry of the provides map
of one of the targets depended upon. The arguments `"dep"` and
`"transition"` are as for `DEP_ARTIFACTS`; additionally, there is
the mandatory argument `"provider"` which has to evaluate to a
string. The function returns the value of the provides map of the
target at the given provider. If the key is not in the provides map
(or the value at that key is `null`), the optional argument
`"default"` is evaluated and returned. The default for `"default"`
is the empty list.

#### `BLOB`

The `BLOB` function takes a single (evaluated) argument `data` which
is optional and defaults to the empty string. This argument has to
evaluate to a string. The function returns an artifact that is a
non-executable file with the given string as content.

#### `TREE`

The `TREE` function takes a single (evaluated) argument `$1` which
has to be a map of artifacts. The result is a single tree artifact
formed from the input map. It is an error if the map cannot be
transformed into a tree (e.g., due to staging conflicts).

### `TREE_OVERLAY` and `DISJOINT_TREE_OVERLAY`

The `TREE_OVERLAY` and `DISJOINT_TREE_OVERLAY` functions take a
single (evaluated) argument `$1` which has to be a list of maps
of artifacts. Each entry in the list is transformed into a single
tree artifact formed from that map and it is an error if the map
cannot be transformed into a tree (e.g., due to a staging conflict).
The result is a single tree artifact defined as the overlay of the
specified trees in that order (latest wins on each path after build
evaluation); in the case of `DISJOINT_TREE_OVERLAY` it is a build
error if a non-trivial conflict arises when computing the overlay
in the build phase.

#### `ACTION`

Actions are a way to define new artifacts from (zero or more)
already defined artifacts by running a command, typically a
compiler, linker, archiver, etc. The action function takes the
following arguments.

 - `"inputs"` A map of artifacts. These artifacts are present when
   the command is executed; the keys of the map are the relative
   path from the working directory of the command. The command must
   not make any assumption about the location of the working
   directory in the file system (and instead should refer to files
   by path relative to the working directory). Moreover, the
   command must not modify the input files in any way. (In-place
   operations can be simulated by staging, as is shown in the
   example later in this document.)

   It is an additional requirement that no conflicts occur when
   interpreting the keys as paths. For example, `"foo.txt"` and
   `"./foo.txt"` are different as strings and hence legitimately
   can be assigned different values in a map. When interpreted as a
   path, however, they name the same path; so, if the `"inputs"`
   map contains both those keys, the corresponding values have to
   be equal.

 - `"cmd"` The command to execute, given as `argv` vector, i.e., a
   non-empty list of strings. The 0'th element of that list will
   also be the program to be executed.

 - `"cwd"` The directory inside the action root to change to before
   executing the command. The directory has to be given as a string
   describing a non-upwards relative path. This field is optional
   and defaults to `""`.

 - `"env"` The environment in which the command should be executed,
   given as a map of strings to strings.

 - `"outs"` and `"out_dirs"` Two list of strings naming the files
   and directories, respectively, the command is expected to
   create. Those paths are interpreted relative to the action root
   (not relative to `"cwd"`).
   It is an error if the command fails to create the
   promised output files. These two lists have to be disjoint, but
   an entry of `"outs"` may well name a location inside one of the
   `"out_dirs"`.

This function returns a map with keys the strings mentioned in
`"outs"` and `"out_dirs"`. As values this map has artifacts defined
to be the ones created by running the given command (in the given
environment with the given inputs).

#### `RESULT`

The `RESULT` function is the only way to obtain a result value. It
takes three (evaluated) arguments, `"artifacts"`, `"runfiles"`, and
`"provides"`, all of which are optional and default to the empty
map. It defines the result of a target that has the given artifacts,
runfiles, and provided data, respectively. In particular,
`"artifacts"` and `"runfiles"` have to be maps to artifacts, and
`"provides"` has to be a map. Moreover, the keys in `"runfiles"`
and `"artifacts"` are treated as paths; it is an error if this
interpretation yields to conflicts. The keys in the artifacts or
runfile maps as seen by other targets are the normalized paths of
the keys given.

Result values themselves are opaque in our expression language and
cannot be deconstructed in any way. Their only purpose is to be the
result of the evaluation of the defining expression of a target.

#### `CALL_EXPRESSION`

This function takes one mandatory argument `"name"` which is
unevaluated; it has to a be a string literal. The expression
imported by that name through the imports field is evaluated in the
current environment restricted to the variables of that expression.
The result of that evaluation is the result of the `CALL_EXPRESSION`
statement.

During the evaluation of an expression, rule fields can still be
accessed through the functions `FIELD`, `DEP_ARTIFACTS`, etc. In
particular, even an expression with no variables (that, hence, is
always evaluated in the empty environment) can carry out non-trivial
computations and be non-constant. The special functions `BLOB`,
`ACTION`, and `RESULT` are also available. If inside the evaluation
of an expression the function `CALL_EXPRESSION` is used, the name
argument refers to the `"imports"` map of that expression. So the
call graph is deliberately recursion free.

Evaluation of a target
----------------------

A target defined by a user-defined rule is evaluated in the following
way.

 - First, the config fields are evaluated.

 - Then, the target-fields are evaluated. This happens for each field
   as follows.

    - The configuration transition for this field is evaluated and the
      transitioned configurations determined.
    - The argument expression for this field is evaluated. The result
      is interpreted as a list of target names. Each of those targets
      is analyzed in all the specified configurations.

 - The string fields are evaluated. If the expression for a string
   field queries a target (via `outs` or `runfiles`), the value for
   that target is returned in the first configuration. The rational
   here is that such generator expressions are intended to refer to the
   corresponding target in its "main" configuration; they are hardly
   used anyway for fields branching their targets over many
   configurations.

 - The effective configuration for the target is determined. The target
   effectively has used of the configuration the variables used by the
   `arguments_config` in the rule invocation, the `config_vars` the
   rule specified, and the parts of the configuration used by a target
   dependent upon. For a target dependent upon, all parts it used of
   its configuration are relevant expect for those fixed by the
   configuration transition.

 - The rule expression is evaluated and the result of that evaluation
   is the result of the rule.

Example of developing a rule
----------------------------

Let's consider step by step an example of writing a rule. Say we want
to write a rule that programmatically patches some files.

### Framework: The minimal rule

Every rule has to have a defining expression evaluating to a `RESULT`.
So the minimally correct rule is the `"null"` rule in the following
example rule file.

    { "null": {"expression": {"type": "RESULT"}}}

This rule accepts no parameters, and has the empty map as artifacts,
runfiles, and provided data. So it is not very useful.

### String inputs

Let's allow the target definition to have some fields. The most simple
fields are `string_fields`; they are given by a list of strings. In the
defining expression we can access them directly via the `FIELD`
function. Strings can be used when defining maps, but we can also create
artifacts from them, using the `BLOB` function. To create a map, we can
use the `singleton_map` function. We define values step by step, using
the `let*` construct.

``` jsonc
{ "script only":
  { "string_fields": ["script"]
  , "expression":
    { "type": "let*"
    , "bindings":
      [ [ "script content"
        , { "type": "join"
          , "separator": "\n"
          , "$1":
            { "type": "++"
            , "$1":
              [["H"], {"type": "FIELD", "name": "script"}, ["w", "q", ""]]
            }
          }
        ]
      , [ "script"
        , { "type": "singleton_map"
          , "key": "script.ed"
          , "value":
            {"type": "BLOB", "data": {"type": "var", "name": "script content"}}
          }
        ]
      ]
    , "body":
      {"type": "RESULT", "artifacts": {"type": "var", "name": "script"}}
    }
  }
}
```

### Target inputs and derived artifacts

Now it is time to add the input files. Source files are targets like any
other target (and happen to contain precisely one artifact). So we add a
target field `"srcs"` for the file to be patched. Here we have to keep
in mind that, on the one hand, target fields accept a list of targets
and, on the other hand, the artifacts of a target are a whole map. We
chose to patch all the artifacts of all given `"srcs"` targets. We can
iterate over lists with `foreach` and maps with `foreach_map`.

Next, we have to keep in mind that targets may place their artifacts at
arbitrary logical locations. For us that means that first we have to
make a decision at which logical locations we want to place the output
artifacts. As one thinks of patching as an in-place operation, we chose
to logically place the outputs where the inputs have been. Of course, we
do not modify the input files in any way; after all, we have to define a
mathematical function computing the output artifacts, not a collection
of side effects. With that choice of logical artifact placement, we have
to decide what to do if two (or more) input targets place their
artifacts at logically the same location. We could simply take a
"latest wins" semantics (keep in mind that target fields give a list
of targets, not a set) as provided by the `map_union` function. We chose
to consider it a user error if targets with conflicting artifacts are
specified. This is provided by the `disjoint_map_union` that also allows
to specify an error message to be provided the user. Here, conflict
means that values for the same map position are defined in a different
way.

The actual patching is done by an `ACTION`. We have the script already;
to make things easy, we stage the input to a fixed place and also expect
a fixed output location. Then the actual command is a simple shell
script. The only thing we have to keep in mind is that we want useful
output precisely if the action fails. Also note that, while we define
our actions sequentially, they will be executed in parallel, as none of
them depends on the output of another one of them.

``` jsonc
{ "ed patch":
  { "string_fields": ["script"]
  , "target_fields": ["srcs"]
  , "expression":
    { "type": "let*"
    , "bindings":
      [ [ "script content"
        , { "type": "join"
          , "separator": "\n"
          , "$1":
            { "type": "++"
            , "$1":
              [["H"], {"type": "FIELD", "name": "script"}, ["w", "q", ""]]
            }
          }
        ]
      , [ "script"
        , { "type": "singleton_map"
          , "key": "script.ed"
          , "value":
            {"type": "BLOB", "data": {"type": "var", "name": "script content"}}
          }
        ]
      , [ "patched files per target"
        , { "type": "foreach"
          , "var": "src"
          , "range": {"type": "FIELD", "name": "srcs"}
          , "body":
            { "type": "foreach_map"
            , "var_key": "file_name"
            , "var_val": "file"
            , "range":
              {"type": "DEP_ARTIFACTS", "dep": {"type": "var", "name": "src"}}
            , "body":
              { "type": "let*"
              , "bindings":
                [ [ "action output"
                  , { "type": "ACTION"
                    , "inputs":
                      { "type": "map_union"
                      , "$1":
                        [ {"type": "var", "name": "script"}
                        , { "type": "singleton_map"
                          , "key": "in"
                          , "value": {"type": "var", "name": "file"}
                          }
                        ]
                      }
                    , "cmd":
                      [ "/bin/sh"
                      , "-c"
                      , "cp in out && chmod 644 out && /bin/ed out < script.ed > log 2>&1 || (cat log && exit 1)"
                      ]
                    , "outs": ["out"]
                    }
                  ]
                ]
              , "body":
                { "type": "singleton_map"
                , "key": {"type": "var", "name": "file_name"}
                , "value":
                  { "type": "lookup"
                  , "map": {"type": "var", "name": "action output"}
                  , "key": "out"
                  }
                }
              }
            }
          }
        ]
      , [ "artifacts"
        , { "type": "disjoint_map_union"
          , "msg": "srcs artifacts must not overlap"
          , "$1":
            { "type": "++"
            , "$1": {"type": "var", "name": "patched files per target"}
            }
          }
        ]
      ]
    , "body":
      {"type": "RESULT", "artifacts": {"type": "var", "name": "artifacts"}}
    }
  }
}
```

A typical invocation of that rule would be a target file like the
following.

``` jsonc
{ "input.txt":
  { "type": "ed patch"
  , "script": ["%g/world/s//user/g", "%g/World/s//USER/g"]
  , "srcs": [["FILE", null, "input.txt"]]
  }
}
```

As the input file has the same name as a target (in the same module), we
use the explicit file reference in the specification of the sources.

### Implicit dependencies and config transitions

Say, instead of patching a file, we want to generate source files from
some high-level description using our actively developed code generator.
Then we have to do some additional considerations.

 - First of all, every target defined by this rule not only depends on
   the targets the user specifies. Additionally, our code generator is
   also an implicit dependency. And as it is under active development,
   we certainly do not want it to be taken from the ambient build
   environment (as we did in the previous example with `ed` which,
   however, is a pretty stable tool). So we use an `implicit` target
   for this.
 - Next, we notice that our code generator is used during the build. In
   particular, we want that tool (written in some compiled language) to
   be built for the platform we run our actions on, not the target
   platform we build our final binaries for. Therefore, we have to use
   a configuration transition.
 - As our defining expression also needs the configuration transition
   to access the artifacts of that implicit target, we better define it
   as a reusable expression. Other rules in our rule collection might
   also have the same task; so `["transitions", "for host"]` might be a
   good place to define it. In fact, it can look like the expression
   with that name in our own code base.

So, the overall organization of our rule might be as follows.

``` jsonc
{ "generated code":
  { "target_fields": ["srcs"]
  , "implicit": {"generator": [["generators", "foogen"]]}
  , "config_vars": ["HOST_ARCH"]
  , "imports": {"for host": ["transitions", "for host"]}
  , "config_transitions":
    {"generator": [{"type": "CALL_EXPRESSION", "name": "for host"}]}
  , "expression": ...
  }
}
```

### Providing information to consuming targets

In the simple case of patching, the resulting file is indeed the only
information the consumer of that target needs; in fact, the main point
was that the resulting target could be a drop-in replacement of a source
file. A typical rule, however, defines something like a library and a
library is much more, than just the actual library file and the public
headers: a library may depend on other libraries; therefore, in order to
use it, we need

 - to have the header files of dependencies available that might be
   included by the public header files of that library,
 - to have the libraries transitively depended upon available during
   linking, and
 - to know the order in which to link the dependencies (as they might
   have dependencies among each other).

In order to keep a maintainable build description, all this should be
taken care of by simply depending on that library. We do
*not* want the consumer of a target having to be aware of
such transitive dependencies (e.g., when constructing the link command
line), as it used to be the case in early build tools like `make`.

It is a deliberate design choice that a target is given only by the
result of its analysis, regardless of where it is coming from.
Therefore, all this information needs to be part of the result of a
target. Such kind of information is precisely, what the mentioned
`"provides"` map is for. As a map, it can contain an arbitrary amount of
information and the interface function `"DEP_PROVIDES"` is in such a way
that adding more providers does not affect targets not aware of them
(there is no function asking for all providers of a target). The keys
and their meaning have to be agreed upon by a target and its consumers.
As the latter, however, typically are a target of the same family
(authored by the same group), this usually is not a problem.

A typical example of computing a provided value is the `"link-args"` in
the rules used by `just` itself. They are defined by the following
expression.

``` jsonc
{ "type": "nub_right"
, "$1":
  { "type": "++"
  , "$1":
    [ {"type": "keys", "$1": {"type": "var", "name": "lib"}}
    , {"type": "CALL_EXPRESSION", "name": "link-args-deps"}
    , {"type": "var", "name": "link external", "default": []}
    ]
  }
}
```

This expression

 - collects the respective provider of its dependencies,
 - adds itself in front, and
 - deduplicates the resulting list, keeping only the right-most
   occurrence of each entry.

In this way, the invariant is kept, that the `"link-args"` from a
topological ordering of the dependencies (in the order that a each entry
is mentioned before its dependencies).