summaryrefslogtreecommitdiff
path: root/doc/tutorial
diff options
context:
space:
mode:
authorOliver Reiche <oliver.reiche@huawei.com>2023-06-01 13:36:32 +0200
committerOliver Reiche <oliver.reiche@huawei.com>2023-06-12 16:29:05 +0200
commitb66a7359fbbff35af630c88c56598bbc06b393e1 (patch)
treed866802c4b44c13cbd90f9919cc7fc472091be0c /doc/tutorial
parent144b2c619f28c91663936cd445251ca28af45f88 (diff)
downloadjustbuild-b66a7359fbbff35af630c88c56598bbc06b393e1.tar.gz
doc: Convert orgmode files to markdown
Diffstat (limited to 'doc/tutorial')
-rw-r--r--doc/tutorial/getting-started.md217
-rw-r--r--doc/tutorial/getting-started.org212
-rw-r--r--doc/tutorial/hello-world.md379
-rw-r--r--doc/tutorial/hello-world.org370
-rw-r--r--doc/tutorial/proto.md (renamed from doc/tutorial/proto.org)245
-rw-r--r--doc/tutorial/rebuild.md (renamed from doc/tutorial/rebuild.org)164
-rw-r--r--doc/tutorial/target-file-glob-tree.md (renamed from doc/tutorial/target-file-glob-tree.org)326
-rw-r--r--doc/tutorial/tests.md (renamed from doc/tutorial/tests.org)270
-rw-r--r--doc/tutorial/third-party-software.md473
-rw-r--r--doc/tutorial/third-party-software.org475
10 files changed, 1573 insertions, 1558 deletions
diff --git a/doc/tutorial/getting-started.md b/doc/tutorial/getting-started.md
new file mode 100644
index 00000000..36a57d26
--- /dev/null
+++ b/doc/tutorial/getting-started.md
@@ -0,0 +1,217 @@
+Getting Started
+===============
+
+In order to use *justbuild*, first make sure that `just`, `just-mr`, and
+`just-import-git` are available in your `PATH`.
+
+Creating a new project
+----------------------
+
+*justbuild* needs to know the root of the project worked on. By default,
+it searches upwards from the current directory till it finds a marker.
+Currently, we support three different markers: the files `ROOT` and
+`WORKSPACE` or the directory `.git`. Lets create a new project by
+creating one of those markers:
+
+``` sh
+$ touch ROOT
+```
+
+Creating a generic target
+-------------------------
+
+By default, targets are described in `TARGETS` files. These files
+contain a `JSON` object with the target name as key and the target
+description as value. A target description is an object with at least a
+single mandatory field: `"type"`. This field specifies which rule
+(built-in or user-defined) to apply for this target.
+
+A simple target that only executes commands can be created using the
+built-in `"generic"` rule, which requires at least one command and one
+output file or directory. To create such a target, create the file
+`TARGETS` with the following content:
+
+``` {.jsonc srcname="TARGETS"}
+{ "greeter":
+ { "type": "generic"
+ , "cmds": ["echo -n 'Hello ' > out.txt", "cat name.txt >> out.txt"]
+ , "outs": ["out.txt"]
+ , "deps": ["name.txt"]
+ }
+}
+```
+
+In this example, the `"greeter"` target will run two commands to produce
+the output file `out.txt`. The second command depends on the input file
+`name.txt` that we need to create as well:
+
+``` sh
+$ echo World > name.txt
+```
+
+Building a generic target
+-------------------------
+
+To build a target, we need to run `just` with the subcommand `build`:
+
+``` sh
+$ just build greeter
+INFO: Requested target is [["@","","","greeter"],{}]
+INFO: Analysed target [["@","","","greeter"],{}]
+INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
+INFO: Discovered 1 actions, 0 trees, 0 blobs
+INFO: Building [["@","","","greeter"],{}].
+INFO: Processed 1 actions, 0 cache hits.
+INFO: Artifacts built, logical paths are:
+ out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f]
+$
+```
+
+The subcommand `build` just builds the artifact but does not stage it to
+any user-defined location on the file system. Instead it reports a
+description of the artifact consisting of `git` blob identifier, size,
+and type (in this case `f` for non-executable file). To also stage the
+produced artifact to the working directory, use the `install` subcommand
+and specify the output directory:
+
+``` sh
+$ just install greeter -o .
+INFO: Requested target is [["@","","","greeter"],{}]
+INFO: Analysed target [["@","","","greeter"],{}]
+INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
+INFO: Discovered 1 actions, 0 trees, 0 blobs
+INFO: Building [["@","","","greeter"],{}].
+INFO: Processed 1 actions, 1 cache hits.
+INFO: Artifacts can be found in:
+ /tmp/tutorial/out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f]
+$ cat out.txt
+Hello World
+$
+```
+
+Note that the `install` subcommand initiates the build a second time,
+without executing any actions as all actions are being served from
+cache. The produced artifact is identical, which is indicated by the
+same hash/size/type.
+
+If one is only interested in a single final artifact, one can also
+request via the `-P` option that this artifact be written to standard
+output after the build. As all messages are reported to standard error,
+this can be used for both, interactively reading a text file, as well as
+for piping the artifact to another program.
+
+``` sh
+$ just build greeter -Pout.txt
+INFO: Requested target is [["@","","","greeter"],{}]
+INFO: Analysed target [["@","","","greeter"],{}]
+INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
+INFO: Discovered 1 actions, 0 trees, 0 blobs
+INFO: Building [["@","","","greeter"],{}].
+INFO: Processed 1 actions, 1 cache hits.
+INFO: Artifacts built, logical paths are:
+ out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f]
+Hello World
+$
+```
+
+Alternatively, we could also directly request the artifact `out.txt`
+from *justbuild*'s CAS (content-addressable storage) and print it on
+the command line via:
+
+``` sh
+$ just install-cas [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f]
+Hello World
+$
+```
+
+The canonical way of requesting an object from the CAS is, as just
+shown, to specify the full triple of hash, size, and type, separated by
+colons and enclosed in square brackets. To simplify usage, the brackets
+can be omitted and the size and type fields have the default values `0`
+and `f`, respectively. While the default value for the size is wrong for
+all but one string, the hash still determines the content of the file
+and hence the local CAS is still able to retrieve the file. So the
+typical invocation would simply specify the hash.
+
+``` sh
+$ just install-cas 557db03de997c86a4a028e1ebd3a1ceb225be238
+Hello World
+$
+```
+
+Targets versus Files: The Stage
+-------------------------------
+
+When invoking the `build` command, we had to specify the target
+`greeter`, not the output file `out.txt`. While other build systems
+allow requests specifying an output file, for *justbuild* this would
+conflict with a fundamental design principle: staging; each target has
+its own logical output space, the "stage", where it can put its
+artifacts. We can, without any problem, add a second target also
+generating a file `out.txt`.
+
+``` {.jsonc srcname="TARGETS"}
+...
+, "upper":
+ { "type": "generic"
+ , "cmds": ["cat name.txt | tr a-z A-Z > out.txt"]
+ , "outs": ["out.txt"]
+ , "deps": ["name.txt"]
+ }
+...
+```
+
+As we only request targets, no conflicts arise.
+
+``` sh
+$ just build upper -P out.txt
+INFO: Requested target is [["@","","","upper"],{}]
+INFO: Analysed target [["@","","","upper"],{}]
+INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
+INFO: Discovered 1 actions, 0 trees, 0 blobs
+INFO: Building [["@","","","upper"],{}].
+INFO: Processed 1 actions, 0 cache hits.
+INFO: Artifacts built, logical paths are:
+ out.txt [83cf24cdfb4891a36bee93421930dd220766299a:6:f]
+WORLD
+$ just build greeter -P out.txt
+INFO: Requested target is [["@","","","greeter"],{}]
+INFO: Analysed target [["@","","","greeter"],{}]
+INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
+INFO: Discovered 1 actions, 0 trees, 0 blobs
+INFO: Building [["@","","","greeter"],{}].
+INFO: Processed 1 actions, 1 cache hits.
+INFO: Artifacts built, logical paths are:
+ out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f]
+Hello World
+$
+```
+
+While one normally tries to design targets in such a way that they
+don't have conflicting files if they should be used together, it is up
+to the receiving target to decide what to do with those artifacts. A
+built-in rule allowing to rearrange artifacts is `"install"`; a detailed
+description of this rule can be found in the documentation. In the
+simple case of a target producing precisely one file, the argument
+`"files"` can be used to map that file to a new location.
+
+``` {.jsonc srcname="TARGETS"}
+...
+, "both":
+ {"type": "install", "files": {"hello.txt": "greeter", "upper.txt": "upper"}}
+...
+```
+
+``` sh
+$ just build both
+INFO: Requested target is [["@","","","both"],{}]
+INFO: Analysed target [["@","","","both"],{}]
+INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
+INFO: Discovered 2 actions, 0 trees, 0 blobs
+INFO: Building [["@","","","both"],{}].
+INFO: Processed 2 actions, 2 cache hits.
+INFO: Artifacts built, logical paths are:
+ hello.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f]
+ upper.txt [83cf24cdfb4891a36bee93421930dd220766299a:6:f]
+$
+```
diff --git a/doc/tutorial/getting-started.org b/doc/tutorial/getting-started.org
deleted file mode 100644
index 5a041397..00000000
--- a/doc/tutorial/getting-started.org
+++ /dev/null
@@ -1,212 +0,0 @@
-* Getting Started
-
-In order to use /justbuild/, first make sure that ~just~, ~just-mr~, and
-~just-import-git~ are available in your ~PATH~.
-
-** Creating a new project
-
-/justbuild/ needs to know the root of the project worked on. By default, it
-searches upwards from the current directory till it finds a marker. Currently,
-we support three different markers: the files ~ROOT~ and ~WORKSPACE~ or the
-directory ~.git~. Lets create a new project by creating one of those markers:
-
-#+BEGIN_SRC sh
-$ touch ROOT
-#+END_SRC
-
-** Creating a generic target
-
-By default, targets are described in ~TARGETS~ files. These files contain a
-~JSON~ object with the target name as key and the target description as value. A
-target description is an object with at least a single mandatory field:
-~"type"~. This field specifies which rule (built-in or user-defined) to apply
-for this target.
-
-A simple target that only executes commands can be created using the built-in
-~"generic"~ rule, which requires at least one command and one output file or
-directory. To create such a target, create the file ~TARGETS~ with the following
-content:
-
-#+SRCNAME: TARGETS
-#+BEGIN_SRC js
-{ "greeter":
- { "type": "generic"
- , "cmds": ["echo -n 'Hello ' > out.txt", "cat name.txt >> out.txt"]
- , "outs": ["out.txt"]
- , "deps": ["name.txt"]
- }
-}
-#+END_SRC
-
-In this example, the ~"greeter"~ target will run two commands to produce the
-output file ~out.txt~. The second command depends on the input file ~name.txt~
-that we need to create as well:
-
-#+BEGIN_SRC sh
-$ echo World > name.txt
-#+END_SRC
-
-** Building a generic target
-
-To build a target, we need to run ~just~ with the subcommand ~build~:
-
-#+BEGIN_SRC sh
-$ just build greeter
-INFO: Requested target is [["@","","","greeter"],{}]
-INFO: Analysed target [["@","","","greeter"],{}]
-INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
-INFO: Discovered 1 actions, 0 trees, 0 blobs
-INFO: Building [["@","","","greeter"],{}].
-INFO: Processed 1 actions, 0 cache hits.
-INFO: Artifacts built, logical paths are:
- out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f]
-$
-#+END_SRC
-
-The subcommand ~build~ just builds the artifact but does not stage it to any
-user-defined location on the file system. Instead it reports a description
-of the artifact consisting of ~git~ blob identifier, size, and type (in
-this case ~f~ for non-executable file). To also stage the produced artifact to
-the working directory, use the ~install~ subcommand and specify the output
-directory:
-
-#+BEGIN_SRC sh
-$ just install greeter -o .
-INFO: Requested target is [["@","","","greeter"],{}]
-INFO: Analysed target [["@","","","greeter"],{}]
-INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
-INFO: Discovered 1 actions, 0 trees, 0 blobs
-INFO: Building [["@","","","greeter"],{}].
-INFO: Processed 1 actions, 1 cache hits.
-INFO: Artifacts can be found in:
- /tmp/tutorial/out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f]
-$ cat out.txt
-Hello World
-$
-#+END_SRC
-
-Note that the ~install~ subcommand initiates the build a second time, without
-executing any actions as all actions are being served from cache. The produced
-artifact is identical, which is indicated by the same hash/size/type.
-
-If one is only interested in a single final artifact, one can
-also request via the ~-P~ option that this artifact be written to
-standard output after the build. As all messages are reported to
-standard error, this can be used for both, interactively reading a
-text file, as well as for piping the artifact to another program.
-
-#+BEGIN_SRC sh
-$ just build greeter -Pout.txt
-INFO: Requested target is [["@","","","greeter"],{}]
-INFO: Analysed target [["@","","","greeter"],{}]
-INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
-INFO: Discovered 1 actions, 0 trees, 0 blobs
-INFO: Building [["@","","","greeter"],{}].
-INFO: Processed 1 actions, 1 cache hits.
-INFO: Artifacts built, logical paths are:
- out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f]
-Hello World
-$
-#+END_SRC
-
-Alternatively, we could also directly request the artifact ~out.txt~ from
-/justbuild/'s CAS (content-addressable storage) and print it on the command line
-via:
-
-#+BEGIN_SRC sh
-$ just install-cas [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f]
-Hello World
-$
-#+END_SRC
-
-The canonical way of requesting an object from the CAS is, as just shown, to
-specify the full triple of hash, size, and type, separated by colons and
-enclosed in square brackets. To simplify usage, the brackets can be omitted
-and the size and type fields have the default values ~0~ and ~f~, respectively.
-While the default value for the size is wrong for all but one string, the hash
-still determines the content of the file and hence the local CAS is still
-able to retrieve the file. So the typical invocation would simply specify the
-hash.
-
-#+BEGIN_SRC sh
-$ just install-cas 557db03de997c86a4a028e1ebd3a1ceb225be238
-Hello World
-$
-#+END_SRC
-
-** Targets versus Files: The Stage
-
-When invoking the ~build~ command, we had to specify the target ~greeter~,
-not the output file ~out.txt~. While other build systems allow requests
-specifying an output file, for /justbuild/ this would conflict with a
-fundamental design principle: staging; each target has its own logical
-output space, the "stage", where it can put its artifacts. We can, without
-any problem, add a second target also generating a file ~out.txt~.
-
-#+SRCNAME: TARGETS
-#+BEGIN_SRC js
-...
-, "upper":
- { "type": "generic"
- , "cmds": ["cat name.txt | tr a-z A-Z > out.txt"]
- , "outs": ["out.txt"]
- , "deps": ["name.txt"]
- }
-...
-#+END_SRC
-
-As we only request targets, no conflicts arise.
-
-#+BEGIN_SRC sh
-$ just build upper -P out.txt
-INFO: Requested target is [["@","","","upper"],{}]
-INFO: Analysed target [["@","","","upper"],{}]
-INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
-INFO: Discovered 1 actions, 0 trees, 0 blobs
-INFO: Building [["@","","","upper"],{}].
-INFO: Processed 1 actions, 0 cache hits.
-INFO: Artifacts built, logical paths are:
- out.txt [83cf24cdfb4891a36bee93421930dd220766299a:6:f]
-WORLD
-$ just build greeter -P out.txt
-INFO: Requested target is [["@","","","greeter"],{}]
-INFO: Analysed target [["@","","","greeter"],{}]
-INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
-INFO: Discovered 1 actions, 0 trees, 0 blobs
-INFO: Building [["@","","","greeter"],{}].
-INFO: Processed 1 actions, 1 cache hits.
-INFO: Artifacts built, logical paths are:
- out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f]
-Hello World
-$
-#+END_SRC
-
-While one normally tries to design targets in such a way that they
-don't have conflicting files if they should be used together, it is
-up to the receiving target to decide what to do with those artifacts.
-A built-in rule allowing to rearrange artifacts is ~"install"~; a
-detailed description of this rule can be found in the documentation.
-In the simple case of a target producing precisely one file, the
-argument ~"files"~ can be used to map that file to a new location.
-
-#+SRCNAME: TARGETS
-#+BEGIN_SRC js
-...
-, "both":
- {"type": "install", "files": {"hello.txt": "greeter", "upper.txt": "upper"}}
-...
-#+END_SRC
-
-#+BEGIN_SRC sh
-$ just build both
-INFO: Requested target is [["@","","","both"],{}]
-INFO: Analysed target [["@","","","both"],{}]
-INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
-INFO: Discovered 2 actions, 0 trees, 0 blobs
-INFO: Building [["@","","","both"],{}].
-INFO: Processed 2 actions, 2 cache hits.
-INFO: Artifacts built, logical paths are:
- hello.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f]
- upper.txt [83cf24cdfb4891a36bee93421930dd220766299a:6:f]
-$
-#+END_SRC
diff --git a/doc/tutorial/hello-world.md b/doc/tutorial/hello-world.md
new file mode 100644
index 00000000..9af68f07
--- /dev/null
+++ b/doc/tutorial/hello-world.md
@@ -0,0 +1,379 @@
+Building C++ Hello World
+========================
+
+*justbuild* is a true language-agnostic (there are no more-equal
+languages) and multi-repository build system. As a consequence,
+high-level concepts (e.g., C++ binaries, C++ libraries, etc.) are not
+hardcoded built-ins of the tool, but rather provided via a set of rules.
+These rules can be specified as a true dependency to your project like
+any other external repository your project might depend on.
+
+Setting up the Multi-Repository Configuration
+---------------------------------------------
+
+To build a project with multi-repository dependencies, we first need to
+provide a configuration that declares the required repositories. Before
+we begin, we need to declare where the root of our workspace is located
+by creating an empty file `ROOT`:
+
+``` sh
+$ touch ROOT
+```
+
+Second, we also need to create the multi-repository configuration
+`repos.json` in the workspace root:
+
+``` {.jsonc srcname="repos.json"}
+{ "main": "tutorial"
+, "repositories":
+ { "rules-cc":
+ { "repository":
+ { "type": "git"
+ , "branch": "master"
+ , "commit": "123d8b03bf2440052626151c14c54abce2726e6f"
+ , "repository": "https://github.com/just-buildsystem/rules-cc.git"
+ , "subdir": "rules"
+ }
+ }
+ , "tutorial":
+ { "repository": {"type": "file", "path": "."}
+ , "bindings": {"rules": "rules-cc"}
+ }
+ }
+}
+```
+
+In that configuration, two repositories are defined:
+
+1. The `"rules-cc"` repository located in the subdirectory `rules` of
+ [just-buildsystem/rules-cc:123d8b03bf2440052626151c14c54abce2726e6f](https://github.com/just-buildsystem/rules-cc/tree/123d8b03bf2440052626151c14c54abce2726e6f),
+ which contains the high-level concepts for building C/C++ binaries
+ and libraries.
+
+2. The `"tutorial"` repository located at `.`, which contains the
+ targets that we want to build. It has a single dependency, which is
+ the *rules* that are needed to build the target. These rules are
+ bound via the open name `"rules"` to the just created repository
+ `"rules-cc"`. In this way, the entities provided by `"rules-cc"` can
+ be accessed from within the `"tutorial"` repository via the
+ fully-qualified name `["@", "rules", "<module>", "<name>"]`;
+ fully-qualified names (for rules, targets to build (like libraries,
+ binaries), etc) are given by a repository name, a path specifying a
+ directory within that repository (the "module") where the
+ specification file is located, and a symbolic name (i.e., an
+ arbitrary string that is used as key in the specification).
+
+The final repository configuration contains a single `JSON` object with
+the key `"repositories"` referring to an object of repository names as
+keys and repository descriptions as values. For convenience, the main
+repository to pick is set to `"tutorial"`.
+
+Description of the helloworld target
+------------------------------------
+
+For this tutorial, we want to create a target `helloworld` that produces
+a binary from the C++ source `main.cpp`. To define such a target, create
+a `TARGETS` file with the following content:
+
+``` {.jsonc srcname="TARGETS"}
+{ "helloworld":
+ { "type": ["@", "rules", "CC", "binary"]
+ , "name": ["helloworld"]
+ , "srcs": ["main.cpp"]
+ }
+}
+```
+
+The `"type"` field refers to the rule `"binary"` from the module `"CC"`
+of the `"rules"` repository. This rule additionally requires the string
+field `"name"`, which specifies the name of the binary to produce; as
+the generic interface of rules is to have fields either take a list of
+strings or a list of targets, we have to specify the name as a list
+(this rule will simply concatenate all strings given in this field).
+Furthermore, at least one input to the binary is required, which can be
+specified via the target fields `"srcs"` or `"deps"`. In our case, the
+former is used, which contains our single source file (files are
+considered targets).
+
+Now, the last file that is missing is the actual source file `main.cpp`:
+
+``` {.cpp srcname="main.cpp"}
+#include <iostream>
+
+int main() {
+ std::cout << "Hello world!\n";
+ return 0;
+}
+```
+
+Building the helloworld target
+------------------------------
+
+To build the `helloworld` target, we need specify it on the `just-mr`
+command line:
+
+``` sh
+$ just-mr build helloworld
+INFO: Requested target is [["@","tutorial","","helloworld"],{}]
+INFO: Analysed target [["@","tutorial","",helloworld"],{}]
+INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
+INFO: Discovered 2 actions, 1 trees, 0 blobs
+INFO: Building [["@","helloworld","","helloworld"],{}].
+INFO: Processed 2 actions, 0 cache hits.
+INFO: Artifacts built, logical paths are:
+ helloworld [b5cfca8b810adc4686f5cac00258a137c5d4a3ba:17088:x]
+$
+```
+
+Note that the target is taken from the `tutorial` repository, as it
+specified as the main repository in `repos.json`. If targets from other
+repositories should be build, the repository to use must be specified
+via the `--main` option.
+
+`just-mr` reads the repository configuration, fetches externals (if
+any), generates the actual build configuration, and stores it in its
+cache directory (by default under `$HOME/.cache/just`). Afterwards, the
+generated configuration is used to call the `just` binary, which
+performs the actual build.
+
+Note that these two programs, `just-mr` and `just`, can also be run
+individually. To do so, first run `just-mr` with `setup` and capture the
+path to the generated build configuration from stdout by assigning it to
+a shell variable (e.g., `CONF`). Afterwards, `just` can be called to
+perform the actual build by explicitly specifying the configuration file
+via `-C`:
+
+``` sh
+$ CONF=$(just-mr setup tutorial)
+$ just build -C $CONF helloworld
+```
+
+Note that `just-mr` only needs to be run the very first time and only
+once again whenever the `repos.json` file is modified.
+
+By default, the BSD-default compiler front-ends (which are also defined
+for most Linux distributions) `cc` and `c++` are used for C and C++
+(variables `"CC"` and `"CXX"`). If you want to temporarily use different
+defaults, you can use `-D` to provide a JSON object that sets different
+default variables. For instance, to use Clang as C++ compiler for a
+single build invocation, you can use the following command to provide an
+object that sets `"CXX"` to `"clang++"`:
+
+``` sh
+$ just-mr build helloworld -D'{"CXX":"clang++"}'
+INFO: Requested target is [["@","tutorial","","helloworld"],{"CXX":"clang++"}]
+INFO: Analysed target [["@","tutorial","","helloworld"],{"CXX":"clang++"}]
+INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
+INFO: Discovered 2 actions, 1 trees, 0 blobs
+INFO: Building [["@","tutorial","","helloworld"],{"CXX":"clang++"}].
+INFO: Processed 2 actions, 0 cache hits.
+INFO: Artifacts built, logical paths are:
+ helloworld [b8cf7b8579d9dc7172b61660139e2c14521cedae:16944:x]
+$
+```
+
+Defining project defaults
+-------------------------
+
+To define a custom set of defaults (toolchain and compile flags) for
+your project, you need to create a separate file root for providing
+required `TARGETS` file, which contains the `"defaults"` target that
+should be used by the rules. This file root is then used as the *target
+root* for the rules, i.e., the search path for `TARGETS` files. In this
+way, the description of the `"defaults"` target is provided in a
+separate file root, to keep the rules repository independent of these
+definitions.
+
+We will call the new file root `tutorial-defaults` and need to create a
+module directory `CC` in it:
+
+``` sh
+$ mkdir -p ./tutorial-defaults/CC
+```
+
+In that module, we need to create the file
+`tutorial-defaults/CC/TARGETS` that contains the target `"defaults"` and
+specifies which toolchain and compile flags to use; it has to specify
+the complete toolchain, but can specify a `"base"` toolchain to inherit
+from. In our case, we don't use any base, but specify all the required
+fields directly.
+
+``` {.jsonc srcname="tutorial-defaults/CC/TARGETS"}
+{ "defaults":
+ { "type": ["CC", "defaults"]
+ , "CC": ["cc"]
+ , "CXX": ["c++"]
+ , "CFLAGS": ["-O2", "-Wall"]
+ , "CXXFLAGS": ["-O2", "-Wall"]
+ , "AR": ["ar"]
+ , "PATH": ["/bin", "/usr/bin"]
+ }
+}
+```
+
+To use the project defaults, modify the existing `repos.json` to reflect
+the following content:
+
+``` {.jsonc srcname="repos.json"}
+{ "main": "tutorial"
+, "repositories":
+ { "rules-cc":
+ { "repository":
+ { "type": "git"
+ , "branch": "master"
+ , "commit": "123d8b03bf2440052626151c14c54abce2726e6f"
+ , "repository": "https://github.com/just-buildsystem/rules-cc.git"
+ , "subdir": "rules"
+ }
+ , "target_root": "tutorial-defaults"
+ , "rule_root": "rules-cc"
+ }
+ , "tutorial":
+ { "repository": {"type": "file", "path": "."}
+ , "bindings": {"rules": "rules-cc"}
+ }
+ , "tutorial-defaults":
+ { "repository": {"type": "file", "path": "./tutorial-defaults"}
+ }
+ }
+}
+```
+
+Note that the `"defaults"` target uses the rule `["CC", "defaults"]`
+without specifying any external repository (e.g.,
+`["@", "rules", ...]`). This is because `"tutorial-defaults"` is not a
+full-fledged repository but merely a file root that is considered local
+to the `"rules-cc"` repository. In fact, the `"rules-cc"` repository
+cannot refer to any external repository as it does not have any defined
+bindings.
+
+To rebuild the project, we need to rerun `just-mr` (note that due to
+configuration changes, rerunning only `just` would not suffice):
+
+``` sh
+$ just-mr build helloworld
+INFO: Requested target is [["@","tutorial","","helloworld"],{}]
+INFO: Analysed target [["@","tutorial","","helloworld"],{}]
+INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
+INFO: Discovered 2 actions, 1 trees, 0 blobs
+INFO: Building [["@","tutorial","","helloworld"],{}].
+INFO: Processed 2 actions, 0 cache hits.
+INFO: Artifacts built, logical paths are:
+ helloworld [487dc9e47b978877ed2f7d80b3395ce84b23be92:16992:x]
+$
+```
+
+Note that the output binary may have changed due to different defaults.
+
+Modeling target dependencies
+----------------------------
+
+For demonstration purposes, we will separate the print statements into a
+static library `greet`, which will become a dependency to our binary.
+Therefore, we create a new subdirectory `greet` with the files
+`greet/greet.hpp`:
+
+``` {.cpp srcname="greet/greet.hpp"}
+#include <string>
+
+void greet(std::string const& s);
+```
+
+and `greet/greet.cpp`:
+
+``` {.cpp srcname="greet/greet.cpp"}
+#include "greet.hpp"
+#include <iostream>
+
+void greet(std::string const& s) {
+ std::cout << "Hello " << s << "!\n";
+}
+```
+
+These files can now be used to create a static library `libgreet.a`. To
+do so, we need to create the following target description in
+`greet/TARGETS`:
+
+``` {.jsonc srcname="greet/TARGETS"}
+{ "greet":
+ { "type": ["@", "rules", "CC", "library"]
+ , "name": ["greet"]
+ , "hdrs": ["greet.hpp"]
+ , "srcs": ["greet.cpp"]
+ , "stage": ["greet"]
+ }
+}
+```
+
+Similar to `"binary"`, we have to provide a name and source file.
+Additionally, a library has public headers defined via `"hdrs"` and an
+optional staging directory `"stage"` (default value `"."`). The staging
+directory specifies where the consumer of this library can expect to
+find the library's artifacts. Note that this does not need to reflect
+the location on the file system (i.e., a full-qualified path like
+`["com", "example", "utils", "greet"]` could be used to distinguish it
+from greeting libraries of other projects). The staging directory does
+not only affect the main artifact `libgreet.a` but also it's
+*runfiles*, a second set of artifacts, usually those a consumer needs to
+make proper use the actual artifact; in the case of a library, the
+runfiles are its public headers. Hence, the public header will be staged
+to `"greet/greet.hpp"`. With that knowledge, we can now perform the
+necessary modifications to `main.cpp`:
+
+``` {.cpp srcname="main.cpp"}
+#include "greet/greet.hpp"
+
+int main() {
+ greet("Universe");
+ return 0;
+}
+```
+
+The target `"helloworld"` will have a direct dependency to the target
+`"greet"` of the module `"greet"` in the top-level `TARGETS` file:
+
+``` {.jsonc srcname="TARGETS"}
+{ "helloworld":
+ { "type": ["@", "rules", "CC", "binary"]
+ , "name": ["helloworld"]
+ , "srcs": ["main.cpp"]
+ , "private-deps": [["greet", "greet"]]
+ }
+}
+```
+
+Note that there is no need to explicitly specify `"greet"`'s public
+headers here as the appropriate artifacts of dependencies are
+automatically added to the inputs of compile and link actions. The new
+binary can be built with the same command as before (no need to rerun
+`just-mr`):
+
+``` sh
+$ just-mr build helloworld
+INFO: Requested target is [["@","tutorial","","helloworld"],{}]
+INFO: Analysed target [["@","tutorial","","helloworld"],{}]
+INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
+INFO: Discovered 4 actions, 2 trees, 0 blobs
+INFO: Building [["@","tutorial","","helloworld"],{}].
+INFO: Processed 4 actions, 0 cache hits.
+INFO: Artifacts built, logical paths are:
+ helloworld [2b81e3177afc382452a2df9f294d3df90a9ccaf0:17664:x]
+$
+```
+
+To only build the static library target `"greet"` from module `"greet"`,
+run the following command:
+
+``` sh
+$ just-mr build greet greet
+INFO: Requested target is [["@","tutorial","greet","greet"],{}]
+INFO: Analysed target [["@","tutorial","greet","greet"],{}]
+INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
+INFO: Discovered 2 actions, 1 trees, 0 blobs
+INFO: Building [["@","tutorial","greet","greet"],{}].
+INFO: Processed 2 actions, 2 cache hits.
+INFO: Artifacts built, logical paths are:
+ greet/libgreet.a [83ed406e21f285337b0c9bd5011f56f656bba683:2992:f]
+ (1 runfiles omitted.)
+$
+```
diff --git a/doc/tutorial/hello-world.org b/doc/tutorial/hello-world.org
deleted file mode 100644
index 342eaf82..00000000
--- a/doc/tutorial/hello-world.org
+++ /dev/null
@@ -1,370 +0,0 @@
-* Building C++ Hello World
-
-/justbuild/ is a true language-agnostic (there are no more-equal languages) and
-multi-repository build system. As a consequence, high-level concepts (e.g., C++
-binaries, C++ libraries, etc.) are not hardcoded built-ins of the tool, but
-rather provided via a set of rules. These rules can be specified as a true
-dependency to your project like any other external repository your project might
-depend on.
-
-** Setting up the Multi-Repository Configuration
-
-To build a project with multi-repository dependencies, we first need to provide
-a configuration that declares the required repositories. Before we begin, we
-need to declare where the root of our workspace is located by creating an empty
-file ~ROOT~:
-
-#+BEGIN_SRC sh
-$ touch ROOT
-#+END_SRC
-
-Second, we also need to create the multi-repository configuration ~repos.json~
-in the workspace root:
-
-#+SRCNAME: repos.json
-#+BEGIN_SRC js
-{ "main": "tutorial"
-, "repositories":
- { "rules-cc":
- { "repository":
- { "type": "git"
- , "branch": "master"
- , "commit": "123d8b03bf2440052626151c14c54abce2726e6f"
- , "repository": "https://github.com/just-buildsystem/rules-cc.git"
- , "subdir": "rules"
- }
- }
- , "tutorial":
- { "repository": {"type": "file", "path": "."}
- , "bindings": {"rules": "rules-cc"}
- }
- }
-}
-#+END_SRC
-
-In that configuration, two repositories are defined:
-
- 1. The ~"rules-cc"~ repository located in the subdirectory ~rules~ of
- [[https://github.com/just-buildsystem/rules-cc/tree/123d8b03bf2440052626151c14c54abce2726e6f][just-buildsystem/rules-cc:123d8b03bf2440052626151c14c54abce2726e6f]],
- which contains the high-level concepts for building C/C++ binaries and
- libraries.
-
- 2. The ~"tutorial"~ repository located at ~.~, which contains the targets that
- we want to build. It has a single dependency, which is the /rules/ that are
- needed to build the target. These rules are bound via the open name
- ~"rules"~ to the just created repository ~"rules-cc"~. In this way, the
- entities provided by ~"rules-cc"~ can be accessed from within the
- ~"tutorial"~ repository via the fully-qualified name
- ~["@", "rules", "<module>", "<name>"]~; fully-qualified
- names (for rules, targets to build (like libraries, binaries),
- etc) are given by a repository name, a path specifying a
- directory within that repository (the "module") where the
- specification file is located, and a symbolic name (i.e., an
- arbitrary string that is used as key in the specification).
-
-The final repository configuration contains a single ~JSON~ object with the key
-~"repositories"~ referring to an object of repository names as keys and
-repository descriptions as values. For convenience, the main repository to pick
-is set to ~"tutorial"~.
-
-** Description of the helloworld target
-
-For this tutorial, we want to create a target ~helloworld~ that produces a
-binary from the C++ source ~main.cpp~. To define such a target, create a
-~TARGETS~ file with the following content:
-
-#+SRCNAME: TARGETS
-#+BEGIN_SRC js
-{ "helloworld":
- { "type": ["@", "rules", "CC", "binary"]
- , "name": ["helloworld"]
- , "srcs": ["main.cpp"]
- }
-}
-#+END_SRC
-
-The ~"type"~ field refers to the rule ~"binary"~ from the module ~"CC"~ of the
-~"rules"~ repository. This rule additionally requires the string field ~"name"~,
-which specifies the name of the binary to produce; as the generic interface of
-rules is to have fields either take a list of strings or a list of targets,
-we have to specify the name as a list (this rule will simply concatenate all
-strings given in this field). Furthermore, at least one
-input to the binary is required, which can be specified via the target fields
-~"srcs"~ or ~"deps"~. In our case, the former is used, which contains our single
-source file (files are considered targets).
-
-Now, the last file that is missing is the actual source file ~main.cpp~:
-
-#+SRCNAME: main.cpp
-#+BEGIN_SRC cpp
-#include <iostream>
-
-int main() {
- std::cout << "Hello world!\n";
- return 0;
-}
-#+END_SRC
-
-** Building the helloworld target
-
-To build the ~helloworld~ target, we need specify it on the ~just-mr~ command
-line:
-
-#+BEGIN_SRC sh
-$ just-mr build helloworld
-INFO: Requested target is [["@","tutorial","","helloworld"],{}]
-INFO: Analysed target [["@","tutorial","",helloworld"],{}]
-INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
-INFO: Discovered 2 actions, 1 trees, 0 blobs
-INFO: Building [["@","helloworld","","helloworld"],{}].
-INFO: Processed 2 actions, 0 cache hits.
-INFO: Artifacts built, logical paths are:
- helloworld [b5cfca8b810adc4686f5cac00258a137c5d4a3ba:17088:x]
-$
-#+END_SRC
-
-Note that the target is taken from the ~tutorial~ repository, as it specified as
-the main repository in ~repos.json~. If targets from other repositories should
-be build, the repository to use must be specified via the ~--main~ option.
-
-~just-mr~ reads the repository configuration, fetches externals (if any),
-generates the actual build configuration, and stores it in its cache directory
-(by default under ~$HOME/.cache/just~). Afterwards, the generated configuration
-is used to call the ~just~ binary, which performs the actual build.
-
-Note that these two programs, ~just-mr~ and ~just~, can also be run
-individually. To do so, first run ~just-mr~ with ~setup~ and capture the path to
-the generated build configuration from stdout by assigning it to a shell
-variable (e.g., ~CONF~). Afterwards, ~just~ can be called to perform the actual
-build by explicitly specifying the configuration file via ~-C~:
-
-#+BEGIN_SRC sh
-$ CONF=$(just-mr setup tutorial)
-$ just build -C $CONF helloworld
-#+END_SRC
-
-Note that ~just-mr~ only needs to be run the very first time and only once again
-whenever the ~repos.json~ file is modified.
-
-By default, the BSD-default compiler front-ends (which are also defined for most
-Linux distributions) ~cc~ and ~c++~ are used for C and C++ (variables ~"CC"~ and
-~"CXX"~). If you want to temporarily use different defaults, you can use ~-D~ to
-provide a JSON object that sets different default variables. For instance, to
-use Clang as C++ compiler for a single build invocation, you can use the
-following command to provide an object that sets ~"CXX"~ to ~"clang++"~:
-
-#+BEGIN_SRC sh
-$ just-mr build helloworld -D'{"CXX":"clang++"}'
-INFO: Requested target is [["@","tutorial","","helloworld"],{"CXX":"clang++"}]
-INFO: Analysed target [["@","tutorial","","helloworld"],{"CXX":"clang++"}]
-INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
-INFO: Discovered 2 actions, 1 trees, 0 blobs
-INFO: Building [["@","tutorial","","helloworld"],{"CXX":"clang++"}].
-INFO: Processed 2 actions, 0 cache hits.
-INFO: Artifacts built, logical paths are:
- helloworld [b8cf7b8579d9dc7172b61660139e2c14521cedae:16944:x]
-$
-#+END_SRC
-
-** Defining project defaults
-
-To define a custom set of defaults (toolchain and compile flags) for your
-project, you need to create a separate file root for providing required
-~TARGETS~ file, which contains the ~"defaults"~ target that should be used by
-the rules. This file root is then used as the /target root/ for the rules, i.e.,
-the search path for ~TARGETS~ files. In this way, the description of the
-~"defaults"~ target is provided in a separate file root, to keep the rules
-repository independent of these definitions.
-
-We will call the new file root ~tutorial-defaults~ and need to create a module
-directory ~CC~ in it:
-
-#+BEGIN_SRC sh
-$ mkdir -p ./tutorial-defaults/CC
-#+END_SRC
-
-In that module, we need to create the file ~tutorial-defaults/CC/TARGETS~ that
-contains the target ~"defaults"~ and specifies which toolchain and compile flags
-to use; it has to specify the complete toolchain, but can specify a ~"base"~
-toolchain to inherit from. In our case, we don't use any base, but specify all
-the required fields directly.
-
-#+SRCNAME: tutorial-defaults/CC/TARGETS
-#+BEGIN_SRC js
-{ "defaults":
- { "type": ["CC", "defaults"]
- , "CC": ["cc"]
- , "CXX": ["c++"]
- , "CFLAGS": ["-O2", "-Wall"]
- , "CXXFLAGS": ["-O2", "-Wall"]
- , "AR": ["ar"]
- , "PATH": ["/bin", "/usr/bin"]
- }
-}
-#+END_SRC
-
-To use the project defaults, modify the existing ~repos.json~ to reflect the
-following content:
-
-#+SRCNAME: repos.json
-#+BEGIN_SRC js
-{ "main": "tutorial"
-, "repositories":
- { "rules-cc":
- { "repository":
- { "type": "git"
- , "branch": "master"
- , "commit": "123d8b03bf2440052626151c14c54abce2726e6f"
- , "repository": "https://github.com/just-buildsystem/rules-cc.git"
- , "subdir": "rules"
- }
- , "target_root": "tutorial-defaults"
- , "rule_root": "rules-cc"
- }
- , "tutorial":
- { "repository": {"type": "file", "path": "."}
- , "bindings": {"rules": "rules-cc"}
- }
- , "tutorial-defaults":
- { "repository": {"type": "file", "path": "./tutorial-defaults"}
- }
- }
-}
-#+END_SRC
-
-Note that the ~"defaults"~ target uses the rule ~["CC", "defaults"]~ without
-specifying any external repository (e.g., ~["@", "rules", ...]~). This is
-because ~"tutorial-defaults"~ is not a full-fledged repository but merely a file
-root that is considered local to the ~"rules-cc"~ repository. In fact, the
-~"rules-cc"~ repository cannot refer to any external repository as it does not
-have any defined bindings.
-
-To rebuild the project, we need to rerun ~just-mr~ (note that due to
-configuration changes, rerunning only ~just~ would not suffice):
-
-#+BEGIN_SRC sh
-$ just-mr build helloworld
-INFO: Requested target is [["@","tutorial","","helloworld"],{}]
-INFO: Analysed target [["@","tutorial","","helloworld"],{}]
-INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
-INFO: Discovered 2 actions, 1 trees, 0 blobs
-INFO: Building [["@","tutorial","","helloworld"],{}].
-INFO: Processed 2 actions, 0 cache hits.
-INFO: Artifacts built, logical paths are:
- helloworld [487dc9e47b978877ed2f7d80b3395ce84b23be92:16992:x]
-$
-#+END_SRC
-
-Note that the output binary may have changed due to different defaults.
-
-** Modeling target dependencies
-
-For demonstration purposes, we will separate the print statements into a static
-library ~greet~, which will become a dependency to our binary. Therefore, we
-create a new subdirectory ~greet~ with the files ~greet/greet.hpp~:
-
-#+SRCNAME: greet/greet.hpp
-#+BEGIN_SRC cpp
-#include <string>
-
-void greet(std::string const& s);
-#+END_SRC
-
-and ~greet/greet.cpp~:
-
-#+SRCNAME: greet/greet.cpp
-#+BEGIN_SRC cpp
-#include "greet.hpp"
-#include <iostream>
-
-void greet(std::string const& s) {
- std::cout << "Hello " << s << "!\n";
-}
-#+END_SRC
-
-These files can now be used to create a static library ~libgreet.a~. To do so,
-we need to create the following target description in ~greet/TARGETS~:
-
-#+SRCNAME: greet/TARGETS
-#+BEGIN_SRC js
-{ "greet":
- { "type": ["@", "rules", "CC", "library"]
- , "name": ["greet"]
- , "hdrs": ["greet.hpp"]
- , "srcs": ["greet.cpp"]
- , "stage": ["greet"]
- }
-}
-#+END_SRC
-
-Similar to ~"binary"~, we have to provide a name and source file. Additionally,
-a library has public headers defined via ~"hdrs"~ and an optional staging
-directory ~"stage"~ (default value ~"."~). The staging directory specifies where
-the consumer of this library can expect to find the library's artifacts. Note
-that this does not need to reflect the location on the file system (i.e., a
-full-qualified path like ~["com", "example", "utils", "greet"]~ could be used to
-distinguish it from greeting libraries of other projects). The staging directory
-does not only affect the main artifact ~libgreet.a~ but also it's /runfiles/,
-a second set of artifacts, usually those a consumer needs to make proper use the
-actual artifact; in the case of a library, the runfiles are its public headers.
-Hence, the public header will be staged to ~"greet/greet.hpp"~. With that
-knowledge, we can now perform the necessary modifications to ~main.cpp~:
-
-#+SRCNAME: main.cpp
-#+BEGIN_SRC cpp
-#include "greet/greet.hpp"
-
-int main() {
- greet("Universe");
- return 0;
-}
-#+END_SRC
-
-The target ~"helloworld"~ will have a direct dependency to the target ~"greet"~
-of the module ~"greet"~ in the top-level ~TARGETS~ file:
-
-#+SRCNAME: TARGETS
-#+BEGIN_SRC js
-{ "helloworld":
- { "type": ["@", "rules", "CC", "binary"]
- , "name": ["helloworld"]
- , "srcs": ["main.cpp"]
- , "private-deps": [["greet", "greet"]]
- }
-}
-#+END_SRC
-
-Note that there is no need to explicitly specify ~"greet"~'s public headers here
-as the appropriate artifacts of dependencies are automatically added to the
-inputs of compile and link actions. The new binary can be built with the same
-command as before (no need to rerun ~just-mr~):
-
-#+BEGIN_SRC sh
-$ just-mr build helloworld
-INFO: Requested target is [["@","tutorial","","helloworld"],{}]
-INFO: Analysed target [["@","tutorial","","helloworld"],{}]
-INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
-INFO: Discovered 4 actions, 2 trees, 0 blobs
-INFO: Building [["@","tutorial","","helloworld"],{}].
-INFO: Processed 4 actions, 0 cache hits.
-INFO: Artifacts built, logical paths are:
- helloworld [2b81e3177afc382452a2df9f294d3df90a9ccaf0:17664:x]
-$
-#+END_SRC
-
-To only build the static library target ~"greet"~ from module ~"greet"~, run the
-following command:
-
-#+BEGIN_SRC sh
-$ just-mr build greet greet
-INFO: Requested target is [["@","tutorial","greet","greet"],{}]
-INFO: Analysed target [["@","tutorial","greet","greet"],{}]
-INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
-INFO: Discovered 2 actions, 1 trees, 0 blobs
-INFO: Building [["@","tutorial","greet","greet"],{}].
-INFO: Processed 2 actions, 2 cache hits.
-INFO: Artifacts built, logical paths are:
- greet/libgreet.a [83ed406e21f285337b0c9bd5011f56f656bba683:2992:f]
- (1 runfiles omitted.)
-$
-#+END_SRC
diff --git a/doc/tutorial/proto.org b/doc/tutorial/proto.md
index b4a02d48..8a04e373 100644
--- a/doc/tutorial/proto.org
+++ b/doc/tutorial/proto.md
@@ -1,27 +1,28 @@
-* Using protocol buffers
+Using protocol buffers
+======================
-The rules /justbuild/ uses for itself also support protocol
-buffers. This tutorial shows how to use those rules and the targets
-associated with them. It is not a tutorial on protocol buffers
-itself; rather, it is assumed that the reader has some knowledge on
-[[https://developers.google.com/protocol-buffers/][protocol buffers]].
+The rules *justbuild* uses for itself also support protocol buffers.
+This tutorial shows how to use those rules and the targets associated
+with them. It is not a tutorial on protocol buffers itself; rather, it
+is assumed that the reader has some knowledge on [protocol
+buffers](https://developers.google.com/protocol-buffers/).
-** Setting up the repository configuration
+Setting up the repository configuration
+---------------------------------------
-Before we begin, we first need to declare where the root of our workspace is
-located by creating the empty file ~ROOT~:
+Before we begin, we first need to declare where the root of our
+workspace is located by creating the empty file `ROOT`:
-#+BEGIN_SRC sh
+``` sh
$ touch ROOT
-#+END_SRC
+```
-The ~protobuf~ repository conveniently contains an
-[[https://github.com/protocolbuffers/protobuf/tree/v3.12.4/examples][example]],
-so we can use this and just add our own target files. We create
-file ~repos.template.json~ as follows.
+The `protobuf` repository conveniently contains an
+[example](https://github.com/protocolbuffers/protobuf/tree/v3.12.4/examples),
+so we can use this and just add our own target files. We create file
+`repos.template.json` as follows.
-#+SRCNAME: repos.template.json
-#+BEGIN_SRC js
+``` {.jsonc srcname="repos.template.json"}
{ "repositories":
{ "":
{ "repository":
@@ -36,45 +37,45 @@ file ~repos.template.json~ as follows.
, "tutorial": {"repository": {"type": "file", "path": "."}}
}
}
-#+END_SRC
+```
-The missing entry ~"rules-cc"~ refers to our C/C++ build rules provided
-[[https://github.com/just-buildsystem/rules-cc][online]]. These rules support
-protobuf if the dependency ~"protoc"~ is provided. To import this rule
-repository including the required transitive dependencies for protobuf, the
-~bin/just-import-git~ script with option ~--as rules-cc~ can be used to
-generate the actual ~repos.json~:
+The missing entry `"rules-cc"` refers to our C/C++ build rules provided
+[online](https://github.com/just-buildsystem/rules-cc). These rules
+support protobuf if the dependency `"protoc"` is provided. To import
+this rule repository including the required transitive dependencies for
+protobuf, the `bin/just-import-git` script with option `--as rules-cc`
+can be used to generate the actual `repos.json`:
-#+BEGIN_SRC sh
+``` sh
$ just-import-git -C repos.template.json -b master --as rules-cc https://github.com/just-buildsystem/rules-cc > repos.json
-#+END_SRC
+```
-To build the example with ~just~, the only task is to write targets files. As
-that contains a couple of new concepts, we will do this step by step.
+To build the example with `just`, the only task is to write targets
+files. As that contains a couple of new concepts, we will do this step
+by step.
-** The proto library
+The proto library
+-----------------
First, we have to declare the proto library. In this case, it only
-contains the file ~addressbook.proto~ and has no dependencies. To
-declare the library, create a ~TARGETS~ file with the following
-content:
+contains the file `addressbook.proto` and has no dependencies. To
+declare the library, create a `TARGETS` file with the following content:
-#+SRCNAME: TARGETS
-#+BEGIN_SRC js
+``` {.jsonc srcname="TARGETS"}
{ "address":
{ "type": ["@", "rules", "proto", "library"]
, "name": ["addressbook"]
, "srcs": ["addressbook.proto"]
}
}
-#+END_SRC
+```
In general, proto libraries could also depend on other proto libraries;
-those would be added to the ~"deps"~ field.
+those would be added to the `"deps"` field.
When building the library, there's very little to do.
-#+BEGIN_SRC sh
+``` sh
$ just-mr build address
INFO: Requested target is [["@","","","address"],{}]
INFO: Analysed target [["@","","","address"],{}]
@@ -84,20 +85,21 @@ INFO: Building [["@","","","address"],{}].
INFO: Processed 0 actions, 0 cache hits.
INFO: Artifacts built, logical paths are:
$
-#+END_SRC
+```
On the other hand, what did we expect? A proto library is an abstract
description of a protocol, so, as long as we don't specify for which
language we want to have bindings, there is nothing to generate.
-Nevertheless, a proto library target is not empty. In fact, it can't be empty,
-as other targets can only access the values of a target and have no
-insights into its definitions. We already relied on this design principle
-implicitly, when we exploited target-level caching for our external dependencies
-and did not even construct the dependency graph for that target. A proto
-library simply provides the dependency structure of the ~.proto~ files.
+Nevertheless, a proto library target is not empty. In fact, it can't be
+empty, as other targets can only access the values of a target and have
+no insights into its definitions. We already relied on this design
+principle implicitly, when we exploited target-level caching for our
+external dependencies and did not even construct the dependency graph
+for that target. A proto library simply provides the dependency
+structure of the `.proto` files.
-#+BEGIN_SRC sh
+``` sh
$ just-mr analyse --dump-nodes - address
INFO: Requested target is [["@","","","address"],{}]
INFO: Result of target [["@","","","address"],{}]: {
@@ -146,36 +148,35 @@ INFO: Target nodes of target [["@","","","address"],{}]:
}
}
$
-#+END_SRC
+```
-The target has one provider ~"proto"~, which is a node. Nodes are
-an abstract representation of a target graph. More precisely, there
-are two kind of nodes, and our example contains one of each.
+The target has one provider `"proto"`, which is a node. Nodes are an
+abstract representation of a target graph. More precisely, there are two
+kind of nodes, and our example contains one of each.
-The simple kind of nodes are the value nodes; they represent a
-target that has a fixed value, and hence are given by artifacts,
-runfiles, and provided data. In our case, we have one value node,
-the one for the ~.proto~ file.
+The simple kind of nodes are the value nodes; they represent a target
+that has a fixed value, and hence are given by artifacts, runfiles, and
+provided data. In our case, we have one value node, the one for the
+`.proto` file.
The other kind of nodes are the abstract nodes. They describe the
-arguments for a target, but only have an abstract name (i.e., a
-string) for the rule. Combining such an abstract target with a
-binding for the abstract rule names gives a concrete "anonymous"
-target that, in our case, will generate the library with the bindings
-for the concrete language. In this example, the abstract name is
-~"library"~. The alternative in our proto rules would have been
-~"service library"~, for proto libraries that also contain ~rpc~
-definitions (which is used by [[https://grpc.io/][gRPC]]).
-
-** Using proto libraries
-
-Using proto libraries requires, as discussed, bindings for the
-abstract names. Fortunately, our ~CC~ rules are aware of proto
-libraries, so we can simply use them. Our target file hence
-continues as follows.
-
-#+SRCNAME: TARGETS
-#+BEGIN_SRC js
+arguments for a target, but only have an abstract name (i.e., a string)
+for the rule. Combining such an abstract target with a binding for the
+abstract rule names gives a concrete "anonymous" target that, in our
+case, will generate the library with the bindings for the concrete
+language. In this example, the abstract name is `"library"`. The
+alternative in our proto rules would have been `"service library"`, for
+proto libraries that also contain `rpc` definitions (which is used by
+[gRPC](https://grpc.io/)).
+
+Using proto libraries
+---------------------
+
+Using proto libraries requires, as discussed, bindings for the abstract
+names. Fortunately, our `CC` rules are aware of proto libraries, so we
+can simply use them. Our target file hence continues as follows.
+
+``` {.jsonc srcname="TARGETS"}
...
, "add_person":
{ "type": ["@", "rules", "CC", "binary"]
@@ -190,14 +191,14 @@ continues as follows.
, "private-proto": ["address"]
}
...
-#+END_SRC
+```
-The first time, we build a target that requires the proto compiler
-(in that particular version, built in that particular way), it takes
-a bit of time, as the proto compiler has to be built. But in follow-up
-builds, also in different projects, the target-level cache is filled already.
+The first time, we build a target that requires the proto compiler (in
+that particular version, built in that particular way), it takes a bit
+of time, as the proto compiler has to be built. But in follow-up builds,
+also in different projects, the target-level cache is filled already.
-#+BEGIN_SRC sh
+``` sh
$ just-mr build add_person
...
$ just-mr build add_person
@@ -210,12 +211,12 @@ INFO: Processed 5 actions, 5 cache hits.
INFO: Artifacts built, logical paths are:
add_person [bcbb3deabfe0d77e6d3ea35615336a2f59a1b0aa:2285928:x]
$
-#+END_SRC
+```
If we look at the actions associated with the binary, we find that those
are still the two actions we expect: a compile action and a link action.
-#+BEGIN_SRC sh
+``` sh
$ just-mr analyse add_person --dump-actions -
INFO: Requested target is [["@","","","add_person"],{}]
INFO: Result of target [["@","","","add_person"],{}]: {
@@ -251,17 +252,17 @@ INFO: Actions for target [["@","","","add_person"],{}]:
}
]
$
-#+END_SRC
+```
-As discussed, the ~libaddressbook.a~ that is conveniently available
-during the linking of the binary (as well as the ~addressbook.pb.h~
-available in the ~include~ tree for the compile action) are generated
-by an anonymous target. Using that during the build we already
-filled the target-level cache, we can have a look at all targets
-still analysed. In the one anonymous target, we find again the
-abstract node we discussed earlier.
+As discussed, the `libaddressbook.a` that is conveniently available
+during the linking of the binary (as well as the `addressbook.pb.h`
+available in the `include` tree for the compile action) are generated by
+an anonymous target. Using that during the build we already filled the
+target-level cache, we can have a look at all targets still analysed. In
+the one anonymous target, we find again the abstract node we discussed
+earlier.
-#+BEGIN_SRC sh
+``` sh
$ just-mr analyse add_person --dump-targets -
INFO: Requested target is [["@","","","add_person"],{}]
INFO: Result of target [["@","","","add_person"],{}]: {
@@ -302,25 +303,24 @@ INFO: List of analysed targets:
}
}
$
-#+END_SRC
-
-It should be noted, however, that this tight integration of proto
-into our ~C++~ rules is just convenience of our code base. If we had
-to cooperate with rules not aware of proto, we could have created
-a separate rule delegating the library creation to the anonymous
-target and then simply reflecting the values of that target.
-In fact, we could simply use an empty library with a public ~proto~
-dependency for this purpose.
-
-#+SRCNAME: TARGETS
-#+BEGIN_SRC js
+```
+
+It should be noted, however, that this tight integration of proto into
+our `C++` rules is just convenience of our code base. If we had to
+cooperate with rules not aware of proto, we could have created a
+separate rule delegating the library creation to the anonymous target
+and then simply reflecting the values of that target. In fact, we could
+simply use an empty library with a public `proto` dependency for this
+purpose.
+
+``` {.jsonc srcname="TARGETS"}
...
, "address proto library":
{"type": ["@", "rules", "CC", "library"], "proto": ["address"]}
...
-#+END_SRC
+```
-#+BEGIN_SRC sh
+``` sh
$ just-mr analyse 'address proto library'
...
INFO: Requested target is [["@","","","address proto library"],{}]
@@ -347,18 +347,18 @@ INFO: Result of target [["@","","","address proto library"],{}]: {
}
}
$
-#+END_SRC
+```
-** Adding a test
+Adding a test
+-------------
-Finally, let's add a test. As we use the ~protobuf~ repository as
-workspace root, we add the test script ad hoc into a targets file,
-using the ~"file_gen"~ rule. For debugging a potentially failing
-test, we also keep the intermediate files the test generates.
-Create a top-level ~TARGETS~ file with the following content:
+Finally, let's add a test. As we use the `protobuf` repository as
+workspace root, we add the test script ad hoc into a targets file, using
+the `"file_gen"` rule. For debugging a potentially failing test, we also
+keep the intermediate files the test generates. Create a top-level
+`TARGETS` file with the following content:
-#+SRCNAME: TARGETS
-#+BEGIN_SRC js
+``` {.jsonc srcname="TARGETS"}
...
, "test.sh":
{ "type": "file_gen"
@@ -382,17 +382,16 @@ Create a top-level ~TARGETS~ file with the following content:
, "keep": ["addressbook.data", "out.txt"]
}
...
-#+END_SRC
+```
-That example also shows why it is important that the generation
-of the language bindings is delegated to an anonymous target: we
-want to analyse only once how the ~C++~ bindings are generated.
-Nevertheless, many targets can depend (directly or indirectly) on
-the same proto library. And, indeed, analysing the test, we get
-the expected additional targets and the one anonymous target is
-reused by both binaries.
+That example also shows why it is important that the generation of the
+language bindings is delegated to an anonymous target: we want to
+analyse only once how the `C++` bindings are generated. Nevertheless,
+many targets can depend (directly or indirectly) on the same proto
+library. And, indeed, analysing the test, we get the expected additional
+targets and the one anonymous target is reused by both binaries.
-#+BEGIN_SRC sh
+``` sh
$ just-mr analyse test --dump-targets -
INFO: Requested target is [["@","","","test"],{}]
INFO: Result of target [["@","","","test"],{}]: {
@@ -444,11 +443,11 @@ INFO: List of analysed targets:
}
INFO: Target tainted ["test"].
$
-#+END_SRC
+```
Finally, the test passes and the output is as expected.
-#+BEGIN_SRC sh
+``` sh
$ just-mr build test -Pwork/out.txt
INFO: Requested target is [["@","","","test"],{}]
INFO: Analysed target [["@","","","test"],{}]
@@ -472,4 +471,4 @@ Person ID: 12345
Updated: 2022-12-14T18:08:36Z
INFO: Target tainted ["test"].
$
-#+END_SRC
+```
diff --git a/doc/tutorial/rebuild.org b/doc/tutorial/rebuild.md
index 80aafb6f..3f1ddd88 100644
--- a/doc/tutorial/rebuild.org
+++ b/doc/tutorial/rebuild.md
@@ -1,15 +1,17 @@
-* Ensuring reproducibility of the build
-
-Software builds should be [[https://reproducible-builds.org/][reproducible]].
-The ~just~ tool, supports this goal in local builds by isolating
-individual actions, setting permissions and file time stamps to
-canonical values, etc; most remote execution systems take even further
-measures to ensure the environment always looks the same to every
-action. Nevertheless, it is always possible to break reproducibility
-by bad actions, both coming from rules not carefully written, as
-well as from ad-hoc actions added by the ~generic~ target.
-
-#+BEGIN_SRC js
+Ensuring reproducibility of the build
+=====================================
+
+Software builds should be
+[reproducible](https://reproducible-builds.org/). The `just` tool,
+supports this goal in local builds by isolating individual actions,
+setting permissions and file time stamps to canonical values, etc; most
+remote execution systems take even further measures to ensure the
+environment always looks the same to every action. Nevertheless, it is
+always possible to break reproducibility by bad actions, both coming
+from rules not carefully written, as well as from ad-hoc actions added
+by the `generic` target.
+
+``` jsonc
...
, "version.h":
{ "type": "generic"
@@ -18,29 +20,29 @@ well as from ad-hoc actions added by the ~generic~ target.
, "outs": ["version.h"]
}
...
-#+END_SRC
-
-Besides time stamps there are many other sources of nondeterminism,
-like properties of the build machine (name, number of CPUs available,
-etc), but also subtle ones like ~readdir~ order. Often, those
-non-reproducible parts get buried deeply in a final artifact (like
-the version string embedded in a binary contained in a compressed
-installation archive); and, as long as the non-reproducible action
-stays in cache, it does not even result in bad incrementality.
-Still, others won't be able to reproduce the exact artifact.
-
-There are tools like [[https://diffoscope.org/][diffoscope]] to deeply
+```
+
+Besides time stamps there are many other sources of nondeterminism, like
+properties of the build machine (name, number of CPUs available, etc),
+but also subtle ones like `readdir` order. Often, those non-reproducible
+parts get buried deeply in a final artifact (like the version string
+embedded in a binary contained in a compressed installation archive);
+and, as long as the non-reproducible action stays in cache, it does not
+even result in bad incrementality. Still, others won't be able to
+reproduce the exact artifact.
+
+There are tools like [diffoscope](https://diffoscope.org/) to deeply
compare archives and other container formats. Nevertheless, it is
desirable to find the root causes, i.e., the first (in topological
order) actions that yield a different output.
-** Rebuilding
+Rebuilding
+----------
-For the remainder of this section, we will consider the following example
-project with the C++ source file ~hello.cpp~:
+For the remainder of this section, we will consider the following
+example project with the C++ source file `hello.cpp`:
-#+SRCNAME: hello.cpp
-#+BEGIN_SRC cpp
+``` {.cpp srcname="hello.cpp"}
#include <iostream>
#include "version.h"
@@ -50,12 +52,11 @@ int main(int argc, const char* argv[]) {
}
return 0;
}
-#+END_SRC
+```
-and the following ~TARGETS~ file:
+and the following `TARGETS` file:
-#+SRCNAME: TARGETS
-#+BEGIN_SRC js
+``` {.jsonc srcname="TARGETS"}
{ "":
{ "type": "install"
, "files":
@@ -95,17 +96,17 @@ and the following ~TARGETS~ file:
, "deps": ["out.txt"]
}
}
-#+END_SRC
+```
-To search for the root cause of non-reproducibility, ~just~ has
-a subcommand ~rebuild~. It builds the specified target again, requesting
+To search for the root cause of non-reproducibility, `just` has a
+subcommand `rebuild`. It builds the specified target again, requesting
that every action be executed again (but target-level cache is still
active); then the result of every action is compared to the one in the
action cache, if present with the same inputs. So, you typically would
-first ~build~ and then ~rebuild~. Note that a repeated ~build~ simply
+first `build` and then `rebuild`. Note that a repeated `build` simply
takes the action result from cache.
-#+BEGIN_SRC sh
+``` sh
$ just-mr build
INFO: Requested target is [["@","tutorial","",""],{}]
INFO: Analysed target [["@","tutorial","",""],{}]
@@ -135,30 +136,31 @@ INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
INFO: Discovered 6 actions, 1 trees, 0 blobs
INFO: Rebuilding [["@","tutorial","",""],{}].
WARN: Found flaky action:
- - id: c854a382ea26628e1a5b8d4af00d6d0cef433436
- - cmd: ["sh","-c","echo '#define VERSION \"0.0.0.'`date +%Y%m%d%H%M%S`'\"' > version.h\n"]
- - output 'version.h' differs:
- - [6aac3477e22cd57e8c98ded78562d3c017e5d611:39:f] (rebuilt)
- - [789a29f39b6aa966f91776bfe092e247614e6acd:39:f] (cached)
+ - id: c854a382ea26628e1a5b8d4af00d6d0cef433436
+ - cmd: ["sh","-c","echo '#define VERSION \"0.0.0.'`date +%Y%m%d%H%M%S`'\"' > version.h\n"]
+ - output 'version.h' differs:
+ - [6aac3477e22cd57e8c98ded78562d3c017e5d611:39:f] (rebuilt)
+ - [789a29f39b6aa966f91776bfe092e247614e6acd:39:f] (cached)
INFO: 2 actions compared with cache, 1 flaky actions found (0 of which tainted), no cache entry found for 4 actions.
INFO: Artifacts built, logical paths are:
bin/hello [73994ff43ec1161aba96708f277e8c88feab0386:16608:x]
share/hello/OUT.txt [428b97b82b6c59cad7488b24e6b618ebbcd819bc:13:f]
share/hello/version.txt [8dd65747395c0feab30891eab9e11d4a9dd0c715:39:f]
$
-#+END_SRC
+```
-In the example, the second action compared to cache is the upper
-casing of the output. Even though the generation of ~out.txt~ depends
-on the non-reproducible ~hello~, the file itself is reproducible.
-Therefore, the follow-up actions are checked as well.
+In the example, the second action compared to cache is the upper casing
+of the output. Even though the generation of `out.txt` depends on the
+non-reproducible `hello`, the file itself is reproducible. Therefore,
+the follow-up actions are checked as well.
-For this simple example, reading the console output is enough to understand
-what's going on. However, checking for reproducibility usually is part
-of a larger, quality-assurance process. To support the automation of such
-processes, the findings can also be reported in machine-readable form.
+For this simple example, reading the console output is enough to
+understand what's going on. However, checking for reproducibility
+usually is part of a larger, quality-assurance process. To support the
+automation of such processes, the findings can also be reported in
+machine-readable form.
-#+BEGIN_SRC sh
+``` sh
$ just-mr rebuild --dump-flaky flakes.json --dump-graph actions.json
[...]
$ cat flakes.json
@@ -186,40 +188,40 @@ $ cat flakes.json
}
}
}$
-#+END_SRC
+```
The file reports the flaky actions together with the non-reproducible
artifacts they generated, reporting both, the cached and the newly
-generated output. The files themselves can be obtained via ~just
-install-cas~ as usual, allowing deeper comparison of the outputs.
-The full definitions of the actions can be found in the action graph,
-in the example dumped as well as ~actions.json~; this definition
-also includes the origins for each action, i.e., the configured
-targets that requested the respective action.
+generated output. The files themselves can be obtained via `just
+install-cas` as usual, allowing deeper comparison of the outputs. The
+full definitions of the actions can be found in the action graph, in the
+example dumped as well as `actions.json`; this definition also includes
+the origins for each action, i.e., the configured targets that requested
+the respective action.
-
-** Comparing build environments
+Comparing build environments
+----------------------------
Simply rebuilding on the same machine is good way to detect embedded
time stamps of sufficiently small granularity; for other sources of
-non-reproducibility, however, more modifications of the environment
-are necessary.
-
-A simple, but effective, way for modifying the build environment
-is the option ~-L~ to set the local launcher, a list of
-strings the argument vector is prefixed with before the action is
-executed. The default ~["env", "--"]~ simply resolves the program
-to be executed in the current value of ~PATH~, but a different
-value for the launcher can obviously be used to set environment
-variables like ~LD_PRELOAD~. Relevant libraries and tools
-include [[https://github.com/wolfcw/libfaketime][libfaketime]],
-[[https://github.com/dtcooper/fakehostname][fakehostname]],
-and [[https://salsa.debian.org/reproducible-builds/disorderfs][disorderfs]].
+non-reproducibility, however, more modifications of the environment are
+necessary.
+
+A simple, but effective, way for modifying the build environment is the
+option `-L` to set the local launcher, a list of strings the argument
+vector is prefixed with before the action is executed. The default
+`["env", "--"]` simply resolves the program to be executed in the
+current value of `PATH`, but a different value for the launcher can
+obviously be used to set environment variables like `LD_PRELOAD`.
+Relevant libraries and tools include
+[libfaketime](https://github.com/wolfcw/libfaketime),
+[fakehostname](https://github.com/dtcooper/fakehostname), and
+[disorderfs](https://salsa.debian.org/reproducible-builds/disorderfs).
More variation can be achieved by comparing remote execution builds,
-either for two different remote-execution end points or comparing
-one remote-execution end point to the local build. The latter is
-also a good way to find out where a build that "works on my machine"
-differs. The endpoint on which the rebuild is executed can be set,
-in the same way as for build with the ~-r~ option; the cache end
-point to compare against can be set via the ~--vs~ option.
+either for two different remote-execution end points or comparing one
+remote-execution end point to the local build. The latter is also a good
+way to find out where a build that "works on my machine" differs. The
+endpoint on which the rebuild is executed can be set, in the same way as
+for build with the `-r` option; the cache end point to compare against
+can be set via the `--vs` option.
diff --git a/doc/tutorial/target-file-glob-tree.org b/doc/tutorial/target-file-glob-tree.md
index 58e9c725..524cf358 100644
--- a/doc/tutorial/target-file-glob-tree.org
+++ b/doc/tutorial/target-file-glob-tree.md
@@ -1,34 +1,35 @@
-* Target versus ~FILE~, ~GLOB~, and ~TREE~
+Target versus `FILE`, `GLOB`, and `TREE`
+========================================
-So far, we referred to defined targets as well as source files
-by their name and it just worked. When considering third-party
-software we already saw the ~TREE~ reference. In this section, we
-will highlight in more detail the ways to refer to sources, as well
-as the difference between defined and source targets. The latter
-is used, e.g., when third-party software has to be patched.
+So far, we referred to defined targets as well as source files by their
+name and it just worked. When considering third-party software we
+already saw the `TREE` reference. In this section, we will highlight in
+more detail the ways to refer to sources, as well as the difference
+between defined and source targets. The latter is used, e.g., when
+third-party software has to be patched.
-As example for this section we use gnu ~units~ where we want to
-patch into the standard units definition add two units of area
-popular in German news.
+As example for this section we use gnu `units` where we want to patch
+into the standard units definition add two units of area popular in
+German news.
-** Repository Config for ~units~ with patches
+Repository Config for `units` with patches
+------------------------------------------
-Before we begin, we first need to declare where the root of our workspace is
-located by creating the empty file ~ROOT~:
+Before we begin, we first need to declare where the root of our
+workspace is located by creating the empty file `ROOT`:
-#+BEGIN_SRC sh
+``` sh
$ touch ROOT
-#+END_SRC
+```
The sources are an archive available on the web. As upstream uses a
-different build system, we have to provide our own build description;
-we take the top-level directory as layer for this. As we also want
-to patch the definition file, we add the subdirectory ~files~ as
-logical repository for the patches. Hence we create a file ~repos.json~
-with the following content.
-
-#+SRCNAME: repos.json
-#+BEGIN_SRC js
+different build system, we have to provide our own build description; we
+take the top-level directory as layer for this. As we also want to patch
+the definition file, we add the subdirectory `files` as logical
+repository for the patches. Hence we create a file `repos.json` with the
+following content.
+
+``` {.jsonc srcname="repos.json"}
{ "main": "units"
, "repositories":
{ "rules-cc":
@@ -55,31 +56,33 @@ with the following content.
}
}
}
-#+END_SRC
+```
-The repository to set up is ~units~ and, as usual, we can use ~just-mr~ to
-fetch the archive and obtain the resulting multi-repository configuration.
+The repository to set up is `units` and, as usual, we can use `just-mr`
+to fetch the archive and obtain the resulting multi-repository
+configuration.
-#+BEGIN_SRC sh
+``` sh
$ just-mr setup units
-#+END_SRC
+```
-** Patching a file: targets versus ~FILE~
+Patching a file: targets versus `FILE`
+--------------------------------------
-Let's start by patching the source file ~definitions.units~. While,
-conceptionally, we want to patch a third-party source file, we do /not/
+Let's start by patching the source file `definitions.units`. While,
+conceptionally, we want to patch a third-party source file, we do *not*
modify the sources. The workspace root is a git tree and stay like this.
-Instead, we remember that we specify /targets/ and the definition of a
+Instead, we remember that we specify *targets* and the definition of a
target is looked up in the targets file; only if not defined there, it
is implicitly considered a source target and taken from the target root.
-So we will define a /target/ named ~definitions.units~ to replace the
+So we will define a *target* named `definitions.units` to replace the
original source file.
-Let's first generate the patch. As we're already referring to source files
-as targets, we have to provide a targets file already; we start with the
-empty object and refine it later.
+Let's first generate the patch. As we're already referring to source
+files as targets, we have to provide a targets file already; we start
+with the empty object and refine it later.
-#+BEGIN_SRC sh
+``` sh
$ echo {} > TARGETS.units
$ just-mr install -o . definitions.units
INFO: Requested target is [["@","units","","definitions.units"],{}]
@@ -100,41 +103,39 @@ $ mkdir files
$ echo {} > files/TARGETS
$ diff -u definitions.units.orig definitions.units > files/definitions.units.diff
$ rm definitions.units*
-#+END_SRC
-
-Our rules conveniently contain a rule ~["patch", "file"]~ to patch
-a single file, and we already created the patch. The only other
-input missing is the source file. So far, we could refer to it as
-~"definitions.units"~ because there was no target of that name, but
-now we're about to define a target with that very name. Fortunately,
-in target files, we can use a special syntax to explicitly refer to
-a source file of the current module, even if there is a target with
-the same name: ~["FILE", null, "definition.units"]~. The syntax
-requires the explicit ~null~ value for the current module, despite
-the fact that explicit file references are only allowed for the
-current module; in this way, the name is a list of length more than
-two and cannot be confused with a top-level module called ~FILE~.
-So we add this target and obtain as ~TARGETS.units~ the following.
-
-#+SRCNAME: TARGETS.units
-#+BEGIN_SRC js
+```
+
+Our rules conveniently contain a rule `["patch", "file"]` to patch a
+single file, and we already created the patch. The only other input
+missing is the source file. So far, we could refer to it as
+`"definitions.units"` because there was no target of that name, but now
+we're about to define a target with that very name. Fortunately, in
+target files, we can use a special syntax to explicitly refer to a
+source file of the current module, even if there is a target with the
+same name: `["FILE", null, "definition.units"]`. The syntax requires the
+explicit `null` value for the current module, despite the fact that
+explicit file references are only allowed for the current module; in
+this way, the name is a list of length more than two and cannot be
+confused with a top-level module called `FILE`. So we add this target
+and obtain as `TARGETS.units` the following.
+
+``` {.jsonc srcname="TARGETS.units"}
{ "definitions.units":
{ "type": ["@", "rules", "patch", "file"]
, "src": [["FILE", ".", "definitions.units"]]
, "patch": [["@", "patches", "", "definitions.units.diff"]]
}
}
-#+END_SRC
+```
-Analysing ~"definitions.units"~ we find our defined target which
-contains an action output. Still, it looks like a patched source
-file; the new artifact is staged to the original location. Staging
-is also used in the action definition, to avoid magic names (like
-file names starting with ~-~), in-place operations (all actions
-must not modify their inputs) and, in fact, have a
-fixed command line.
+Analysing `"definitions.units"` we find our defined target which
+contains an action output. Still, it looks like a patched source file;
+the new artifact is staged to the original location. Staging is also
+used in the action definition, to avoid magic names (like file names
+starting with `-`), in-place operations (all actions must not modify
+their inputs) and, in fact, have a fixed command line.
-#+BEGIN_SRC sh
+``` sh
$ just-mr analyse definitions.units --dump-actions -
INFO: Requested target is [["@","units","","definitions.units"],{}]
INFO: Result of target [["@","units","","definitions.units"],{}]: {
@@ -172,11 +173,11 @@ INFO: Actions for target [["@","units","","definitions.units"],{}]:
}
]
$
-#+END_SRC
+```
-Building ~"definitions.units"~ we find out patch applied correctly.
+Building `"definitions.units"` we find out patch applied correctly.
-#+BEGIN_SRC sh
+``` sh
$ just-mr build definitions.units -P definitions.units | grep -A 5 'German units'
INFO: Requested target is [["@","units","","definitions.units"],{}]
INFO: Analysed target [["@","units","","definitions.units"],{}]
@@ -193,24 +194,24 @@ area_soccerfield 105 m * 68 m
area_saarland 2570 km^2
zentner 50 kg
$
-#+END_SRC
+```
-** Globbing source files: ~"GLOB"~
+Globbing source files: `"GLOB"`
+-------------------------------
-Next, we collect all ~.units~ files. We could simply do this by enumerating
-them in a target.
+Next, we collect all `.units` files. We could simply do this by
+enumerating them in a target.
-#+SRCNAME: TARGETS.units
-#+BEGIN_SRC js
+``` {.jsonc srcname="TARGETS.units"}
...
, "data-draft": { "type": "install", "deps": ["definitions.units", "currency.units"]}
...
-#+END_SRC
+```
-In this way, we get the desired collection of one unmodified source file and
-the output of the patch action.
+In this way, we get the desired collection of one unmodified source file
+and the output of the patch action.
-#+BEGIN_SRC sh
+``` sh
$ just-mr analyse data-draft
INFO: Requested target is [["@","units","","data-draft"],{}]
INFO: Result of target [["@","units","","data-draft"],{}]: {
@@ -226,77 +227,76 @@ INFO: Result of target [["@","units","","data-draft"],{}]: {
}
}
$
-#+END_SRC
-
-The disadvantage, however, that we might miss newly added ~.units~
-files if we update and upstream added new files. So we want all
-source files that have the respective ending. The corresponding
-source reference is ~"GLOB"~. A glob expands to the /collection/
-of all /sources/ that are /files/ in the /top-level/ directory of
-the current module and that match the given pattern. It is important
-to understand this in detail and the rational behind it.
-- First of all, the artifact (and runfiles) map has an entry for
- each file that matches. In particular, targets have the option to
- define individual actions for each file, like ~["CC", "binary"]~
- does for the source files. This is different from ~"TREE"~ where
- the artifact map contains a single artifact that happens to be a
- directory. The tree behaviour is preferable when the internals
- of the directory only matter for the execution of actions and not
- for analysis; then there are less entries to carry around during
- analysis and action-key computation, and the whole directory
- is "reserved" for that tree avoid staging conflicts when latter
- adding entries there.
-- As a source reference, a glob expands to explicit source files;
- targets having the same name as a source file are not taken into
- account. In our example, ~["GLOB", null, "*.units"]~ therefore
- contains the unpatched source file ~definitions.units~. In this
- way, we avoid any surprises in the expansion of a glob when a new
- source file is added with a name equal to an already existing target.
-- Only files are considered for matching the glob. Directories
- are ignored.
-- Matches are only considered at the top-level directory. In this
- way, only one directory has to be read during analysis; allowing
- deeper globs would require traversal of subdirectories requiring
- larger cost. While the explicit ~"TREE"~ reference allows recursive
- traversal, in the typical use case of the respective workspace root
- being a ~git~ root, it is actually cheap; we can look up the
- ~git~ tree identifier without traversing the tree. Such a quick
- look up would not be possible if matches had to be selected.
-
-So, ~["GLOB", null, "*.units"]~ expands to all the relevant source
-files; but we still want to keep the patching. Most rules, like ~"install"~,
-disallow staging conflicts to avoid accidentally ignoring a file due
-to conflicting name. In our case, however, the dropping of the source
-file in favour of the patched one is deliberate. For this, there is
-the rule ~["data", "overlay"]~ taking the union of the artifacts of
+```
+
+The disadvantage, however, that we might miss newly added `.units` files
+if we update and upstream added new files. So we want all source files
+that have the respective ending. The corresponding source reference is
+`"GLOB"`. A glob expands to the *collection* of all *sources* that are
+*files* in the *top-level* directory of the current module and that
+match the given pattern. It is important to understand this in detail
+and the rational behind it.
+
+ - First of all, the artifact (and runfiles) map has an entry for each
+ file that matches. In particular, targets have the option to define
+ individual actions for each file, like `["CC", "binary"]` does for
+ the source files. This is different from `"TREE"` where the artifact
+ map contains a single artifact that happens to be a directory. The
+ tree behaviour is preferable when the internals of the directory
+ only matter for the execution of actions and not for analysis; then
+ there are less entries to carry around during analysis and
+ action-key computation, and the whole directory is "reserved" for
+ that tree avoid staging conflicts when latter adding entries there.
+ - As a source reference, a glob expands to explicit source files;
+ targets having the same name as a source file are not taken into
+ account. In our example, `["GLOB", null, "*.units"]` therefore
+ contains the unpatched source file `definitions.units`. In this way,
+ we avoid any surprises in the expansion of a glob when a new source
+ file is added with a name equal to an already existing target.
+ - Only files are considered for matching the glob. Directories are
+ ignored.
+ - Matches are only considered at the top-level directory. In this way,
+ only one directory has to be read during analysis; allowing deeper
+ globs would require traversal of subdirectories requiring larger
+ cost. While the explicit `"TREE"` reference allows recursive
+ traversal, in the typical use case of the respective workspace root
+ being a `git` root, it is actually cheap; we can look up the `git`
+ tree identifier without traversing the tree. Such a quick look up
+ would not be possible if matches had to be selected.
+
+So, `["GLOB", null, "*.units"]` expands to all the relevant source
+files; but we still want to keep the patching. Most rules, like
+`"install"`, disallow staging conflicts to avoid accidentally ignoring a
+file due to conflicting name. In our case, however, the dropping of the
+source file in favour of the patched one is deliberate. For this, there
+is the rule `["data", "overlay"]` taking the union of the artifacts of
the specified targets, accepting conflicts and resolving them in a
-latest-wins fashion. Keep in mind, that our target fields are list,
-not sets. Looking at the definition of the rule, one finds that
-it is simply a ~"map_union"~. Hence we refine our ~"data"~ target.
+latest-wins fashion. Keep in mind, that our target fields are list, not
+sets. Looking at the definition of the rule, one finds that it is simply
+a `"map_union"`. Hence we refine our `"data"` target.
-#+SRCNAME: TARGETS.units
-#+BEGIN_SRC js
+``` {.jsonc srcname="TARGETS.units"}
...
, "data":
{ "type": ["@", "rules", "data", "overlay"]
, "deps": [["GLOB", null, "*.units"], "definitions.units"]
}
...
-#+END_SRC
+```
The result of the analysis, of course, still is the same.
-** Finishing the example: binaries from globbed sources
+Finishing the example: binaries from globbed sources
+----------------------------------------------------
-The source-code organisation of units is pretty simple. All source
-and header files are in the top-level directory. As the header files
-are not in a directory of their own, we can't use a tree, so we use
-a glob, which is fine for the private headers of a binary. For the
-source files, we have to have them individually anyway. So our first
-attempt of defining the binary is as follows.
+The source-code organisation of units is pretty simple. All source and
+header files are in the top-level directory. As the header files are not
+in a directory of their own, we can't use a tree, so we use a glob,
+which is fine for the private headers of a binary. For the source files,
+we have to have them individually anyway. So our first attempt of
+defining the binary is as follows.
-#+SRCNAME: TARGETS.units
-#+BEGIN_SRC js
+``` {.jsonc srcname="TARGETS.units"}
...
, "units-draft":
{ "type": ["@", "rules", "CC", "binary"]
@@ -307,12 +307,12 @@ attempt of defining the binary is as follows.
, "private-hdrs": [["GLOB", null, "*.h"]]
}
...
-#+END_SRC
+```
-The result basically work and shows that we have 5 source files in total,
-giving 5 compile and one link action.
+The result basically work and shows that we have 5 source files in
+total, giving 5 compile and one link action.
-#+BEGIN_SRC sh
+``` sh
$ just-mr build units-draft
INFO: Requested target is [["@","units","","units-draft"],{}]
INFO: Analysed target [["@","units","","units-draft"],{}]
@@ -328,13 +328,14 @@ INFO: Processed 6 actions, 0 cache hits.
INFO: Artifacts built, logical paths are:
units [718cb1489bd006082f966ea73e3fba3dd072d084:124488:x]
$
-#+END_SRC
+```
-To keep the build clean, we want to get rid of the warning. Of course, we could
-simply set an appropriate compiler flag, but let's do things properly and patch
-away the underlying reason. To do so, we first create a patch.
+To keep the build clean, we want to get rid of the warning. Of course,
+we could simply set an appropriate compiler flag, but let's do things
+properly and patch away the underlying reason. To do so, we first create
+a patch.
-#+BEGIN_SRC sh
+``` sh
$ just-mr install -o . strfunc.c
INFO: Requested target is [["@","units","","strfunc.c"],{}]
INFO: Analysed target [["@","units","","strfunc.c"],{}]
@@ -353,12 +354,11 @@ $ echo -e "109\ns|N|// N\nw\nq" | ed strfunc.c
$ diff strfunc.c.orig strfunc.c > files/strfunc.c.diff
$ rm strfunc.c*
$
-#+END_SRC
+```
-Then we amend our ~"units"~ target.
+Then we amend our `"units"` target.
-#+SRCNAME: TARGETS.units
-#+BEGIN_SRC js
+``` {.jsonc srcname="TARGETS.units"}
...
, "units":
{ "type": ["@", "rules", "CC", "binary"]
@@ -378,14 +378,15 @@ Then we amend our ~"units"~ target.
, "patch": [["@", "patches", "", "strfunc.c.diff"]]
}
...
-#+END_SRC
+```
-Building the new target, 2 actions have to be executed: the patching, and
-the compiling of the patched source file. As the patched file still generates
-the same object file as the unpatched file (after all, we only wanted to get
-rid of a warning), the linking step can be taken from cache.
+Building the new target, 2 actions have to be executed: the patching,
+and the compiling of the patched source file. As the patched file still
+generates the same object file as the unpatched file (after all, we only
+wanted to get rid of a warning), the linking step can be taken from
+cache.
-#+BEGIN_SRC sh
+``` sh
$ just-mr build units
INFO: Requested target is [["@","units","","units"],{}]
INFO: Analysed target [["@","units","","units"],{}]
@@ -396,22 +397,21 @@ INFO: Processed 7 actions, 5 cache hits.
INFO: Artifacts built, logical paths are:
units [718cb1489bd006082f966ea73e3fba3dd072d084:124488:x]
$
-#+END_SRC
+```
-To finish the example, we also add a default target (using that, if no target
-is specified, ~just~ builds the lexicographically first target), staging
-artifacts according to the usual conventions.
+To finish the example, we also add a default target (using that, if no
+target is specified, `just` builds the lexicographically first target),
+staging artifacts according to the usual conventions.
-#+SRCNAME: TARGETS.units
-#+BEGIN_SRC js
+``` {.jsonc srcname="TARGETS.units"}
...
, "": {"type": "install", "dirs": [["units", "bin"], ["data", "share/units"]]}
...
-#+END_SRC
+```
Then things work as expected
-#+BEGIN_SRC sh
+``` sh
$ just-mr install -o /tmp/testinstall
INFO: Requested target is [["@","units","",""],{}]
INFO: Analysed target [["@","units","",""],{}]
@@ -427,4 +427,4 @@ $ /tmp/testinstall/bin/units 'area_saarland' 'area_soccerfield'
* 359943.98
/ 2.7782101e-06
$
-#+END_SRC
+```
diff --git a/doc/tutorial/tests.org b/doc/tutorial/tests.md
index d6842ab2..138769b1 100644
--- a/doc/tutorial/tests.org
+++ b/doc/tutorial/tests.md
@@ -1,38 +1,41 @@
-* Creating Tests
+Creating Tests
+==============
-To run tests with justbuild, we do /not/ have a dedicated ~test~
+To run tests with justbuild, we do *not* have a dedicated `test`
subcommand. Instead, we consider tests being a specific action that
-generates a test report. Consequently, we use the ~build~ subcommand
-to build the test report, and thereby run the test action. Test
-actions, however, are slightly different from normal actions in
-that we don't want the build of the test report to be aborted if
-a test action fails (but still, we want only successfully actions
-taken from cache). Rules defining targets containing such special
-actions have to identify themselves as /tainted/ by specifying
-a string explaining why such special actions are justified; in
-our case, the string is ~"test"~. Besides the implicit marking by
-using a tainted rule, those tainting strings can also be explicitly
-assigned by the user in the definition of a target, e.g., to mark
-test data. Any target has to be tainted with (at least) all the
-strings any of its dependencies is tainted with. In this way, it
-is ensured that no test target will end up in a production build.
-
-For the remainder of this section, we expect to have the project files available
-resulting from successfully completing the tutorial section on /Building C++
-Hello World/. We will demonstrate how to write a test binary for the ~greet~
-library and a shell test for the ~helloworld~ binary.
-
-** Creating a C++ test binary
-
-First, we will create a C++ test binary for testing the correct functionality of
-the ~greet~ library. Therefore, we need to provide a C++ source file that performs
-the actual testing and returns non-~0~ on failure. For simplicity reasons, we do
-not use a testing framework for this tutorial. A simple test that captures
-standard output and verifies it with the expected output should be provided in
-the file ~tests/greet.test.cpp~:
-
-#+SRCNAME: tests/greet.test.cpp
-#+BEGIN_SRC cpp
+generates a test report. Consequently, we use the `build` subcommand to
+build the test report, and thereby run the test action. Test actions,
+however, are slightly different from normal actions in that we don't
+want the build of the test report to be aborted if a test action fails
+(but still, we want only successfully actions taken from cache). Rules
+defining targets containing such special actions have to identify
+themselves as *tainted* by specifying a string explaining why such
+special actions are justified; in our case, the string is `"test"`.
+Besides the implicit marking by using a tainted rule, those tainting
+strings can also be explicitly assigned by the user in the definition of
+a target, e.g., to mark test data. Any target has to be tainted with (at
+least) all the strings any of its dependencies is tainted with. In this
+way, it is ensured that no test target will end up in a production
+build.
+
+For the remainder of this section, we expect to have the project files
+available resulting from successfully completing the tutorial section on
+*Building C++ Hello World*. We will demonstrate how to write a test
+binary for the `greet` library and a shell test for the `helloworld`
+binary.
+
+Creating a C++ test binary
+--------------------------
+
+First, we will create a C++ test binary for testing the correct
+functionality of the `greet` library. Therefore, we need to provide a
+C++ source file that performs the actual testing and returns non-`0` on
+failure. For simplicity reasons, we do not use a testing framework for
+this tutorial. A simple test that captures standard output and verifies
+it with the expected output should be provided in the file
+`tests/greet.test.cpp`:
+
+``` {.cpp srcname="tests/greet.test.cpp"}
#include <functional>
#include <iostream>
#include <string>
@@ -68,15 +71,14 @@ auto test_greet(std::string const& name) -> bool {
int main() {
return test_greet("World") && test_greet("Universe") ? 0 : 1;
}
-#+END_SRC
+```
-Next, a new test target needs to be created in module ~greet~. This target uses
-the rule ~["@", "rules", "CC/test", "test"]~ and needs to depend on the
-~["greet", "greet"]~ target. To create the test target, add the following to
-~tests/TARGETS~:
+Next, a new test target needs to be created in module `greet`. This
+target uses the rule `["@", "rules", "CC/test", "test"]` and needs to
+depend on the `["greet", "greet"]` target. To create the test target,
+add the following to `tests/TARGETS`:
-#+SRCNAME: tests/TARGETS
-#+BEGIN_SRC js
+``` {.jsonc srcname="tests/TARGETS"}
{ "greet":
{ "type": ["@", "rules", "CC/test", "test"]
, "name": ["test_greet"]
@@ -84,35 +86,33 @@ the rule ~["@", "rules", "CC/test", "test"]~ and needs to depend on the
, "private-deps": [["greet", "greet"]]
}
}
-#+END_SRC
+```
-Before we can run the test, a proper default module for ~CC/test~ must be
-provided. By specifying the appropriate target in this module the default test
-runner can be overwritten by a different test runner fom the rule's workspace
-root. Moreover, all test targets share runner infrastructure from ~shell/test~,
-e.g., summarizing multiple runs per test (to detect flakyness) if the configuration
-variable ~RUNS_PER_TEST~ is set.
+Before we can run the test, a proper default module for `CC/test` must
+be provided. By specifying the appropriate target in this module the
+default test runner can be overwritten by a different test runner fom
+the rule's workspace root. Moreover, all test targets share runner
+infrastructure from `shell/test`, e.g., summarizing multiple runs per
+test (to detect flakyness) if the configuration variable `RUNS_PER_TEST`
+is set.
However, in our case, we want to use the default runner and therefore it
is sufficient to create an empty module. To do so, create the file
-~tutorial-defaults/CC/test/TARGETS~ with content
+`tutorial-defaults/CC/test/TARGETS` with content
-#+SRCNAME: tutorial-defaults/CC/test/TARGETS
-#+BEGIN_SRC js
+``` {.jsonc srcname="tutorial-defaults/CC/test/TARGETS"}
{}
-#+END_SRC
+```
-as well as the file ~tutorial-defaults/shell/test/TARGETS~ with content
+as well as the file `tutorial-defaults/shell/test/TARGETS` with content
-#+SRCNAME: tutorial-defaults/shell/test/TARGETS
-#+BEGIN_SRC js
+``` {.jsonc srcname="tutorial-defaults/shell/test/TARGETS"}
{}
-#+END_SRC
-
+```
Now we can run the test (i.e., build the test result):
-#+BEGIN_SRC sh
+``` sh
$ just-mr build tests greet
INFO: Requested target is [["@","tutorial","tests","greet"],{}]
INFO: Analysed target [["@","tutorial","tests","greet"],{}]
@@ -130,41 +130,45 @@ INFO: Artifacts built, logical paths are:
(1 runfiles omitted.)
INFO: Target tainted ["test"].
$
-#+END_SRC
-
-Note that the target is correctly reported as tainted with ~"test"~. It will
-produce 3 additional actions for compiling, linking and running the test binary.
-
-The result of the test target are 5 artifacts: ~result~ (containing ~UNKNOWN~,
-~PASS~, or ~FAIL~), ~stderr~, ~stdout~, ~time-start~, and ~time-stop~, and a
-single runfile (omitted in the output above), which is a tree artifact with the
-name ~test_greet~ that contains all of the above artifacts. The test was run
-successfully as otherwise all reported artifacts would have been reported as
-~FAILED~ in the output, and justbuild would have returned the exit code ~2~.
-
-To immediately print the standard output produced by the test binary on the
-command line, the ~-P~ option can be used. Argument to this option is the name
-of the artifact that should be printed on the command line, in our case
-~stdout~:
-
-#+BEGIN_SRC sh
+```
+
+Note that the target is correctly reported as tainted with `"test"`. It
+will produce 3 additional actions for compiling, linking and running the
+test binary.
+
+The result of the test target are 5 artifacts: `result` (containing
+`UNKNOWN`, `PASS`, or `FAIL`), `stderr`, `stdout`, `time-start`, and
+`time-stop`, and a single runfile (omitted in the output above), which
+is a tree artifact with the name `test_greet` that contains all of the
+above artifacts. The test was run successfully as otherwise all reported
+artifacts would have been reported as `FAILED` in the output, and
+justbuild would have returned the exit code `2`.
+
+To immediately print the standard output produced by the test binary on
+the command line, the `-P` option can be used. Argument to this option
+is the name of the artifact that should be printed on the command line,
+in our case `stdout`:
+
+``` sh
$ just-mr build tests greet --log-limit 1 -P stdout
greet output: Hello World!
greet output: Hello Universe!
$
-#+END_SRC
+```
-Note that ~--log-limit 1~ was just added to omit justbuild's ~INFO:~ prints.
+Note that `--log-limit 1` was just added to omit justbuild's `INFO:`
+prints.
-Our test binary does not have any useful options for directly interacting
-with it. When working with test frameworks, it sometimes can be desirable to
-get hold of the test binary itself for manual interaction. The running of
-the test binary is the last action associated with the test and the test
-binary is, of course, one of its inputs.
+Our test binary does not have any useful options for directly
+interacting with it. When working with test frameworks, it sometimes can
+be desirable to get hold of the test binary itself for manual
+interaction. The running of the test binary is the last action
+associated with the test and the test binary is, of course, one of its
+inputs.
-#+BEGIN_SRC sh
+``` sh
$ just-mr analyse --request-action-input -1 tests greet
INFO: Requested target is [["@","tutorial","tests","greet"],{}]
INFO: Request is input of action #-1
@@ -197,15 +201,15 @@ INFO: Result of input of action #-1 of target [["@","tutorial","tests","greet"],
}
INFO: Target tainted ["test"].
$
-#+END_SRC
+```
The provided data also shows us the precise description of the action
-for which we request the input. This allows us to manually rerun
-the action. Or we can simply interact with the test binary manually
-after installing the inputs to this action. Requesting the inputs
-of an action can also be useful when debugging a build failure.
+for which we request the input. This allows us to manually rerun the
+action. Or we can simply interact with the test binary manually after
+installing the inputs to this action. Requesting the inputs of an action
+can also be useful when debugging a build failure.
-#+BEGIN_SRC sh
+``` sh
$ just-mr install -o work --request-action-input -1 tests greet
INFO: Requested target is [["@","tutorial","tests","greet"],{}]
INFO: Request is input of action #-1
@@ -231,26 +235,25 @@ $ echo $?
0
$ cd ..
$ rm -rf work
-#+END_SRC
+```
-** Creating a shell test
+Creating a shell test
+---------------------
-Similarly, to create a shell test for testing the ~helloworld~ binary, a test
-script ~tests/test_helloworld.sh~ must be provided:
+Similarly, to create a shell test for testing the `helloworld` binary, a
+test script `tests/test_helloworld.sh` must be provided:
-#+SRCNAME: tests/test_helloworld.sh
-#+BEGIN_SRC sh
+``` {.sh srcname="tests/test_helloworld.sh"}
set -e
[ "$(./helloworld)" = "Hello Universe!" ]
-#+END_SRC
+```
The test target for this shell tests uses the rule
-~["@", "rules", "shell/test", "script"]~ and must depend on the ~"helloworld"~
-target. To create the test target, add the following to the ~tests/TARGETS~
-file:
+`["@", "rules", "shell/test", "script"]` and must depend on the
+`"helloworld"` target. To create the test target, add the following to
+the `tests/TARGETS` file:
-#+SRCNAME: tests/TARGETS
-#+BEGIN_SRC js
+``` {.jsonc srcname="tests/TARGETS"}
...
, "helloworld":
{ "type": ["@", "rules", "shell/test", "script"]
@@ -259,11 +262,11 @@ file:
, "deps": [["", "helloworld"]]
}
...
-#+END_SRC
+```
Now we can run the shell test (i.e., build the test result):
-#+BEGIN_SRC sh
+``` sh
$ just-mr build tests helloworld
INFO: Requested target is [["@","tutorial","tests","helloworld"],{}]
INFO: Analysed target [["@","tutorial","tests","helloworld"],{}]
@@ -281,29 +284,28 @@ INFO: Artifacts built, logical paths are:
(1 runfiles omitted.)
INFO: Target tainted ["test"].
$
-#+END_SRC
-
-The result is also similar, containing also the 5 artifacts and a single runfile
-(omitted in the output above), which is a tree artifact with the name
-~test_helloworld~ that contains all of the above artifacts.
-
-** Creating a compound test target
-
-As most people probably do not want to call every test target by hand, it is
-desirable to compound test target that triggers the build of multiple test
-reports. To do so, an ~"install"~ target can be used. The field ~"deps"~ of
-an install target is a list of targets for which the runfiles are collected.
-As for the tests the runfiles happen to be
-tree artifacts named the same way as the test and containing all test results,
-this is precisely what we need.
-Furthermore, as the dependent test targets are tainted by ~"test"~, also the
-compound test target must be tainted by the same string. To create the compound
-test target combining the two tests above (the tests ~"greet"~ and
-~"helloworld"~ from module ~"tests"~), add the following to the ~tests/TARGETS~
-file:
-
-#+SRCNAME: tests/TARGETS
-#+BEGIN_SRC js
+```
+
+The result is also similar, containing also the 5 artifacts and a single
+runfile (omitted in the output above), which is a tree artifact with the
+name `test_helloworld` that contains all of the above artifacts.
+
+Creating a compound test target
+-------------------------------
+
+As most people probably do not want to call every test target by hand,
+it is desirable to compound test target that triggers the build of
+multiple test reports. To do so, an `"install"` target can be used. The
+field `"deps"` of an install target is a list of targets for which the
+runfiles are collected. As for the tests the runfiles happen to be tree
+artifacts named the same way as the test and containing all test
+results, this is precisely what we need. Furthermore, as the dependent
+test targets are tainted by `"test"`, also the compound test target must
+be tainted by the same string. To create the compound test target
+combining the two tests above (the tests `"greet"` and `"helloworld"`
+from module `"tests"`), add the following to the `tests/TARGETS` file:
+
+``` {.jsonc srcname="tests/TARGETS"}
...
, "ALL":
{ "type": "install"
@@ -311,12 +313,12 @@ file:
, "deps": ["greet", "helloworld"]
}
...
-#+END_SRC
+```
-Now we can run all tests at once by just building the compound test target
-~"ALL"~:
+Now we can run all tests at once by just building the compound test
+target `"ALL"`:
-#+BEGIN_SRC sh
+``` sh
$ just-mr build tests ALL
INFO: Requested target is [["@","tutorial","tests","ALL"],{}]
INFO: Analysed target [["@","tutorial","tests","ALL"],{}]
@@ -330,8 +332,8 @@ INFO: Artifacts built, logical paths are:
test_helloworld [63fa5954161b52b275b05c270e1626feaa8e178b:177:t]
INFO: Target tainted ["test"].
$
-#+END_SRC
+```
-As a result it reports the runfiles (result directories) of both tests as
-artifacts. Both tests ran successfully as none of those artifacts in this output
-above are tagged as ~FAILED~.
+As a result it reports the runfiles (result directories) of both tests
+as artifacts. Both tests ran successfully as none of those artifacts in
+this output above are tagged as `FAILED`.
diff --git a/doc/tutorial/third-party-software.md b/doc/tutorial/third-party-software.md
new file mode 100644
index 00000000..daaf5b2d
--- /dev/null
+++ b/doc/tutorial/third-party-software.md
@@ -0,0 +1,473 @@
+Building Third-party Software
+=============================
+
+Third-party projects usually ship with their own build description,
+which often happens to be not compatible with justbuild. Nevertheless,
+it is highly desireable to include external projects via their source
+code base, instead of relying on the integration of out-of-band binary
+distributions. justbuild offers a flexible approach to provide the
+required build description via an overlay layer without the need to
+touch the original code base.
+
+For the remainder of this section, we expect to have the project files
+available resulting from successfully completing the tutorial section on
+*Building C++ Hello World*. We will demonstrate how to use the
+open-source project [fmtlib](https://github.com/fmtlib/fmt) as an
+example for integrating third-party software to a justbuild project.
+
+Creating the target overlay layer for fmtlib
+--------------------------------------------
+
+Before we construct the overlay layer for fmtlib, we need to determine
+its file structure ([tag
+8.1.1](https://github.com/fmtlib/fmt/tree/8.1.1)). The relevant header
+and source files are structured as follows:
+
+ fmt
+ |
+ +--include
+ | +--fmt
+ | +--*.h
+ |
+ +--src
+ +--format.cc
+ +--os.cc
+
+The public headers can be found in `include/fmt`, while the library's
+source files are located in `src`. For the overlay layer, the `TARGETS`
+files should be placed in a tree structure that resembles the original
+code base's structure. It is also good practice to provide a top-level
+`TARGETS` file, leading to the following structure for the overlay:
+
+ fmt-layer
+ |
+ +--TARGETS
+ +--include
+ | +--fmt
+ | +--TARGETS
+ |
+ +--src
+ +--TARGETS
+
+Let's create the overlay structure:
+
+``` sh
+$ mkdir -p fmt-layer/include/fmt
+$ mkdir -p fmt-layer/src
+```
+
+The directory `include/fmt` contains only header files. As we want all
+files in this directory to be included in the `"hdrs"` target, we can
+safely use the explicit `TREE` reference[^1], which collects, in a
+single artifact (describing a directory) *all* directory contents from
+`"."` of the workspace root. Note that the `TARGETS` file is only part
+of the overlay, and therefore will not be part of this tree.
+Furthermore, this tree should be staged to `"fmt"`, so that any consumer
+can include those headers via `<fmt/...>`. The resulting header
+directory target `"hdrs"` in `include/fmt/TARGETS` should be described
+as:
+
+``` {.jsonc srcname="fmt-layer/include/fmt/TARGETS"}
+{ "hdrs":
+ { "type": ["@", "rules", "data", "staged"]
+ , "srcs": [["TREE", null, "."]]
+ , "stage": ["fmt"]
+ }
+}
+```
+
+The actual library target is defined in the directory `src`. For the
+public headers, it refers to the previously created `"hdrs"` target via
+its fully-qualified target name (`["include/fmt", "hdrs"]`). Source
+files are the two local files `format.cc`, and `os.cc`. The final target
+description in `src/TARGETS` will look like this:
+
+``` {.jsonc srcname="fmt-layer/src/TARGETS"}
+{ "fmt":
+ { "type": ["@", "rules", "CC", "library"]
+ , "name": ["fmt"]
+ , "hdrs": [["include/fmt", "hdrs"]]
+ , "srcs": ["format.cc", "os.cc"]
+ }
+}
+```
+
+Finally, the top-level `TARGETS` file can be created. While it is
+technically not strictly required, it is considered good practice to
+*export* every target that may be used by another project. Exported
+targets are subject to high-level target caching, which allows to skip
+the analysis and traversal of entire subgraphs in the action graph.
+Therefore, we create an export target that exports the target
+`["src", "fmt"]`, with only the variables in the field
+`"flexible_config"` being propagated. The top-level `TARGETS` file
+contains the following content:
+
+``` {.jsonc srcname="fmt-layer/TARGETS"}
+{ "fmt":
+ { "type": "export"
+ , "target": ["src", "fmt"]
+ , "flexible_config": ["CXX", "CXXFLAGS", "ADD_CXXFLAGS", "AR", "ENV"]
+ }
+}
+```
+
+After adding the library to the multi-repository configuration (next
+step), the list of configuration variables a target, like `["src",
+"fmt"]`, actually depends on can be obtained using the `--dump-vars`
+option of the `analyse` subcommand. In this way, an informed decision
+can be taken when deciding which variables of the export target to make
+tunable for the consumer.
+
+Adding fmtlib to the Multi-Repository Configuration
+---------------------------------------------------
+
+Based on the *hello world* tutorial, we can extend the existing
+`repos.json` by the layer definition `"fmt-targets-layer"` and the
+repository `"fmtlib"`, which is based on the Git repository with its
+target root being overlayed. Furthermore, we want to use `"fmtlib"` in
+the repository `"tutorial"`, and therefore need to introduce an
+additional binding `"format"` for it:
+
+``` {.jsonc srcname="repos.json"}
+{ "main": "tutorial"
+, "repositories":
+ { "rules-cc":
+ { "repository":
+ { "type": "git"
+ , "branch": "master"
+ , "commit": "123d8b03bf2440052626151c14c54abce2726e6f"
+ , "repository": "https://github.com/just-buildsystem/rules-cc.git"
+ , "subdir": "rules"
+ }
+ , "target_root": "tutorial-defaults"
+ , "rule_root": "rules-cc"
+ }
+ , "tutorial":
+ { "repository": {"type": "file", "path": "."}
+ , "bindings": {"rules": "rules-cc", "format": "fmtlib"}
+ }
+ , "tutorial-defaults":
+ { "repository": {"type": "file", "path": "./tutorial-defaults"}
+ }
+ , "fmt-targets-layer":
+ { "repository": {"type": "file", "path": "./fmt-layer"}
+ }
+ , "fmtlib":
+ { "repository":
+ { "type": "git"
+ , "branch": "8.1.1"
+ , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9"
+ , "repository": "https://github.com/fmtlib/fmt.git"
+ }
+ , "target_root": "fmt-targets-layer"
+ , "bindings": {"rules": "rules-cc"}
+ }
+ }
+}
+```
+
+This `"format"` binding can you be used to add a new private dependency
+in `greet/TARGETS`:
+
+``` {.jsonc srcname="greet/TARGETS"}
+{ "greet":
+ { "type": ["@", "rules", "CC", "library"]
+ , "name": ["greet"]
+ , "hdrs": ["greet.hpp"]
+ , "srcs": ["greet.cpp"]
+ , "stage": ["greet"]
+ , "private-deps": [["@", "format", "", "fmt"]]
+ }
+}
+```
+
+Consequently, the `fmtlib` library can now be used by `greet/greet.cpp`:
+
+``` {.cpp srcname="greet/greet.cpp"}
+#include "greet.hpp"
+#include <fmt/format.h>
+
+void greet(std::string const& s) {
+ fmt::print("Hello {}!\n", s);
+}
+```
+
+Due to changes made to `repos.json`, building this tutorial requires to
+rerun `just-mr`, which will fetch the necessary sources for the external
+repositories:
+
+``` sh
+$ just-mr build helloworld
+INFO: Requested target is [["@","tutorial","","helloworld"],{}]
+INFO: Analysed target [["@","tutorial","","helloworld"],{}]
+INFO: Export targets found: 0 cached, 0 uncached, 1 not eligible for caching
+INFO: Discovered 7 actions, 3 trees, 0 blobs
+INFO: Building [["@","tutorial","","helloworld"],{}].
+INFO: Processed 7 actions, 1 cache hits.
+INFO: Artifacts built, logical paths are:
+ helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x]
+$
+```
+
+Note to build the `fmt` target alone, its containing repository `fmtlib`
+must be specified via the `--main` option:
+
+``` sh
+$ just-mr --main fmtlib build fmt
+INFO: Requested target is [["@","fmtlib","","fmt"],{}]
+INFO: Analysed target [["@","fmtlib","","fmt"],{}]
+INFO: Export targets found: 0 cached, 0 uncached, 1 not eligible for caching
+INFO: Discovered 3 actions, 1 trees, 0 blobs
+INFO: Building [["@","fmtlib","","fmt"],{}].
+INFO: Processed 3 actions, 3 cache hits.
+INFO: Artifacts built, logical paths are:
+ libfmt.a [513b2ac17c557675fc841f3ebf279003ff5a73ae:240914:f]
+ (1 runfiles omitted.)
+$
+```
+
+Employing high-level target caching
+-----------------------------------
+
+The make use of high-level target caching for exported targets, we need
+to ensure that all inputs to an export target are transitively
+content-fixed. This is automatically the case for `"type":"git"`
+repositories. However, the `libfmt` repository also depends on
+`"rules-cc"`, `"tutorial-defaults"`, and `"fmt-target-layer"`. As the
+latter two are `"type":"file"` repositories, they must be put under Git
+versioning first:
+
+``` sh
+$ git init .
+$ git add tutorial-defaults fmt-layer
+$ git commit -m"fix compile flags and fmt targets layer"
+```
+
+Note that `rules-cc` already is under Git versioning.
+
+Now, to instruct `just-mr` to use the content-fixed, committed source
+trees of those `"type":"file"` repositories the pragma `"to_git"` must
+be set for them in `repos.json`:
+
+``` {.jsonc srcname="repos.json"}
+{ "main": "tutorial"
+, "repositories":
+ { "rules-cc":
+ { "repository":
+ { "type": "git"
+ , "branch": "master"
+ , "commit": "123d8b03bf2440052626151c14c54abce2726e6f"
+ , "repository": "https://github.com/just-buildsystem/rules-cc.git"
+ , "subdir": "rules"
+ }
+ , "target_root": "tutorial-defaults"
+ , "rule_root": "rules-cc"
+ }
+ , "tutorial":
+ { "repository": {"type": "file", "path": "."}
+ , "bindings": {"rules": "rules-cc", "format": "fmtlib"}
+ }
+ , "tutorial-defaults":
+ { "repository":
+ { "type": "file"
+ , "path": "./tutorial-defaults"
+ , "pragma": {"to_git": true}
+ }
+ }
+ , "fmt-targets-layer":
+ { "repository":
+ { "type": "file"
+ , "path": "./fmt-layer"
+ , "pragma": {"to_git": true}
+ }
+ }
+ , "fmtlib":
+ { "repository":
+ { "type": "git"
+ , "branch": "master"
+ , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9"
+ , "repository": "https://github.com/fmtlib/fmt.git"
+ }
+ , "target_root": "fmt-targets-layer"
+ , "bindings": {"rules": "rules-cc"}
+ }
+ }
+}
+```
+
+Due to changes in the repository configuration, we need to rebuild and
+the benefits of the target cache should be visible on the second build:
+
+``` sh
+$ just-mr build helloworld
+INFO: Requested target is [["@","tutorial","","helloworld"],{}]
+INFO: Analysed target [["@","tutorial","","helloworld"],{}]
+INFO: Export targets found: 0 cached, 1 uncached, 0 not eligible for caching
+INFO: Discovered 7 actions, 3 trees, 0 blobs
+INFO: Building [["@","tutorial","","helloworld"],{}].
+INFO: Processed 7 actions, 7 cache hits.
+INFO: Artifacts built, logical paths are:
+ helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x]
+$
+$ just-mr build helloworld
+INFO: Requested target is [["@","tutorial","","helloworld"],{}]
+INFO: Analysed target [["@","tutorial","","helloworld"],{}]
+INFO: Export targets found: 1 cached, 0 uncached, 0 not eligible for caching
+INFO: Discovered 4 actions, 2 trees, 0 blobs
+INFO: Building [["@","tutorial","","helloworld"],{}].
+INFO: Processed 4 actions, 4 cache hits.
+INFO: Artifacts built, logical paths are:
+ helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x]
+$
+```
+
+Note that in the second run the export target `"fmt"` was taken from
+cache and its 3 actions were eliminated, as their result has been
+recorded to the high-level target cache during the first run.
+
+Combining overlay layers for multiple projects
+----------------------------------------------
+
+Projects typically depend on multiple external repositories. Creating an
+overlay layer for each external repository might unnecessarily clutter
+up the repository configuration and the file structure of your
+repository. One solution to mitigate this issue is to combine the
+`TARGETS` files of multiple external repositories in a single overlay
+layer. To avoid conflicts, the `TARGETS` files can be assigned different
+file names per repository. As an example, imagine a common overlay layer
+with the files `TARGETS.fmt` and `TARGETS.gsl` for the repositories
+`"fmtlib"` and `"gsl-lite"`, respectively:
+
+ common-layer
+ |
+ +--TARGETS.fmt
+ +--TARGETS.gsl
+ +--include
+ | +--fmt
+ | | +--TARGETS.fmt
+ | +--gsl
+ | +--TARGETS.gsl
+ |
+ +--src
+ +--TARGETS.fmt
+
+Such a common overlay layer can be used as the target root for both
+repositories with only one difference: the `"target_file_name"` field.
+By specifying this field, the dispatch where to find the respective
+target description for each repository is implemented. For the given
+example, the following `repos.json` defines the overlay
+`"common-targets-layer"`, which is used by `"fmtlib"` and `"gsl-lite"`:
+
+``` {.jsonc srcname="repos.json"}
+{ "main": "tutorial"
+, "repositories":
+ { "rules-cc":
+ { "repository":
+ { "type": "git"
+ , "branch": "master"
+ , "commit": "123d8b03bf2440052626151c14c54abce2726e6f"
+ , "repository": "https://github.com/just-buildsystem/rules-cc.git"
+ , "subdir": "rules"
+ }
+ , "target_root": "tutorial-defaults"
+ , "rule_root": "rules-cc"
+ }
+ , "tutorial":
+ { "repository": {"type": "file", "path": "."}
+ , "bindings": {"rules": "rules-cc", "format": "fmtlib"}
+ }
+ , "tutorial-defaults":
+ { "repository":
+ { "type": "file"
+ , "path": "./tutorial-defaults"
+ , "pragma": {"to_git": true}
+ }
+ }
+ , "common-targets-layer":
+ { "repository":
+ { "type": "file"
+ , "path": "./common-layer"
+ , "pragma": {"to_git": true}
+ }
+ }
+ , "fmtlib":
+ { "repository":
+ { "type": "git"
+ , "branch": "8.1.1"
+ , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9"
+ , "repository": "https://github.com/fmtlib/fmt.git"
+ }
+ , "target_root": "common-targets-layer"
+ , "target_file_name": "TARGETS.fmt"
+ , "bindings": {"rules": "rules-cc"}
+ }
+ , "gsl-lite":
+ { "repository":
+ { "type": "git"
+ , "branch": "v0.40.0"
+ , "commit": "d6c8af99a1d95b3db36f26b4f22dc3bad89952de"
+ , "repository": "https://github.com/gsl-lite/gsl-lite.git"
+ }
+ , "target_root": "common-targets-layer"
+ , "target_file_name": "TARGETS.gsl"
+ , "bindings": {"rules": "rules-cc"}
+ }
+ }
+}
+```
+
+Using pre-built dependencies
+----------------------------
+
+While building external dependencies from source brings advantages, most
+prominently the flexibility to quickly and seamlessly switch to a
+different build configuration (production, debug, instrumented for
+performance analysis; cross-compiling for a different target
+architecture), there are also legitimate reasons to use pre-built
+dependencies. The most prominent one is if your project is packaged as
+part of a larger distribution. For that reason, just also has (in
+`etc/import.prebuilt`) target files for all its dependencies assuming
+they are pre-installed. The reason why target files are used at all for
+this situation is twofold.
+
+ - On the one hand, having a target allows the remaining targets to not
+ care about where their dependencies come from, or if it is a build
+ against pre-installed dependencies or not. Also, the top-level
+ binary does not have to know the linking requirements of its
+ transitive dependencies. In other words, information stays where it
+ belongs to and if one target acquires a new dependency, the
+ information is automatically propagated to all targets using it.
+ - Still some information is needed to use a pre-installed library and,
+ as explained, a target describing the pre-installed library is the
+ right place to collect this information.
+ - The public header files of the library. By having this explicit,
+ we do not accumulate directories in the include search path and
+ hence also properly detect include conflicts.
+ - The information on how to link the library itself (i.e.,
+ basically its base name).
+ - Any dependencies on other libraries that the library might have.
+ This information is used to obtain the correct linking order and
+ complete transitive linking arguments while keeping the
+ description maintainable, as each target still only declares its
+ direct dependencies.
+
+The target description for a pre-built version of the format library
+that was used as an example in this section is shown next; with our
+staging mechanism the logical repository it belongs to is rooted in the
+`fmt` subdirectory of the `include` directory of the ambient system.
+
+``` {.jsonc srcname="etc/import.prebuilt/TARGETS.fmt"}
+{ "fmt":
+ { "type": ["@", "rules", "CC", "library"]
+ , "name": ["fmt"]
+ , "stage": ["fmt"]
+ , "hdrs": [["TREE", null, "."]]
+ , "private-ldflags": ["-lfmt"]
+ }
+}
+```
+
+[^1]: Explicit `TREE` references are always a list of length 3, to
+ distinguish them from target references of length 2 (module and
+ target name). Furthermore, the second list element is always `null`
+ as we only want to allow tree references from the current module.
diff --git a/doc/tutorial/third-party-software.org b/doc/tutorial/third-party-software.org
deleted file mode 100644
index d1712cc8..00000000
--- a/doc/tutorial/third-party-software.org
+++ /dev/null
@@ -1,475 +0,0 @@
-* Building Third-party Software
-
-Third-party projects usually ship with their own build description, which often
-happens to be not compatible with justbuild. Nevertheless, it is highly
-desireable to include external projects via their source code base, instead of
-relying on the integration of out-of-band binary distributions. justbuild offers
-a flexible approach to provide the required build description via an overlay
-layer without the need to touch the original code base.
-
-For the remainder of this section, we expect to have the project files available
-resulting from successfully completing the tutorial section on /Building C++
-Hello World/. We will demonstrate how to use the open-source project
-[[https://github.com/fmtlib/fmt][fmtlib]] as an example for integrating
-third-party software to a justbuild project.
-
-** Creating the target overlay layer for fmtlib
-
-Before we construct the overlay layer for fmtlib, we need to determine its file
-structure ([[https://github.com/fmtlib/fmt/tree/8.1.1][tag 8.1.1]]). The
-relevant header and source files are structured as follows:
-
-#+BEGIN_SRC
- fmt
- |
- +--include
- | +--fmt
- | +--*.h
- |
- +--src
- +--format.cc
- +--os.cc
-#+END_SRC
-
-The public headers can be found in ~include/fmt~, while the library's source
-files are located in ~src~. For the overlay layer, the ~TARGETS~ files should be
-placed in a tree structure that resembles the original code base's structure.
-It is also good practice to provide a top-level ~TARGETS~ file, leading to the
-following structure for the overlay:
-
-#+BEGIN_SRC
- fmt-layer
- |
- +--TARGETS
- +--include
- | +--fmt
- | +--TARGETS
- |
- +--src
- +--TARGETS
-#+END_SRC
-
-Let's create the overlay structure:
-
-#+BEGIN_SRC sh
-$ mkdir -p fmt-layer/include/fmt
-$ mkdir -p fmt-layer/src
-#+END_SRC
-
-The directory ~include/fmt~ contains only header files. As we want all files in
-this directory to be included in the ~"hdrs"~ target, we can safely
-use the explicit ~TREE~ reference[fn:1], which collects, in a single
-artifact (describing a directory) /all/ directory contents
-from ~"."~ of the workspace root. Note that the ~TARGETS~ file is only part of
-the overlay, and
-therefore will not be part of this tree. Furthermore, this tree should be staged
-to ~"fmt"~, so that any consumer can include those headers via ~<fmt/...>~. The
-resulting header directory target ~"hdrs"~ in ~include/fmt/TARGETS~ should be
-described as:
-
-[fn:1] Explicit ~TREE~ references are always a list of length 3, to distinguish
-them from target references of length 2 (module and target name). Furthermore,
-the second list element is always ~null~ as we only want to allow tree
-references from the current module.
-
-
-#+SRCNAME: fmt-layer/include/fmt/TARGETS
-#+BEGIN_SRC js
-{ "hdrs":
- { "type": ["@", "rules", "data", "staged"]
- , "srcs": [["TREE", null, "."]]
- , "stage": ["fmt"]
- }
-}
-#+END_SRC
-
-The actual library target is defined in the directory ~src~. For the public
-headers, it refers to the previously created ~"hdrs"~ target via its
-fully-qualified target name (~["include/fmt", "hdrs"]~). Source files are the
-two local files ~format.cc~, and ~os.cc~. The final target description in
-~src/TARGETS~ will look like this:
-
-#+SRCNAME: fmt-layer/src/TARGETS
-#+BEGIN_SRC js
-{ "fmt":
- { "type": ["@", "rules", "CC", "library"]
- , "name": ["fmt"]
- , "hdrs": [["include/fmt", "hdrs"]]
- , "srcs": ["format.cc", "os.cc"]
- }
-}
-#+END_SRC
-
-Finally, the top-level ~TARGETS~ file can be created. While it is technically
-not strictly required, it is considered good practice to /export/ every target
-that may be used by another project. Exported targets are subject to high-level
-target caching, which allows to skip the analysis and traversal of entire
-subgraphs in the action graph. Therefore, we create an export target that
-exports the target ~["src", "fmt"]~, with only the variables in the field
-~"flexible_config"~ being propagated. The top-level ~TARGETS~ file contains the
-following content:
-
-#+SRCNAME: fmt-layer/TARGETS
-#+BEGIN_SRC js
-{ "fmt":
- { "type": "export"
- , "target": ["src", "fmt"]
- , "flexible_config": ["CXX", "CXXFLAGS", "ADD_CXXFLAGS", "AR", "ENV"]
- }
-}
-#+END_SRC
-
-After adding the library to the multi-repository configuration (next
-step), the list of configuration variables a target, like ~["src",
-"fmt"]~, actually depends on can be obtained using the ~--dump-vars~
-option of the ~analyse~ subcommand. In this way, an informed decision
-can be taken when deciding which variables of the export target to
-make tunable for the consumer.
-
-** Adding fmtlib to the Multi-Repository Configuration
-
-Based on the /hello world/ tutorial, we can extend the existing ~repos.json~ by
-the layer definition ~"fmt-targets-layer"~ and the repository ~"fmtlib"~, which
-is based on the Git repository with its target root being overlayed.
-Furthermore, we want to use ~"fmtlib"~ in the repository ~"tutorial"~, and
-therefore need to introduce an additional binding ~"format"~ for it:
-
-#+SRCNAME: repos.json
-#+BEGIN_SRC js
-{ "main": "tutorial"
-, "repositories":
- { "rules-cc":
- { "repository":
- { "type": "git"
- , "branch": "master"
- , "commit": "123d8b03bf2440052626151c14c54abce2726e6f"
- , "repository": "https://github.com/just-buildsystem/rules-cc.git"
- , "subdir": "rules"
- }
- , "target_root": "tutorial-defaults"
- , "rule_root": "rules-cc"
- }
- , "tutorial":
- { "repository": {"type": "file", "path": "."}
- , "bindings": {"rules": "rules-cc", "format": "fmtlib"}
- }
- , "tutorial-defaults":
- { "repository": {"type": "file", "path": "./tutorial-defaults"}
- }
- , "fmt-targets-layer":
- { "repository": {"type": "file", "path": "./fmt-layer"}
- }
- , "fmtlib":
- { "repository":
- { "type": "git"
- , "branch": "8.1.1"
- , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9"
- , "repository": "https://github.com/fmtlib/fmt.git"
- }
- , "target_root": "fmt-targets-layer"
- , "bindings": {"rules": "rules-cc"}
- }
- }
-}
-#+END_SRC
-
-This ~"format"~ binding can you be used to add a new private dependency in
-~greet/TARGETS~:
-
-#+SRCNAME: greet/TARGETS
-#+BEGIN_SRC js
-{ "greet":
- { "type": ["@", "rules", "CC", "library"]
- , "name": ["greet"]
- , "hdrs": ["greet.hpp"]
- , "srcs": ["greet.cpp"]
- , "stage": ["greet"]
- , "private-deps": [["@", "format", "", "fmt"]]
- }
-}
-#+END_SRC
-
-Consequently, the ~fmtlib~ library can now be used by ~greet/greet.cpp~:
-
-#+SRCNAME: greet/greet.cpp
-#+BEGIN_SRC cpp
-#include "greet.hpp"
-#include <fmt/format.h>
-
-void greet(std::string const& s) {
- fmt::print("Hello {}!\n", s);
-}
-#+END_SRC
-
-Due to changes made to ~repos.json~, building this tutorial requires to rerun
-~just-mr~, which will fetch the necessary sources for the external repositories:
-
-#+BEGIN_SRC sh
-$ just-mr build helloworld
-INFO: Requested target is [["@","tutorial","","helloworld"],{}]
-INFO: Analysed target [["@","tutorial","","helloworld"],{}]
-INFO: Export targets found: 0 cached, 0 uncached, 1 not eligible for caching
-INFO: Discovered 7 actions, 3 trees, 0 blobs
-INFO: Building [["@","tutorial","","helloworld"],{}].
-INFO: Processed 7 actions, 1 cache hits.
-INFO: Artifacts built, logical paths are:
- helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x]
-$
-#+END_SRC
-
-Note to build the ~fmt~ target alone, its containing repository ~fmtlib~ must be
-specified via the ~--main~ option:
-#+BEGIN_SRC sh
-$ just-mr --main fmtlib build fmt
-INFO: Requested target is [["@","fmtlib","","fmt"],{}]
-INFO: Analysed target [["@","fmtlib","","fmt"],{}]
-INFO: Export targets found: 0 cached, 0 uncached, 1 not eligible for caching
-INFO: Discovered 3 actions, 1 trees, 0 blobs
-INFO: Building [["@","fmtlib","","fmt"],{}].
-INFO: Processed 3 actions, 3 cache hits.
-INFO: Artifacts built, logical paths are:
- libfmt.a [513b2ac17c557675fc841f3ebf279003ff5a73ae:240914:f]
- (1 runfiles omitted.)
-$
-#+END_SRC
-
-** Employing high-level target caching
-
-The make use of high-level target caching for exported targets, we need to
-ensure that all inputs to an export target are transitively content-fixed. This
-is automatically the case for ~"type":"git"~ repositories. However, the ~libfmt~
-repository also depends on ~"rules-cc"~, ~"tutorial-defaults"~, and
-~"fmt-target-layer"~. As the latter two are ~"type":"file"~ repositories, they
-must be put under Git versioning first:
-
-#+BEGIN_SRC sh
-$ git init .
-$ git add tutorial-defaults fmt-layer
-$ git commit -m"fix compile flags and fmt targets layer"
-#+END_SRC
-
-Note that ~rules-cc~ already is under Git versioning.
-
-Now, to instruct ~just-mr~ to use the content-fixed, committed source trees of
-those ~"type":"file"~ repositories the pragma ~"to_git"~ must be set for them in
-~repos.json~:
-
-#+SRCNAME: repos.json
-#+BEGIN_SRC js
-{ "main": "tutorial"
-, "repositories":
- { "rules-cc":
- { "repository":
- { "type": "git"
- , "branch": "master"
- , "commit": "123d8b03bf2440052626151c14c54abce2726e6f"
- , "repository": "https://github.com/just-buildsystem/rules-cc.git"
- , "subdir": "rules"
- }
- , "target_root": "tutorial-defaults"
- , "rule_root": "rules-cc"
- }
- , "tutorial":
- { "repository": {"type": "file", "path": "."}
- , "bindings": {"rules": "rules-cc", "format": "fmtlib"}
- }
- , "tutorial-defaults":
- { "repository":
- { "type": "file"
- , "path": "./tutorial-defaults"
- , "pragma": {"to_git": true}
- }
- }
- , "fmt-targets-layer":
- { "repository":
- { "type": "file"
- , "path": "./fmt-layer"
- , "pragma": {"to_git": true}
- }
- }
- , "fmtlib":
- { "repository":
- { "type": "git"
- , "branch": "master"
- , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9"
- , "repository": "https://github.com/fmtlib/fmt.git"
- }
- , "target_root": "fmt-targets-layer"
- , "bindings": {"rules": "rules-cc"}
- }
- }
-}
-#+END_SRC
-
-Due to changes in the repository configuration, we need to rebuild and the
-benefits of the target cache should be visible on the second build:
-
-#+BEGIN_SRC sh
-$ just-mr build helloworld
-INFO: Requested target is [["@","tutorial","","helloworld"],{}]
-INFO: Analysed target [["@","tutorial","","helloworld"],{}]
-INFO: Export targets found: 0 cached, 1 uncached, 0 not eligible for caching
-INFO: Discovered 7 actions, 3 trees, 0 blobs
-INFO: Building [["@","tutorial","","helloworld"],{}].
-INFO: Processed 7 actions, 7 cache hits.
-INFO: Artifacts built, logical paths are:
- helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x]
-$
-$ just-mr build helloworld
-INFO: Requested target is [["@","tutorial","","helloworld"],{}]
-INFO: Analysed target [["@","tutorial","","helloworld"],{}]
-INFO: Export targets found: 1 cached, 0 uncached, 0 not eligible for caching
-INFO: Discovered 4 actions, 2 trees, 0 blobs
-INFO: Building [["@","tutorial","","helloworld"],{}].
-INFO: Processed 4 actions, 4 cache hits.
-INFO: Artifacts built, logical paths are:
- helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x]
-$
-#+END_SRC
-
-Note that in the second run the export target ~"fmt"~ was taken from cache and
-its 3 actions were eliminated, as their result has been recorded to the
-high-level target cache during the first run.
-
-** Combining overlay layers for multiple projects
-
-Projects typically depend on multiple external repositories. Creating an overlay
-layer for each external repository might unnecessarily clutter up the repository
-configuration and the file structure of your repository. One solution to
-mitigate this issue is to combine the ~TARGETS~ files of multiple external
-repositories in a single overlay layer. To avoid conflicts, the ~TARGETS~ files
-can be assigned different file names per repository. As an example, imagine a
-common overlay layer with the files ~TARGETS.fmt~ and ~TARGETS.gsl~ for the
-repositories ~"fmtlib"~ and ~"gsl-lite"~, respectively:
-
-#+BEGIN_SRC
- common-layer
- |
- +--TARGETS.fmt
- +--TARGETS.gsl
- +--include
- | +--fmt
- | | +--TARGETS.fmt
- | +--gsl
- | +--TARGETS.gsl
- |
- +--src
- +--TARGETS.fmt
-#+END_SRC
-
-Such a common overlay layer can be used as the target root for both repositories
-with only one difference: the ~"target_file_name"~ field. By specifying this
-field, the dispatch where to find the respective target description for each
-repository is implemented. For the given example, the following ~repos.json~
-defines the overlay ~"common-targets-layer"~, which is used by ~"fmtlib"~ and
-~"gsl-lite"~:
-
-#+SRCNAME: repos.json
-#+BEGIN_SRC js
-{ "main": "tutorial"
-, "repositories":
- { "rules-cc":
- { "repository":
- { "type": "git"
- , "branch": "master"
- , "commit": "123d8b03bf2440052626151c14c54abce2726e6f"
- , "repository": "https://github.com/just-buildsystem/rules-cc.git"
- , "subdir": "rules"
- }
- , "target_root": "tutorial-defaults"
- , "rule_root": "rules-cc"
- }
- , "tutorial":
- { "repository": {"type": "file", "path": "."}
- , "bindings": {"rules": "rules-cc", "format": "fmtlib"}
- }
- , "tutorial-defaults":
- { "repository":
- { "type": "file"
- , "path": "./tutorial-defaults"
- , "pragma": {"to_git": true}
- }
- }
- , "common-targets-layer":
- { "repository":
- { "type": "file"
- , "path": "./common-layer"
- , "pragma": {"to_git": true}
- }
- }
- , "fmtlib":
- { "repository":
- { "type": "git"
- , "branch": "8.1.1"
- , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9"
- , "repository": "https://github.com/fmtlib/fmt.git"
- }
- , "target_root": "common-targets-layer"
- , "target_file_name": "TARGETS.fmt"
- , "bindings": {"rules": "rules-cc"}
- }
- , "gsl-lite":
- { "repository":
- { "type": "git"
- , "branch": "v0.40.0"
- , "commit": "d6c8af99a1d95b3db36f26b4f22dc3bad89952de"
- , "repository": "https://github.com/gsl-lite/gsl-lite.git"
- }
- , "target_root": "common-targets-layer"
- , "target_file_name": "TARGETS.gsl"
- , "bindings": {"rules": "rules-cc"}
- }
- }
-}
-#+END_SRC
-
-** Using pre-built dependencies
-
-While building external dependencies from source brings advantages,
-most prominently the flexibility to quickly and seamlessly switch
-to a different build configuration (production, debug, instrumented
-for performance analysis; cross-compiling for a different target
-architecture), there are also legitimate reasons to use pre-built
-dependencies. The most prominent one is if your project is packaged
-as part of a larger distribution. For that reason, just also has (in
-~etc/import.prebuilt~) target files for all its dependencies assuming
-they are pre-installed. The reason why target files are used at
-all for this situation is twofold.
-- On the one hand, having a target allows the remaining targets
- to not care about where their dependencies come from, or if it
- is a build against pre-installed dependencies or not. Also, the
- top-level binary does not have to know the linking requirements
- of its transitive dependencies. In other words, information stays
- where it belongs to and if one target acquires a new dependency,
- the information is automatically propagated to all targets using it.
-- Still some information is needed to use a pre-installed library
- and, as explained, a target describing the pre-installed library
- is the right place to collect this information.
- - The public header files of the library. By having this explicit,
- we do not accumulate directories in the include search path
- and hence also properly detect include conflicts.
- - The information on how to link the library itself (i.e.,
- basically its base name).
- - Any dependencies on other libraries that the library might have.
- This information is used to obtain the correct linking order
- and complete transitive linking arguments while keeping the
- description maintainable, as each target still only declares
- its direct dependencies.
-
-The target description for a pre-built version of the format
-library that was used as an example in this section is shown next;
-with our staging mechanism the logical repository it belongs to is
-rooted in the ~fmt~ subdirectory of the ~include~ directory of the
-ambient system.
-
-#+SRCNAME: etc/import.prebuilt/TARGETS.fmt
-#+BEGIN_SRC js
-{ "fmt":
- { "type": ["@", "rules", "CC", "library"]
- , "name": ["fmt"]
- , "stage": ["fmt"]
- , "hdrs": [["TREE", null, "."]]
- , "private-ldflags": ["-lfmt"]
- }
-}
-#+END_SRC