From b66a7359fbbff35af630c88c56598bbc06b393e1 Mon Sep 17 00:00:00 2001 From: Oliver Reiche Date: Thu, 1 Jun 2023 13:36:32 +0200 Subject: doc: Convert orgmode files to markdown --- doc/tutorial/getting-started.md | 217 +++++++++++++++ doc/tutorial/getting-started.org | 212 --------------- doc/tutorial/hello-world.md | 379 ++++++++++++++++++++++++++ doc/tutorial/hello-world.org | 370 ------------------------- doc/tutorial/proto.md | 474 ++++++++++++++++++++++++++++++++ doc/tutorial/proto.org | 475 --------------------------------- doc/tutorial/rebuild.md | 227 ++++++++++++++++ doc/tutorial/rebuild.org | 225 ---------------- doc/tutorial/target-file-glob-tree.md | 430 +++++++++++++++++++++++++++++ doc/tutorial/target-file-glob-tree.org | 430 ----------------------------- doc/tutorial/tests.md | 339 +++++++++++++++++++++++ doc/tutorial/tests.org | 337 ----------------------- doc/tutorial/third-party-software.md | 473 ++++++++++++++++++++++++++++++++ doc/tutorial/third-party-software.org | 475 --------------------------------- 14 files changed, 2539 insertions(+), 2524 deletions(-) create mode 100644 doc/tutorial/getting-started.md delete mode 100644 doc/tutorial/getting-started.org create mode 100644 doc/tutorial/hello-world.md delete mode 100644 doc/tutorial/hello-world.org create mode 100644 doc/tutorial/proto.md delete mode 100644 doc/tutorial/proto.org create mode 100644 doc/tutorial/rebuild.md delete mode 100644 doc/tutorial/rebuild.org create mode 100644 doc/tutorial/target-file-glob-tree.md delete mode 100644 doc/tutorial/target-file-glob-tree.org create mode 100644 doc/tutorial/tests.md delete mode 100644 doc/tutorial/tests.org create mode 100644 doc/tutorial/third-party-software.md delete mode 100644 doc/tutorial/third-party-software.org (limited to 'doc/tutorial') diff --git a/doc/tutorial/getting-started.md b/doc/tutorial/getting-started.md new file mode 100644 index 00000000..36a57d26 --- /dev/null +++ b/doc/tutorial/getting-started.md @@ -0,0 +1,217 @@ +Getting Started +=============== + +In order to use *justbuild*, first make sure that `just`, `just-mr`, and +`just-import-git` are available in your `PATH`. + +Creating a new project +---------------------- + +*justbuild* needs to know the root of the project worked on. By default, +it searches upwards from the current directory till it finds a marker. +Currently, we support three different markers: the files `ROOT` and +`WORKSPACE` or the directory `.git`. Lets create a new project by +creating one of those markers: + +``` sh +$ touch ROOT +``` + +Creating a generic target +------------------------- + +By default, targets are described in `TARGETS` files. These files +contain a `JSON` object with the target name as key and the target +description as value. A target description is an object with at least a +single mandatory field: `"type"`. This field specifies which rule +(built-in or user-defined) to apply for this target. + +A simple target that only executes commands can be created using the +built-in `"generic"` rule, which requires at least one command and one +output file or directory. To create such a target, create the file +`TARGETS` with the following content: + +``` {.jsonc srcname="TARGETS"} +{ "greeter": + { "type": "generic" + , "cmds": ["echo -n 'Hello ' > out.txt", "cat name.txt >> out.txt"] + , "outs": ["out.txt"] + , "deps": ["name.txt"] + } +} +``` + +In this example, the `"greeter"` target will run two commands to produce +the output file `out.txt`. The second command depends on the input file +`name.txt` that we need to create as well: + +``` sh +$ echo World > name.txt +``` + +Building a generic target +------------------------- + +To build a target, we need to run `just` with the subcommand `build`: + +``` sh +$ just build greeter +INFO: Requested target is [["@","","","greeter"],{}] +INFO: Analysed target [["@","","","greeter"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 1 actions, 0 trees, 0 blobs +INFO: Building [["@","","","greeter"],{}]. +INFO: Processed 1 actions, 0 cache hits. +INFO: Artifacts built, logical paths are: + out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] +$ +``` + +The subcommand `build` just builds the artifact but does not stage it to +any user-defined location on the file system. Instead it reports a +description of the artifact consisting of `git` blob identifier, size, +and type (in this case `f` for non-executable file). To also stage the +produced artifact to the working directory, use the `install` subcommand +and specify the output directory: + +``` sh +$ just install greeter -o . +INFO: Requested target is [["@","","","greeter"],{}] +INFO: Analysed target [["@","","","greeter"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 1 actions, 0 trees, 0 blobs +INFO: Building [["@","","","greeter"],{}]. +INFO: Processed 1 actions, 1 cache hits. +INFO: Artifacts can be found in: + /tmp/tutorial/out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] +$ cat out.txt +Hello World +$ +``` + +Note that the `install` subcommand initiates the build a second time, +without executing any actions as all actions are being served from +cache. The produced artifact is identical, which is indicated by the +same hash/size/type. + +If one is only interested in a single final artifact, one can also +request via the `-P` option that this artifact be written to standard +output after the build. As all messages are reported to standard error, +this can be used for both, interactively reading a text file, as well as +for piping the artifact to another program. + +``` sh +$ just build greeter -Pout.txt +INFO: Requested target is [["@","","","greeter"],{}] +INFO: Analysed target [["@","","","greeter"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 1 actions, 0 trees, 0 blobs +INFO: Building [["@","","","greeter"],{}]. +INFO: Processed 1 actions, 1 cache hits. +INFO: Artifacts built, logical paths are: + out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] +Hello World +$ +``` + +Alternatively, we could also directly request the artifact `out.txt` +from *justbuild*'s CAS (content-addressable storage) and print it on +the command line via: + +``` sh +$ just install-cas [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] +Hello World +$ +``` + +The canonical way of requesting an object from the CAS is, as just +shown, to specify the full triple of hash, size, and type, separated by +colons and enclosed in square brackets. To simplify usage, the brackets +can be omitted and the size and type fields have the default values `0` +and `f`, respectively. While the default value for the size is wrong for +all but one string, the hash still determines the content of the file +and hence the local CAS is still able to retrieve the file. So the +typical invocation would simply specify the hash. + +``` sh +$ just install-cas 557db03de997c86a4a028e1ebd3a1ceb225be238 +Hello World +$ +``` + +Targets versus Files: The Stage +------------------------------- + +When invoking the `build` command, we had to specify the target +`greeter`, not the output file `out.txt`. While other build systems +allow requests specifying an output file, for *justbuild* this would +conflict with a fundamental design principle: staging; each target has +its own logical output space, the "stage", where it can put its +artifacts. We can, without any problem, add a second target also +generating a file `out.txt`. + +``` {.jsonc srcname="TARGETS"} +... +, "upper": + { "type": "generic" + , "cmds": ["cat name.txt | tr a-z A-Z > out.txt"] + , "outs": ["out.txt"] + , "deps": ["name.txt"] + } +... +``` + +As we only request targets, no conflicts arise. + +``` sh +$ just build upper -P out.txt +INFO: Requested target is [["@","","","upper"],{}] +INFO: Analysed target [["@","","","upper"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 1 actions, 0 trees, 0 blobs +INFO: Building [["@","","","upper"],{}]. +INFO: Processed 1 actions, 0 cache hits. +INFO: Artifacts built, logical paths are: + out.txt [83cf24cdfb4891a36bee93421930dd220766299a:6:f] +WORLD +$ just build greeter -P out.txt +INFO: Requested target is [["@","","","greeter"],{}] +INFO: Analysed target [["@","","","greeter"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 1 actions, 0 trees, 0 blobs +INFO: Building [["@","","","greeter"],{}]. +INFO: Processed 1 actions, 1 cache hits. +INFO: Artifacts built, logical paths are: + out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] +Hello World +$ +``` + +While one normally tries to design targets in such a way that they +don't have conflicting files if they should be used together, it is up +to the receiving target to decide what to do with those artifacts. A +built-in rule allowing to rearrange artifacts is `"install"`; a detailed +description of this rule can be found in the documentation. In the +simple case of a target producing precisely one file, the argument +`"files"` can be used to map that file to a new location. + +``` {.jsonc srcname="TARGETS"} +... +, "both": + {"type": "install", "files": {"hello.txt": "greeter", "upper.txt": "upper"}} +... +``` + +``` sh +$ just build both +INFO: Requested target is [["@","","","both"],{}] +INFO: Analysed target [["@","","","both"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 2 actions, 0 trees, 0 blobs +INFO: Building [["@","","","both"],{}]. +INFO: Processed 2 actions, 2 cache hits. +INFO: Artifacts built, logical paths are: + hello.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] + upper.txt [83cf24cdfb4891a36bee93421930dd220766299a:6:f] +$ +``` diff --git a/doc/tutorial/getting-started.org b/doc/tutorial/getting-started.org deleted file mode 100644 index 5a041397..00000000 --- a/doc/tutorial/getting-started.org +++ /dev/null @@ -1,212 +0,0 @@ -* Getting Started - -In order to use /justbuild/, first make sure that ~just~, ~just-mr~, and -~just-import-git~ are available in your ~PATH~. - -** Creating a new project - -/justbuild/ needs to know the root of the project worked on. By default, it -searches upwards from the current directory till it finds a marker. Currently, -we support three different markers: the files ~ROOT~ and ~WORKSPACE~ or the -directory ~.git~. Lets create a new project by creating one of those markers: - -#+BEGIN_SRC sh -$ touch ROOT -#+END_SRC - -** Creating a generic target - -By default, targets are described in ~TARGETS~ files. These files contain a -~JSON~ object with the target name as key and the target description as value. A -target description is an object with at least a single mandatory field: -~"type"~. This field specifies which rule (built-in or user-defined) to apply -for this target. - -A simple target that only executes commands can be created using the built-in -~"generic"~ rule, which requires at least one command and one output file or -directory. To create such a target, create the file ~TARGETS~ with the following -content: - -#+SRCNAME: TARGETS -#+BEGIN_SRC js -{ "greeter": - { "type": "generic" - , "cmds": ["echo -n 'Hello ' > out.txt", "cat name.txt >> out.txt"] - , "outs": ["out.txt"] - , "deps": ["name.txt"] - } -} -#+END_SRC - -In this example, the ~"greeter"~ target will run two commands to produce the -output file ~out.txt~. The second command depends on the input file ~name.txt~ -that we need to create as well: - -#+BEGIN_SRC sh -$ echo World > name.txt -#+END_SRC - -** Building a generic target - -To build a target, we need to run ~just~ with the subcommand ~build~: - -#+BEGIN_SRC sh -$ just build greeter -INFO: Requested target is [["@","","","greeter"],{}] -INFO: Analysed target [["@","","","greeter"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 1 actions, 0 trees, 0 blobs -INFO: Building [["@","","","greeter"],{}]. -INFO: Processed 1 actions, 0 cache hits. -INFO: Artifacts built, logical paths are: - out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] -$ -#+END_SRC - -The subcommand ~build~ just builds the artifact but does not stage it to any -user-defined location on the file system. Instead it reports a description -of the artifact consisting of ~git~ blob identifier, size, and type (in -this case ~f~ for non-executable file). To also stage the produced artifact to -the working directory, use the ~install~ subcommand and specify the output -directory: - -#+BEGIN_SRC sh -$ just install greeter -o . -INFO: Requested target is [["@","","","greeter"],{}] -INFO: Analysed target [["@","","","greeter"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 1 actions, 0 trees, 0 blobs -INFO: Building [["@","","","greeter"],{}]. -INFO: Processed 1 actions, 1 cache hits. -INFO: Artifacts can be found in: - /tmp/tutorial/out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] -$ cat out.txt -Hello World -$ -#+END_SRC - -Note that the ~install~ subcommand initiates the build a second time, without -executing any actions as all actions are being served from cache. The produced -artifact is identical, which is indicated by the same hash/size/type. - -If one is only interested in a single final artifact, one can -also request via the ~-P~ option that this artifact be written to -standard output after the build. As all messages are reported to -standard error, this can be used for both, interactively reading a -text file, as well as for piping the artifact to another program. - -#+BEGIN_SRC sh -$ just build greeter -Pout.txt -INFO: Requested target is [["@","","","greeter"],{}] -INFO: Analysed target [["@","","","greeter"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 1 actions, 0 trees, 0 blobs -INFO: Building [["@","","","greeter"],{}]. -INFO: Processed 1 actions, 1 cache hits. -INFO: Artifacts built, logical paths are: - out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] -Hello World -$ -#+END_SRC - -Alternatively, we could also directly request the artifact ~out.txt~ from -/justbuild/'s CAS (content-addressable storage) and print it on the command line -via: - -#+BEGIN_SRC sh -$ just install-cas [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] -Hello World -$ -#+END_SRC - -The canonical way of requesting an object from the CAS is, as just shown, to -specify the full triple of hash, size, and type, separated by colons and -enclosed in square brackets. To simplify usage, the brackets can be omitted -and the size and type fields have the default values ~0~ and ~f~, respectively. -While the default value for the size is wrong for all but one string, the hash -still determines the content of the file and hence the local CAS is still -able to retrieve the file. So the typical invocation would simply specify the -hash. - -#+BEGIN_SRC sh -$ just install-cas 557db03de997c86a4a028e1ebd3a1ceb225be238 -Hello World -$ -#+END_SRC - -** Targets versus Files: The Stage - -When invoking the ~build~ command, we had to specify the target ~greeter~, -not the output file ~out.txt~. While other build systems allow requests -specifying an output file, for /justbuild/ this would conflict with a -fundamental design principle: staging; each target has its own logical -output space, the "stage", where it can put its artifacts. We can, without -any problem, add a second target also generating a file ~out.txt~. - -#+SRCNAME: TARGETS -#+BEGIN_SRC js -... -, "upper": - { "type": "generic" - , "cmds": ["cat name.txt | tr a-z A-Z > out.txt"] - , "outs": ["out.txt"] - , "deps": ["name.txt"] - } -... -#+END_SRC - -As we only request targets, no conflicts arise. - -#+BEGIN_SRC sh -$ just build upper -P out.txt -INFO: Requested target is [["@","","","upper"],{}] -INFO: Analysed target [["@","","","upper"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 1 actions, 0 trees, 0 blobs -INFO: Building [["@","","","upper"],{}]. -INFO: Processed 1 actions, 0 cache hits. -INFO: Artifacts built, logical paths are: - out.txt [83cf24cdfb4891a36bee93421930dd220766299a:6:f] -WORLD -$ just build greeter -P out.txt -INFO: Requested target is [["@","","","greeter"],{}] -INFO: Analysed target [["@","","","greeter"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 1 actions, 0 trees, 0 blobs -INFO: Building [["@","","","greeter"],{}]. -INFO: Processed 1 actions, 1 cache hits. -INFO: Artifacts built, logical paths are: - out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] -Hello World -$ -#+END_SRC - -While one normally tries to design targets in such a way that they -don't have conflicting files if they should be used together, it is -up to the receiving target to decide what to do with those artifacts. -A built-in rule allowing to rearrange artifacts is ~"install"~; a -detailed description of this rule can be found in the documentation. -In the simple case of a target producing precisely one file, the -argument ~"files"~ can be used to map that file to a new location. - -#+SRCNAME: TARGETS -#+BEGIN_SRC js -... -, "both": - {"type": "install", "files": {"hello.txt": "greeter", "upper.txt": "upper"}} -... -#+END_SRC - -#+BEGIN_SRC sh -$ just build both -INFO: Requested target is [["@","","","both"],{}] -INFO: Analysed target [["@","","","both"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 2 actions, 0 trees, 0 blobs -INFO: Building [["@","","","both"],{}]. -INFO: Processed 2 actions, 2 cache hits. -INFO: Artifacts built, logical paths are: - hello.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] - upper.txt [83cf24cdfb4891a36bee93421930dd220766299a:6:f] -$ -#+END_SRC diff --git a/doc/tutorial/hello-world.md b/doc/tutorial/hello-world.md new file mode 100644 index 00000000..9af68f07 --- /dev/null +++ b/doc/tutorial/hello-world.md @@ -0,0 +1,379 @@ +Building C++ Hello World +======================== + +*justbuild* is a true language-agnostic (there are no more-equal +languages) and multi-repository build system. As a consequence, +high-level concepts (e.g., C++ binaries, C++ libraries, etc.) are not +hardcoded built-ins of the tool, but rather provided via a set of rules. +These rules can be specified as a true dependency to your project like +any other external repository your project might depend on. + +Setting up the Multi-Repository Configuration +--------------------------------------------- + +To build a project with multi-repository dependencies, we first need to +provide a configuration that declares the required repositories. Before +we begin, we need to declare where the root of our workspace is located +by creating an empty file `ROOT`: + +``` sh +$ touch ROOT +``` + +Second, we also need to create the multi-repository configuration +`repos.json` in the workspace root: + +``` {.jsonc srcname="repos.json"} +{ "main": "tutorial" +, "repositories": + { "rules-cc": + { "repository": + { "type": "git" + , "branch": "master" + , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" + , "repository": "https://github.com/just-buildsystem/rules-cc.git" + , "subdir": "rules" + } + } + , "tutorial": + { "repository": {"type": "file", "path": "."} + , "bindings": {"rules": "rules-cc"} + } + } +} +``` + +In that configuration, two repositories are defined: + +1. The `"rules-cc"` repository located in the subdirectory `rules` of + [just-buildsystem/rules-cc:123d8b03bf2440052626151c14c54abce2726e6f](https://github.com/just-buildsystem/rules-cc/tree/123d8b03bf2440052626151c14c54abce2726e6f), + which contains the high-level concepts for building C/C++ binaries + and libraries. + +2. The `"tutorial"` repository located at `.`, which contains the + targets that we want to build. It has a single dependency, which is + the *rules* that are needed to build the target. These rules are + bound via the open name `"rules"` to the just created repository + `"rules-cc"`. In this way, the entities provided by `"rules-cc"` can + be accessed from within the `"tutorial"` repository via the + fully-qualified name `["@", "rules", "", ""]`; + fully-qualified names (for rules, targets to build (like libraries, + binaries), etc) are given by a repository name, a path specifying a + directory within that repository (the "module") where the + specification file is located, and a symbolic name (i.e., an + arbitrary string that is used as key in the specification). + +The final repository configuration contains a single `JSON` object with +the key `"repositories"` referring to an object of repository names as +keys and repository descriptions as values. For convenience, the main +repository to pick is set to `"tutorial"`. + +Description of the helloworld target +------------------------------------ + +For this tutorial, we want to create a target `helloworld` that produces +a binary from the C++ source `main.cpp`. To define such a target, create +a `TARGETS` file with the following content: + +``` {.jsonc srcname="TARGETS"} +{ "helloworld": + { "type": ["@", "rules", "CC", "binary"] + , "name": ["helloworld"] + , "srcs": ["main.cpp"] + } +} +``` + +The `"type"` field refers to the rule `"binary"` from the module `"CC"` +of the `"rules"` repository. This rule additionally requires the string +field `"name"`, which specifies the name of the binary to produce; as +the generic interface of rules is to have fields either take a list of +strings or a list of targets, we have to specify the name as a list +(this rule will simply concatenate all strings given in this field). +Furthermore, at least one input to the binary is required, which can be +specified via the target fields `"srcs"` or `"deps"`. In our case, the +former is used, which contains our single source file (files are +considered targets). + +Now, the last file that is missing is the actual source file `main.cpp`: + +``` {.cpp srcname="main.cpp"} +#include + +int main() { + std::cout << "Hello world!\n"; + return 0; +} +``` + +Building the helloworld target +------------------------------ + +To build the `helloworld` target, we need specify it on the `just-mr` +command line: + +``` sh +$ just-mr build helloworld +INFO: Requested target is [["@","tutorial","","helloworld"],{}] +INFO: Analysed target [["@","tutorial","",helloworld"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 2 actions, 1 trees, 0 blobs +INFO: Building [["@","helloworld","","helloworld"],{}]. +INFO: Processed 2 actions, 0 cache hits. +INFO: Artifacts built, logical paths are: + helloworld [b5cfca8b810adc4686f5cac00258a137c5d4a3ba:17088:x] +$ +``` + +Note that the target is taken from the `tutorial` repository, as it +specified as the main repository in `repos.json`. If targets from other +repositories should be build, the repository to use must be specified +via the `--main` option. + +`just-mr` reads the repository configuration, fetches externals (if +any), generates the actual build configuration, and stores it in its +cache directory (by default under `$HOME/.cache/just`). Afterwards, the +generated configuration is used to call the `just` binary, which +performs the actual build. + +Note that these two programs, `just-mr` and `just`, can also be run +individually. To do so, first run `just-mr` with `setup` and capture the +path to the generated build configuration from stdout by assigning it to +a shell variable (e.g., `CONF`). Afterwards, `just` can be called to +perform the actual build by explicitly specifying the configuration file +via `-C`: + +``` sh +$ CONF=$(just-mr setup tutorial) +$ just build -C $CONF helloworld +``` + +Note that `just-mr` only needs to be run the very first time and only +once again whenever the `repos.json` file is modified. + +By default, the BSD-default compiler front-ends (which are also defined +for most Linux distributions) `cc` and `c++` are used for C and C++ +(variables `"CC"` and `"CXX"`). If you want to temporarily use different +defaults, you can use `-D` to provide a JSON object that sets different +default variables. For instance, to use Clang as C++ compiler for a +single build invocation, you can use the following command to provide an +object that sets `"CXX"` to `"clang++"`: + +``` sh +$ just-mr build helloworld -D'{"CXX":"clang++"}' +INFO: Requested target is [["@","tutorial","","helloworld"],{"CXX":"clang++"}] +INFO: Analysed target [["@","tutorial","","helloworld"],{"CXX":"clang++"}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 2 actions, 1 trees, 0 blobs +INFO: Building [["@","tutorial","","helloworld"],{"CXX":"clang++"}]. +INFO: Processed 2 actions, 0 cache hits. +INFO: Artifacts built, logical paths are: + helloworld [b8cf7b8579d9dc7172b61660139e2c14521cedae:16944:x] +$ +``` + +Defining project defaults +------------------------- + +To define a custom set of defaults (toolchain and compile flags) for +your project, you need to create a separate file root for providing +required `TARGETS` file, which contains the `"defaults"` target that +should be used by the rules. This file root is then used as the *target +root* for the rules, i.e., the search path for `TARGETS` files. In this +way, the description of the `"defaults"` target is provided in a +separate file root, to keep the rules repository independent of these +definitions. + +We will call the new file root `tutorial-defaults` and need to create a +module directory `CC` in it: + +``` sh +$ mkdir -p ./tutorial-defaults/CC +``` + +In that module, we need to create the file +`tutorial-defaults/CC/TARGETS` that contains the target `"defaults"` and +specifies which toolchain and compile flags to use; it has to specify +the complete toolchain, but can specify a `"base"` toolchain to inherit +from. In our case, we don't use any base, but specify all the required +fields directly. + +``` {.jsonc srcname="tutorial-defaults/CC/TARGETS"} +{ "defaults": + { "type": ["CC", "defaults"] + , "CC": ["cc"] + , "CXX": ["c++"] + , "CFLAGS": ["-O2", "-Wall"] + , "CXXFLAGS": ["-O2", "-Wall"] + , "AR": ["ar"] + , "PATH": ["/bin", "/usr/bin"] + } +} +``` + +To use the project defaults, modify the existing `repos.json` to reflect +the following content: + +``` {.jsonc srcname="repos.json"} +{ "main": "tutorial" +, "repositories": + { "rules-cc": + { "repository": + { "type": "git" + , "branch": "master" + , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" + , "repository": "https://github.com/just-buildsystem/rules-cc.git" + , "subdir": "rules" + } + , "target_root": "tutorial-defaults" + , "rule_root": "rules-cc" + } + , "tutorial": + { "repository": {"type": "file", "path": "."} + , "bindings": {"rules": "rules-cc"} + } + , "tutorial-defaults": + { "repository": {"type": "file", "path": "./tutorial-defaults"} + } + } +} +``` + +Note that the `"defaults"` target uses the rule `["CC", "defaults"]` +without specifying any external repository (e.g., +`["@", "rules", ...]`). This is because `"tutorial-defaults"` is not a +full-fledged repository but merely a file root that is considered local +to the `"rules-cc"` repository. In fact, the `"rules-cc"` repository +cannot refer to any external repository as it does not have any defined +bindings. + +To rebuild the project, we need to rerun `just-mr` (note that due to +configuration changes, rerunning only `just` would not suffice): + +``` sh +$ just-mr build helloworld +INFO: Requested target is [["@","tutorial","","helloworld"],{}] +INFO: Analysed target [["@","tutorial","","helloworld"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 2 actions, 1 trees, 0 blobs +INFO: Building [["@","tutorial","","helloworld"],{}]. +INFO: Processed 2 actions, 0 cache hits. +INFO: Artifacts built, logical paths are: + helloworld [487dc9e47b978877ed2f7d80b3395ce84b23be92:16992:x] +$ +``` + +Note that the output binary may have changed due to different defaults. + +Modeling target dependencies +---------------------------- + +For demonstration purposes, we will separate the print statements into a +static library `greet`, which will become a dependency to our binary. +Therefore, we create a new subdirectory `greet` with the files +`greet/greet.hpp`: + +``` {.cpp srcname="greet/greet.hpp"} +#include + +void greet(std::string const& s); +``` + +and `greet/greet.cpp`: + +``` {.cpp srcname="greet/greet.cpp"} +#include "greet.hpp" +#include + +void greet(std::string const& s) { + std::cout << "Hello " << s << "!\n"; +} +``` + +These files can now be used to create a static library `libgreet.a`. To +do so, we need to create the following target description in +`greet/TARGETS`: + +``` {.jsonc srcname="greet/TARGETS"} +{ "greet": + { "type": ["@", "rules", "CC", "library"] + , "name": ["greet"] + , "hdrs": ["greet.hpp"] + , "srcs": ["greet.cpp"] + , "stage": ["greet"] + } +} +``` + +Similar to `"binary"`, we have to provide a name and source file. +Additionally, a library has public headers defined via `"hdrs"` and an +optional staging directory `"stage"` (default value `"."`). The staging +directory specifies where the consumer of this library can expect to +find the library's artifacts. Note that this does not need to reflect +the location on the file system (i.e., a full-qualified path like +`["com", "example", "utils", "greet"]` could be used to distinguish it +from greeting libraries of other projects). The staging directory does +not only affect the main artifact `libgreet.a` but also it's +*runfiles*, a second set of artifacts, usually those a consumer needs to +make proper use the actual artifact; in the case of a library, the +runfiles are its public headers. Hence, the public header will be staged +to `"greet/greet.hpp"`. With that knowledge, we can now perform the +necessary modifications to `main.cpp`: + +``` {.cpp srcname="main.cpp"} +#include "greet/greet.hpp" + +int main() { + greet("Universe"); + return 0; +} +``` + +The target `"helloworld"` will have a direct dependency to the target +`"greet"` of the module `"greet"` in the top-level `TARGETS` file: + +``` {.jsonc srcname="TARGETS"} +{ "helloworld": + { "type": ["@", "rules", "CC", "binary"] + , "name": ["helloworld"] + , "srcs": ["main.cpp"] + , "private-deps": [["greet", "greet"]] + } +} +``` + +Note that there is no need to explicitly specify `"greet"`'s public +headers here as the appropriate artifacts of dependencies are +automatically added to the inputs of compile and link actions. The new +binary can be built with the same command as before (no need to rerun +`just-mr`): + +``` sh +$ just-mr build helloworld +INFO: Requested target is [["@","tutorial","","helloworld"],{}] +INFO: Analysed target [["@","tutorial","","helloworld"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 4 actions, 2 trees, 0 blobs +INFO: Building [["@","tutorial","","helloworld"],{}]. +INFO: Processed 4 actions, 0 cache hits. +INFO: Artifacts built, logical paths are: + helloworld [2b81e3177afc382452a2df9f294d3df90a9ccaf0:17664:x] +$ +``` + +To only build the static library target `"greet"` from module `"greet"`, +run the following command: + +``` sh +$ just-mr build greet greet +INFO: Requested target is [["@","tutorial","greet","greet"],{}] +INFO: Analysed target [["@","tutorial","greet","greet"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 2 actions, 1 trees, 0 blobs +INFO: Building [["@","tutorial","greet","greet"],{}]. +INFO: Processed 2 actions, 2 cache hits. +INFO: Artifacts built, logical paths are: + greet/libgreet.a [83ed406e21f285337b0c9bd5011f56f656bba683:2992:f] + (1 runfiles omitted.) +$ +``` diff --git a/doc/tutorial/hello-world.org b/doc/tutorial/hello-world.org deleted file mode 100644 index 342eaf82..00000000 --- a/doc/tutorial/hello-world.org +++ /dev/null @@ -1,370 +0,0 @@ -* Building C++ Hello World - -/justbuild/ is a true language-agnostic (there are no more-equal languages) and -multi-repository build system. As a consequence, high-level concepts (e.g., C++ -binaries, C++ libraries, etc.) are not hardcoded built-ins of the tool, but -rather provided via a set of rules. These rules can be specified as a true -dependency to your project like any other external repository your project might -depend on. - -** Setting up the Multi-Repository Configuration - -To build a project with multi-repository dependencies, we first need to provide -a configuration that declares the required repositories. Before we begin, we -need to declare where the root of our workspace is located by creating an empty -file ~ROOT~: - -#+BEGIN_SRC sh -$ touch ROOT -#+END_SRC - -Second, we also need to create the multi-repository configuration ~repos.json~ -in the workspace root: - -#+SRCNAME: repos.json -#+BEGIN_SRC js -{ "main": "tutorial" -, "repositories": - { "rules-cc": - { "repository": - { "type": "git" - , "branch": "master" - , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" - , "repository": "https://github.com/just-buildsystem/rules-cc.git" - , "subdir": "rules" - } - } - , "tutorial": - { "repository": {"type": "file", "path": "."} - , "bindings": {"rules": "rules-cc"} - } - } -} -#+END_SRC - -In that configuration, two repositories are defined: - - 1. The ~"rules-cc"~ repository located in the subdirectory ~rules~ of - [[https://github.com/just-buildsystem/rules-cc/tree/123d8b03bf2440052626151c14c54abce2726e6f][just-buildsystem/rules-cc:123d8b03bf2440052626151c14c54abce2726e6f]], - which contains the high-level concepts for building C/C++ binaries and - libraries. - - 2. The ~"tutorial"~ repository located at ~.~, which contains the targets that - we want to build. It has a single dependency, which is the /rules/ that are - needed to build the target. These rules are bound via the open name - ~"rules"~ to the just created repository ~"rules-cc"~. In this way, the - entities provided by ~"rules-cc"~ can be accessed from within the - ~"tutorial"~ repository via the fully-qualified name - ~["@", "rules", "", ""]~; fully-qualified - names (for rules, targets to build (like libraries, binaries), - etc) are given by a repository name, a path specifying a - directory within that repository (the "module") where the - specification file is located, and a symbolic name (i.e., an - arbitrary string that is used as key in the specification). - -The final repository configuration contains a single ~JSON~ object with the key -~"repositories"~ referring to an object of repository names as keys and -repository descriptions as values. For convenience, the main repository to pick -is set to ~"tutorial"~. - -** Description of the helloworld target - -For this tutorial, we want to create a target ~helloworld~ that produces a -binary from the C++ source ~main.cpp~. To define such a target, create a -~TARGETS~ file with the following content: - -#+SRCNAME: TARGETS -#+BEGIN_SRC js -{ "helloworld": - { "type": ["@", "rules", "CC", "binary"] - , "name": ["helloworld"] - , "srcs": ["main.cpp"] - } -} -#+END_SRC - -The ~"type"~ field refers to the rule ~"binary"~ from the module ~"CC"~ of the -~"rules"~ repository. This rule additionally requires the string field ~"name"~, -which specifies the name of the binary to produce; as the generic interface of -rules is to have fields either take a list of strings or a list of targets, -we have to specify the name as a list (this rule will simply concatenate all -strings given in this field). Furthermore, at least one -input to the binary is required, which can be specified via the target fields -~"srcs"~ or ~"deps"~. In our case, the former is used, which contains our single -source file (files are considered targets). - -Now, the last file that is missing is the actual source file ~main.cpp~: - -#+SRCNAME: main.cpp -#+BEGIN_SRC cpp -#include - -int main() { - std::cout << "Hello world!\n"; - return 0; -} -#+END_SRC - -** Building the helloworld target - -To build the ~helloworld~ target, we need specify it on the ~just-mr~ command -line: - -#+BEGIN_SRC sh -$ just-mr build helloworld -INFO: Requested target is [["@","tutorial","","helloworld"],{}] -INFO: Analysed target [["@","tutorial","",helloworld"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 2 actions, 1 trees, 0 blobs -INFO: Building [["@","helloworld","","helloworld"],{}]. -INFO: Processed 2 actions, 0 cache hits. -INFO: Artifacts built, logical paths are: - helloworld [b5cfca8b810adc4686f5cac00258a137c5d4a3ba:17088:x] -$ -#+END_SRC - -Note that the target is taken from the ~tutorial~ repository, as it specified as -the main repository in ~repos.json~. If targets from other repositories should -be build, the repository to use must be specified via the ~--main~ option. - -~just-mr~ reads the repository configuration, fetches externals (if any), -generates the actual build configuration, and stores it in its cache directory -(by default under ~$HOME/.cache/just~). Afterwards, the generated configuration -is used to call the ~just~ binary, which performs the actual build. - -Note that these two programs, ~just-mr~ and ~just~, can also be run -individually. To do so, first run ~just-mr~ with ~setup~ and capture the path to -the generated build configuration from stdout by assigning it to a shell -variable (e.g., ~CONF~). Afterwards, ~just~ can be called to perform the actual -build by explicitly specifying the configuration file via ~-C~: - -#+BEGIN_SRC sh -$ CONF=$(just-mr setup tutorial) -$ just build -C $CONF helloworld -#+END_SRC - -Note that ~just-mr~ only needs to be run the very first time and only once again -whenever the ~repos.json~ file is modified. - -By default, the BSD-default compiler front-ends (which are also defined for most -Linux distributions) ~cc~ and ~c++~ are used for C and C++ (variables ~"CC"~ and -~"CXX"~). If you want to temporarily use different defaults, you can use ~-D~ to -provide a JSON object that sets different default variables. For instance, to -use Clang as C++ compiler for a single build invocation, you can use the -following command to provide an object that sets ~"CXX"~ to ~"clang++"~: - -#+BEGIN_SRC sh -$ just-mr build helloworld -D'{"CXX":"clang++"}' -INFO: Requested target is [["@","tutorial","","helloworld"],{"CXX":"clang++"}] -INFO: Analysed target [["@","tutorial","","helloworld"],{"CXX":"clang++"}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 2 actions, 1 trees, 0 blobs -INFO: Building [["@","tutorial","","helloworld"],{"CXX":"clang++"}]. -INFO: Processed 2 actions, 0 cache hits. -INFO: Artifacts built, logical paths are: - helloworld [b8cf7b8579d9dc7172b61660139e2c14521cedae:16944:x] -$ -#+END_SRC - -** Defining project defaults - -To define a custom set of defaults (toolchain and compile flags) for your -project, you need to create a separate file root for providing required -~TARGETS~ file, which contains the ~"defaults"~ target that should be used by -the rules. This file root is then used as the /target root/ for the rules, i.e., -the search path for ~TARGETS~ files. In this way, the description of the -~"defaults"~ target is provided in a separate file root, to keep the rules -repository independent of these definitions. - -We will call the new file root ~tutorial-defaults~ and need to create a module -directory ~CC~ in it: - -#+BEGIN_SRC sh -$ mkdir -p ./tutorial-defaults/CC -#+END_SRC - -In that module, we need to create the file ~tutorial-defaults/CC/TARGETS~ that -contains the target ~"defaults"~ and specifies which toolchain and compile flags -to use; it has to specify the complete toolchain, but can specify a ~"base"~ -toolchain to inherit from. In our case, we don't use any base, but specify all -the required fields directly. - -#+SRCNAME: tutorial-defaults/CC/TARGETS -#+BEGIN_SRC js -{ "defaults": - { "type": ["CC", "defaults"] - , "CC": ["cc"] - , "CXX": ["c++"] - , "CFLAGS": ["-O2", "-Wall"] - , "CXXFLAGS": ["-O2", "-Wall"] - , "AR": ["ar"] - , "PATH": ["/bin", "/usr/bin"] - } -} -#+END_SRC - -To use the project defaults, modify the existing ~repos.json~ to reflect the -following content: - -#+SRCNAME: repos.json -#+BEGIN_SRC js -{ "main": "tutorial" -, "repositories": - { "rules-cc": - { "repository": - { "type": "git" - , "branch": "master" - , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" - , "repository": "https://github.com/just-buildsystem/rules-cc.git" - , "subdir": "rules" - } - , "target_root": "tutorial-defaults" - , "rule_root": "rules-cc" - } - , "tutorial": - { "repository": {"type": "file", "path": "."} - , "bindings": {"rules": "rules-cc"} - } - , "tutorial-defaults": - { "repository": {"type": "file", "path": "./tutorial-defaults"} - } - } -} -#+END_SRC - -Note that the ~"defaults"~ target uses the rule ~["CC", "defaults"]~ without -specifying any external repository (e.g., ~["@", "rules", ...]~). This is -because ~"tutorial-defaults"~ is not a full-fledged repository but merely a file -root that is considered local to the ~"rules-cc"~ repository. In fact, the -~"rules-cc"~ repository cannot refer to any external repository as it does not -have any defined bindings. - -To rebuild the project, we need to rerun ~just-mr~ (note that due to -configuration changes, rerunning only ~just~ would not suffice): - -#+BEGIN_SRC sh -$ just-mr build helloworld -INFO: Requested target is [["@","tutorial","","helloworld"],{}] -INFO: Analysed target [["@","tutorial","","helloworld"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 2 actions, 1 trees, 0 blobs -INFO: Building [["@","tutorial","","helloworld"],{}]. -INFO: Processed 2 actions, 0 cache hits. -INFO: Artifacts built, logical paths are: - helloworld [487dc9e47b978877ed2f7d80b3395ce84b23be92:16992:x] -$ -#+END_SRC - -Note that the output binary may have changed due to different defaults. - -** Modeling target dependencies - -For demonstration purposes, we will separate the print statements into a static -library ~greet~, which will become a dependency to our binary. Therefore, we -create a new subdirectory ~greet~ with the files ~greet/greet.hpp~: - -#+SRCNAME: greet/greet.hpp -#+BEGIN_SRC cpp -#include - -void greet(std::string const& s); -#+END_SRC - -and ~greet/greet.cpp~: - -#+SRCNAME: greet/greet.cpp -#+BEGIN_SRC cpp -#include "greet.hpp" -#include - -void greet(std::string const& s) { - std::cout << "Hello " << s << "!\n"; -} -#+END_SRC - -These files can now be used to create a static library ~libgreet.a~. To do so, -we need to create the following target description in ~greet/TARGETS~: - -#+SRCNAME: greet/TARGETS -#+BEGIN_SRC js -{ "greet": - { "type": ["@", "rules", "CC", "library"] - , "name": ["greet"] - , "hdrs": ["greet.hpp"] - , "srcs": ["greet.cpp"] - , "stage": ["greet"] - } -} -#+END_SRC - -Similar to ~"binary"~, we have to provide a name and source file. Additionally, -a library has public headers defined via ~"hdrs"~ and an optional staging -directory ~"stage"~ (default value ~"."~). The staging directory specifies where -the consumer of this library can expect to find the library's artifacts. Note -that this does not need to reflect the location on the file system (i.e., a -full-qualified path like ~["com", "example", "utils", "greet"]~ could be used to -distinguish it from greeting libraries of other projects). The staging directory -does not only affect the main artifact ~libgreet.a~ but also it's /runfiles/, -a second set of artifacts, usually those a consumer needs to make proper use the -actual artifact; in the case of a library, the runfiles are its public headers. -Hence, the public header will be staged to ~"greet/greet.hpp"~. With that -knowledge, we can now perform the necessary modifications to ~main.cpp~: - -#+SRCNAME: main.cpp -#+BEGIN_SRC cpp -#include "greet/greet.hpp" - -int main() { - greet("Universe"); - return 0; -} -#+END_SRC - -The target ~"helloworld"~ will have a direct dependency to the target ~"greet"~ -of the module ~"greet"~ in the top-level ~TARGETS~ file: - -#+SRCNAME: TARGETS -#+BEGIN_SRC js -{ "helloworld": - { "type": ["@", "rules", "CC", "binary"] - , "name": ["helloworld"] - , "srcs": ["main.cpp"] - , "private-deps": [["greet", "greet"]] - } -} -#+END_SRC - -Note that there is no need to explicitly specify ~"greet"~'s public headers here -as the appropriate artifacts of dependencies are automatically added to the -inputs of compile and link actions. The new binary can be built with the same -command as before (no need to rerun ~just-mr~): - -#+BEGIN_SRC sh -$ just-mr build helloworld -INFO: Requested target is [["@","tutorial","","helloworld"],{}] -INFO: Analysed target [["@","tutorial","","helloworld"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 4 actions, 2 trees, 0 blobs -INFO: Building [["@","tutorial","","helloworld"],{}]. -INFO: Processed 4 actions, 0 cache hits. -INFO: Artifacts built, logical paths are: - helloworld [2b81e3177afc382452a2df9f294d3df90a9ccaf0:17664:x] -$ -#+END_SRC - -To only build the static library target ~"greet"~ from module ~"greet"~, run the -following command: - -#+BEGIN_SRC sh -$ just-mr build greet greet -INFO: Requested target is [["@","tutorial","greet","greet"],{}] -INFO: Analysed target [["@","tutorial","greet","greet"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 2 actions, 1 trees, 0 blobs -INFO: Building [["@","tutorial","greet","greet"],{}]. -INFO: Processed 2 actions, 2 cache hits. -INFO: Artifacts built, logical paths are: - greet/libgreet.a [83ed406e21f285337b0c9bd5011f56f656bba683:2992:f] - (1 runfiles omitted.) -$ -#+END_SRC diff --git a/doc/tutorial/proto.md b/doc/tutorial/proto.md new file mode 100644 index 00000000..8a04e373 --- /dev/null +++ b/doc/tutorial/proto.md @@ -0,0 +1,474 @@ +Using protocol buffers +====================== + +The rules *justbuild* uses for itself also support protocol buffers. +This tutorial shows how to use those rules and the targets associated +with them. It is not a tutorial on protocol buffers itself; rather, it +is assumed that the reader has some knowledge on [protocol +buffers](https://developers.google.com/protocol-buffers/). + +Setting up the repository configuration +--------------------------------------- + +Before we begin, we first need to declare where the root of our +workspace is located by creating the empty file `ROOT`: + +``` sh +$ touch ROOT +``` + +The `protobuf` repository conveniently contains an +[example](https://github.com/protocolbuffers/protobuf/tree/v3.12.4/examples), +so we can use this and just add our own target files. We create file +`repos.template.json` as follows. + +``` {.jsonc srcname="repos.template.json"} +{ "repositories": + { "": + { "repository": + { "type": "zip" + , "content": "7af7165b585e4aed714555a747b6822376176ef4" + , "fetch": "https://github.com/protocolbuffers/protobuf/archive/refs/tags/v3.12.4.zip" + , "subdir": "protobuf-3.12.4/examples" + } + , "target_root": "tutorial" + , "bindings": {"rules": "rules-cc"} + } + , "tutorial": {"repository": {"type": "file", "path": "."}} + } +} +``` + +The missing entry `"rules-cc"` refers to our C/C++ build rules provided +[online](https://github.com/just-buildsystem/rules-cc). These rules +support protobuf if the dependency `"protoc"` is provided. To import +this rule repository including the required transitive dependencies for +protobuf, the `bin/just-import-git` script with option `--as rules-cc` +can be used to generate the actual `repos.json`: + +``` sh +$ just-import-git -C repos.template.json -b master --as rules-cc https://github.com/just-buildsystem/rules-cc > repos.json +``` + +To build the example with `just`, the only task is to write targets +files. As that contains a couple of new concepts, we will do this step +by step. + +The proto library +----------------- + +First, we have to declare the proto library. In this case, it only +contains the file `addressbook.proto` and has no dependencies. To +declare the library, create a `TARGETS` file with the following content: + +``` {.jsonc srcname="TARGETS"} +{ "address": + { "type": ["@", "rules", "proto", "library"] + , "name": ["addressbook"] + , "srcs": ["addressbook.proto"] + } +} +``` + +In general, proto libraries could also depend on other proto libraries; +those would be added to the `"deps"` field. + +When building the library, there's very little to do. + +``` sh +$ just-mr build address +INFO: Requested target is [["@","","","address"],{}] +INFO: Analysed target [["@","","","address"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 0 actions, 0 trees, 0 blobs +INFO: Building [["@","","","address"],{}]. +INFO: Processed 0 actions, 0 cache hits. +INFO: Artifacts built, logical paths are: +$ +``` + +On the other hand, what did we expect? A proto library is an abstract +description of a protocol, so, as long as we don't specify for which +language we want to have bindings, there is nothing to generate. + +Nevertheless, a proto library target is not empty. In fact, it can't be +empty, as other targets can only access the values of a target and have +no insights into its definitions. We already relied on this design +principle implicitly, when we exploited target-level caching for our +external dependencies and did not even construct the dependency graph +for that target. A proto library simply provides the dependency +structure of the `.proto` files. + +``` sh +$ just-mr analyse --dump-nodes - address +INFO: Requested target is [["@","","","address"],{}] +INFO: Result of target [["@","","","address"],{}]: { + "artifacts": { + }, + "provides": { + "proto": [ + {"id":"2a483a2de7f25c1bc066e47245f55ec9a2d4a719","type":"NODE"} + ] + }, + "runfiles": { + } + } +INFO: Target nodes of target [["@","","","address"],{}]: +{ + "089f6cae7ca77bb786578d3e0138b6ff445c5c92": { + "result": { + "artifact_stage": { + "addressbook.proto": { + "data": { + "file_type": "f", + "id": "b4b33b4c658924f0321ab4e7a9dc9cf8da1acec3", + "size": 1234 + }, + "type": "KNOWN" + } + }, + "provides": { + }, + "runfiles": { + } + }, + "type": "VALUE_NODE" + }, + "2a483a2de7f25c1bc066e47245f55ec9a2d4a719": { + "node_type": "library", + "string_fields": { + "name": ["addressbook"], + "stage": [""] + }, + "target_fields": { + "deps": [], + "srcs": [{"id":"089f6cae7ca77bb786578d3e0138b6ff445c5c92","type":"NODE"}] + }, + "type": "ABSTRACT_NODE" + } +} +$ +``` + +The target has one provider `"proto"`, which is a node. Nodes are an +abstract representation of a target graph. More precisely, there are two +kind of nodes, and our example contains one of each. + +The simple kind of nodes are the value nodes; they represent a target +that has a fixed value, and hence are given by artifacts, runfiles, and +provided data. In our case, we have one value node, the one for the +`.proto` file. + +The other kind of nodes are the abstract nodes. They describe the +arguments for a target, but only have an abstract name (i.e., a string) +for the rule. Combining such an abstract target with a binding for the +abstract rule names gives a concrete "anonymous" target that, in our +case, will generate the library with the bindings for the concrete +language. In this example, the abstract name is `"library"`. The +alternative in our proto rules would have been `"service library"`, for +proto libraries that also contain `rpc` definitions (which is used by +[gRPC](https://grpc.io/)). + +Using proto libraries +--------------------- + +Using proto libraries requires, as discussed, bindings for the abstract +names. Fortunately, our `CC` rules are aware of proto libraries, so we +can simply use them. Our target file hence continues as follows. + +``` {.jsonc srcname="TARGETS"} +... +, "add_person": + { "type": ["@", "rules", "CC", "binary"] + , "name": ["add_person"] + , "srcs": ["add_person.cc"] + , "private-proto": ["address"] + } +, "list_people": + { "type": ["@", "rules", "CC", "binary"] + , "name": ["list_people"] + , "srcs": ["list_people.cc"] + , "private-proto": ["address"] + } +... +``` + +The first time, we build a target that requires the proto compiler (in +that particular version, built in that particular way), it takes a bit +of time, as the proto compiler has to be built. But in follow-up builds, +also in different projects, the target-level cache is filled already. + +``` sh +$ just-mr build add_person +... +$ just-mr build add_person +INFO: Requested target is [["@","","","add_person"],{}] +INFO: Analysed target [["@","","","add_person"],{}] +INFO: Export targets found: 3 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 5 actions, 2 trees, 0 blobs +INFO: Building [["@","","","add_person"],{}]. +INFO: Processed 5 actions, 5 cache hits. +INFO: Artifacts built, logical paths are: + add_person [bcbb3deabfe0d77e6d3ea35615336a2f59a1b0aa:2285928:x] +$ +``` + +If we look at the actions associated with the binary, we find that those +are still the two actions we expect: a compile action and a link action. + +``` sh +$ just-mr analyse add_person --dump-actions - +INFO: Requested target is [["@","","","add_person"],{}] +INFO: Result of target [["@","","","add_person"],{}]: { + "artifacts": { + "add_person": {"data":{"id":"fcf211e2291b2867375e915538ce04cb4dfae86d","path":"add_person"},"type":"ACTION"} + }, + "provides": { + }, + "runfiles": { + } + } +INFO: Actions for target [["@","","","add_person"],{}]: +[ + { + "command": ["c++","-I","work","-isystem","include","-c","work/add_person.cc","-o","work/add_person.o"], + "env": { + "PATH": "/bin:/usr/bin" + }, + "input": { + ... + }, + "output": ["work/add_person.o"] + }, + { + "command": ["c++","-o","add_person","add_person.o","libaddressbook.a","libprotobuf.a","libprotobuf_lite.a","libzlib.a"], + "env": { + "PATH": "/bin:/usr/bin" + }, + "input": { + ... + }, + "output": ["add_person"] + } +] +$ +``` + +As discussed, the `libaddressbook.a` that is conveniently available +during the linking of the binary (as well as the `addressbook.pb.h` +available in the `include` tree for the compile action) are generated by +an anonymous target. Using that during the build we already filled the +target-level cache, we can have a look at all targets still analysed. In +the one anonymous target, we find again the abstract node we discussed +earlier. + +``` sh +$ just-mr analyse add_person --dump-targets - +INFO: Requested target is [["@","","","add_person"],{}] +INFO: Result of target [["@","","","add_person"],{}]: { + "artifacts": { + "add_person": {"data":{"id":"fcf211e2291b2867375e915538ce04cb4dfae86d","path":"add_person"},"type":"ACTION"} + }, + "provides": { + }, + "runfiles": { + } + } +INFO: List of analysed targets: +{ + "#": { + "eda46ea21de25033ff7250e6a4cdc0b2c24be0c7": { + "2a483a2de7f25c1bc066e47245f55ec9a2d4a719": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}] + } + }, + "@": { + "": { + "": { + "add_person": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}], + "address": [{}] + } + }, + "rules-cc": { + "CC": { + "defaults": [{}] + } + }, + "rules-cc/just/protobuf": { + "": { + "C++ runtime": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}], + "protoc": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}], + "well_known_protos": [{}] + } + } + } +} +$ +``` + +It should be noted, however, that this tight integration of proto into +our `C++` rules is just convenience of our code base. If we had to +cooperate with rules not aware of proto, we could have created a +separate rule delegating the library creation to the anonymous target +and then simply reflecting the values of that target. In fact, we could +simply use an empty library with a public `proto` dependency for this +purpose. + +``` {.jsonc srcname="TARGETS"} +... +, "address proto library": + {"type": ["@", "rules", "CC", "library"], "proto": ["address"]} +... +``` + +``` sh +$ just-mr analyse 'address proto library' +... +INFO: Requested target is [["@","","","address proto library"],{}] +INFO: Result of target [["@","","","address proto library"],{}]: { + "artifacts": { + }, + "provides": { + ... + "compile-deps": { + "addressbook.pb.h": {"data":{"id":"6d70cd10fabcbc7591cd82aae2f100cca39d3879","path":"work/addressbook.pb.h"},"type":"ACTION"}, + ... + }, + "link-args": [ + "libaddressbook.a", + ... + ], + "link-deps": { + "libaddressbook.a": {"data":{"id":"753073bd026b6470138c47e004469dd1d3df08d4","path":"libaddressbook.a"},"type":"ACTION"}, + ... + }, + ... + }, + "runfiles": { + } + } +$ +``` + +Adding a test +------------- + +Finally, let's add a test. As we use the `protobuf` repository as +workspace root, we add the test script ad hoc into a targets file, using +the `"file_gen"` rule. For debugging a potentially failing test, we also +keep the intermediate files the test generates. Create a top-level +`TARGETS` file with the following content: + +``` {.jsonc srcname="TARGETS"} +... +, "test.sh": + { "type": "file_gen" + , "name": "test.sh" + , "data": + { "type": "join" + , "separator": "\n" + , "$1": + [ "set -e" + , "(echo 12345; echo 'John Doe'; echo 'jdoe@example.org'; echo) | ./add_person addressbook.data" + , "./list_people addressbook.data > out.txt" + , "grep Doe out.txt" + ] + } + } +, "test": + { "type": ["@", "rules", "shell/test", "script"] + , "name": ["read-write-test"] + , "test": ["test.sh"] + , "deps": ["add_person", "list_people"] + , "keep": ["addressbook.data", "out.txt"] + } +... +``` + +That example also shows why it is important that the generation of the +language bindings is delegated to an anonymous target: we want to +analyse only once how the `C++` bindings are generated. Nevertheless, +many targets can depend (directly or indirectly) on the same proto +library. And, indeed, analysing the test, we get the expected additional +targets and the one anonymous target is reused by both binaries. + +``` sh +$ just-mr analyse test --dump-targets - +INFO: Requested target is [["@","","","test"],{}] +INFO: Result of target [["@","","","test"],{}]: { + "artifacts": { + "result": {"data":{"id":"20967787c42a289f5598249e696f851dde50065c","path":"result"},"type":"ACTION"}, + "stderr": {"data":{"id":"20967787c42a289f5598249e696f851dde50065c","path":"stderr"},"type":"ACTION"}, + "stdout": {"data":{"id":"20967787c42a289f5598249e696f851dde50065c","path":"stdout"},"type":"ACTION"}, + "time-start": {"data":{"id":"20967787c42a289f5598249e696f851dde50065c","path":"time-start"},"type":"ACTION"}, + "time-stop": {"data":{"id":"20967787c42a289f5598249e696f851dde50065c","path":"time-stop"},"type":"ACTION"}, + "work/addressbook.data": {"data":{"id":"20967787c42a289f5598249e696f851dde50065c","path":"work/addressbook.data"},"type":"ACTION"}, + "work/out.txt": {"data":{"id":"20967787c42a289f5598249e696f851dde50065c","path":"work/out.txt"},"type":"ACTION"} + }, + "provides": { + }, + "runfiles": { + "read-write-test": {"data":{"id":"c9d7bfc5bc8448bfef25b4e73e4494560bf6c350"},"type":"TREE"} + } + } +INFO: List of analysed targets: +{ + "#": { + "eda46ea21de25033ff7250e6a4cdc0b2c24be0c7": { + "2a483a2de7f25c1bc066e47245f55ec9a2d4a719": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}] + } + }, + "@": { + "": { + "": { + "add_person": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}], + "address": [{}], + "list_people": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}], + "test": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"RUNS_PER_TEST":null,"TEST_ENV":null}], + "test.sh": [{}] + } + }, + "rules-cc": { + "CC": { + "defaults": [{}] + } + }, + "rules-cc/just/protobuf": { + "": { + "C++ runtime": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}], + "protoc": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}], + "well_known_protos": [{}] + } + } + } +} +INFO: Target tainted ["test"]. +$ +``` + +Finally, the test passes and the output is as expected. + +``` sh +$ just-mr build test -Pwork/out.txt +INFO: Requested target is [["@","","","test"],{}] +INFO: Analysed target [["@","","","test"],{}] +INFO: Export targets found: 3 cached, 0 uncached, 0 not eligible for caching +INFO: Target tainted ["test"]. +INFO: Discovered 8 actions, 4 trees, 1 blobs +INFO: Building [["@","","","test"],{}]. +INFO: Processed 8 actions, 5 cache hits. +INFO: Artifacts built, logical paths are: + result [7ef22e9a431ad0272713b71fdc8794016c8ef12f:5:f] + stderr [e69de29bb2d1d6434b8b29ae775ad8c2e48c5391:0:f] + stdout [7fab9dd1ee66a1e76a3697a27524f905600afbd0:196:f] + time-start [7ac216a2a98b7739ae5304d96cdfa6f0b0ed87b6:11:f] + time-stop [7ac216a2a98b7739ae5304d96cdfa6f0b0ed87b6:11:f] + work/addressbook.data [baa6f28731ff6d93fbef9fcc5f7e8ae900da5ba5:41:f] + work/out.txt [7fb178dd66ecf24fdb786a0f96ae5969b55442da:101:f] + (1 runfiles omitted.) +Person ID: 12345 + Name: John Doe + E-mail address: jdoe@example.org + Updated: 2022-12-14T18:08:36Z +INFO: Target tainted ["test"]. +$ +``` diff --git a/doc/tutorial/proto.org b/doc/tutorial/proto.org deleted file mode 100644 index b4a02d48..00000000 --- a/doc/tutorial/proto.org +++ /dev/null @@ -1,475 +0,0 @@ -* Using protocol buffers - -The rules /justbuild/ uses for itself also support protocol -buffers. This tutorial shows how to use those rules and the targets -associated with them. It is not a tutorial on protocol buffers -itself; rather, it is assumed that the reader has some knowledge on -[[https://developers.google.com/protocol-buffers/][protocol buffers]]. - -** Setting up the repository configuration - -Before we begin, we first need to declare where the root of our workspace is -located by creating the empty file ~ROOT~: - -#+BEGIN_SRC sh -$ touch ROOT -#+END_SRC - -The ~protobuf~ repository conveniently contains an -[[https://github.com/protocolbuffers/protobuf/tree/v3.12.4/examples][example]], -so we can use this and just add our own target files. We create -file ~repos.template.json~ as follows. - -#+SRCNAME: repos.template.json -#+BEGIN_SRC js -{ "repositories": - { "": - { "repository": - { "type": "zip" - , "content": "7af7165b585e4aed714555a747b6822376176ef4" - , "fetch": "https://github.com/protocolbuffers/protobuf/archive/refs/tags/v3.12.4.zip" - , "subdir": "protobuf-3.12.4/examples" - } - , "target_root": "tutorial" - , "bindings": {"rules": "rules-cc"} - } - , "tutorial": {"repository": {"type": "file", "path": "."}} - } -} -#+END_SRC - -The missing entry ~"rules-cc"~ refers to our C/C++ build rules provided -[[https://github.com/just-buildsystem/rules-cc][online]]. These rules support -protobuf if the dependency ~"protoc"~ is provided. To import this rule -repository including the required transitive dependencies for protobuf, the -~bin/just-import-git~ script with option ~--as rules-cc~ can be used to -generate the actual ~repos.json~: - -#+BEGIN_SRC sh -$ just-import-git -C repos.template.json -b master --as rules-cc https://github.com/just-buildsystem/rules-cc > repos.json -#+END_SRC - -To build the example with ~just~, the only task is to write targets files. As -that contains a couple of new concepts, we will do this step by step. - -** The proto library - -First, we have to declare the proto library. In this case, it only -contains the file ~addressbook.proto~ and has no dependencies. To -declare the library, create a ~TARGETS~ file with the following -content: - -#+SRCNAME: TARGETS -#+BEGIN_SRC js -{ "address": - { "type": ["@", "rules", "proto", "library"] - , "name": ["addressbook"] - , "srcs": ["addressbook.proto"] - } -} -#+END_SRC - -In general, proto libraries could also depend on other proto libraries; -those would be added to the ~"deps"~ field. - -When building the library, there's very little to do. - -#+BEGIN_SRC sh -$ just-mr build address -INFO: Requested target is [["@","","","address"],{}] -INFO: Analysed target [["@","","","address"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 0 actions, 0 trees, 0 blobs -INFO: Building [["@","","","address"],{}]. -INFO: Processed 0 actions, 0 cache hits. -INFO: Artifacts built, logical paths are: -$ -#+END_SRC - -On the other hand, what did we expect? A proto library is an abstract -description of a protocol, so, as long as we don't specify for which -language we want to have bindings, there is nothing to generate. - -Nevertheless, a proto library target is not empty. In fact, it can't be empty, -as other targets can only access the values of a target and have no -insights into its definitions. We already relied on this design principle -implicitly, when we exploited target-level caching for our external dependencies -and did not even construct the dependency graph for that target. A proto -library simply provides the dependency structure of the ~.proto~ files. - -#+BEGIN_SRC sh -$ just-mr analyse --dump-nodes - address -INFO: Requested target is [["@","","","address"],{}] -INFO: Result of target [["@","","","address"],{}]: { - "artifacts": { - }, - "provides": { - "proto": [ - {"id":"2a483a2de7f25c1bc066e47245f55ec9a2d4a719","type":"NODE"} - ] - }, - "runfiles": { - } - } -INFO: Target nodes of target [["@","","","address"],{}]: -{ - "089f6cae7ca77bb786578d3e0138b6ff445c5c92": { - "result": { - "artifact_stage": { - "addressbook.proto": { - "data": { - "file_type": "f", - "id": "b4b33b4c658924f0321ab4e7a9dc9cf8da1acec3", - "size": 1234 - }, - "type": "KNOWN" - } - }, - "provides": { - }, - "runfiles": { - } - }, - "type": "VALUE_NODE" - }, - "2a483a2de7f25c1bc066e47245f55ec9a2d4a719": { - "node_type": "library", - "string_fields": { - "name": ["addressbook"], - "stage": [""] - }, - "target_fields": { - "deps": [], - "srcs": [{"id":"089f6cae7ca77bb786578d3e0138b6ff445c5c92","type":"NODE"}] - }, - "type": "ABSTRACT_NODE" - } -} -$ -#+END_SRC - -The target has one provider ~"proto"~, which is a node. Nodes are -an abstract representation of a target graph. More precisely, there -are two kind of nodes, and our example contains one of each. - -The simple kind of nodes are the value nodes; they represent a -target that has a fixed value, and hence are given by artifacts, -runfiles, and provided data. In our case, we have one value node, -the one for the ~.proto~ file. - -The other kind of nodes are the abstract nodes. They describe the -arguments for a target, but only have an abstract name (i.e., a -string) for the rule. Combining such an abstract target with a -binding for the abstract rule names gives a concrete "anonymous" -target that, in our case, will generate the library with the bindings -for the concrete language. In this example, the abstract name is -~"library"~. The alternative in our proto rules would have been -~"service library"~, for proto libraries that also contain ~rpc~ -definitions (which is used by [[https://grpc.io/][gRPC]]). - -** Using proto libraries - -Using proto libraries requires, as discussed, bindings for the -abstract names. Fortunately, our ~CC~ rules are aware of proto -libraries, so we can simply use them. Our target file hence -continues as follows. - -#+SRCNAME: TARGETS -#+BEGIN_SRC js -... -, "add_person": - { "type": ["@", "rules", "CC", "binary"] - , "name": ["add_person"] - , "srcs": ["add_person.cc"] - , "private-proto": ["address"] - } -, "list_people": - { "type": ["@", "rules", "CC", "binary"] - , "name": ["list_people"] - , "srcs": ["list_people.cc"] - , "private-proto": ["address"] - } -... -#+END_SRC - -The first time, we build a target that requires the proto compiler -(in that particular version, built in that particular way), it takes -a bit of time, as the proto compiler has to be built. But in follow-up -builds, also in different projects, the target-level cache is filled already. - -#+BEGIN_SRC sh -$ just-mr build add_person -... -$ just-mr build add_person -INFO: Requested target is [["@","","","add_person"],{}] -INFO: Analysed target [["@","","","add_person"],{}] -INFO: Export targets found: 3 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 5 actions, 2 trees, 0 blobs -INFO: Building [["@","","","add_person"],{}]. -INFO: Processed 5 actions, 5 cache hits. -INFO: Artifacts built, logical paths are: - add_person [bcbb3deabfe0d77e6d3ea35615336a2f59a1b0aa:2285928:x] -$ -#+END_SRC - -If we look at the actions associated with the binary, we find that those -are still the two actions we expect: a compile action and a link action. - -#+BEGIN_SRC sh -$ just-mr analyse add_person --dump-actions - -INFO: Requested target is [["@","","","add_person"],{}] -INFO: Result of target [["@","","","add_person"],{}]: { - "artifacts": { - "add_person": {"data":{"id":"fcf211e2291b2867375e915538ce04cb4dfae86d","path":"add_person"},"type":"ACTION"} - }, - "provides": { - }, - "runfiles": { - } - } -INFO: Actions for target [["@","","","add_person"],{}]: -[ - { - "command": ["c++","-I","work","-isystem","include","-c","work/add_person.cc","-o","work/add_person.o"], - "env": { - "PATH": "/bin:/usr/bin" - }, - "input": { - ... - }, - "output": ["work/add_person.o"] - }, - { - "command": ["c++","-o","add_person","add_person.o","libaddressbook.a","libprotobuf.a","libprotobuf_lite.a","libzlib.a"], - "env": { - "PATH": "/bin:/usr/bin" - }, - "input": { - ... - }, - "output": ["add_person"] - } -] -$ -#+END_SRC - -As discussed, the ~libaddressbook.a~ that is conveniently available -during the linking of the binary (as well as the ~addressbook.pb.h~ -available in the ~include~ tree for the compile action) are generated -by an anonymous target. Using that during the build we already -filled the target-level cache, we can have a look at all targets -still analysed. In the one anonymous target, we find again the -abstract node we discussed earlier. - -#+BEGIN_SRC sh -$ just-mr analyse add_person --dump-targets - -INFO: Requested target is [["@","","","add_person"],{}] -INFO: Result of target [["@","","","add_person"],{}]: { - "artifacts": { - "add_person": {"data":{"id":"fcf211e2291b2867375e915538ce04cb4dfae86d","path":"add_person"},"type":"ACTION"} - }, - "provides": { - }, - "runfiles": { - } - } -INFO: List of analysed targets: -{ - "#": { - "eda46ea21de25033ff7250e6a4cdc0b2c24be0c7": { - "2a483a2de7f25c1bc066e47245f55ec9a2d4a719": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}] - } - }, - "@": { - "": { - "": { - "add_person": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}], - "address": [{}] - } - }, - "rules-cc": { - "CC": { - "defaults": [{}] - } - }, - "rules-cc/just/protobuf": { - "": { - "C++ runtime": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}], - "protoc": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}], - "well_known_protos": [{}] - } - } - } -} -$ -#+END_SRC - -It should be noted, however, that this tight integration of proto -into our ~C++~ rules is just convenience of our code base. If we had -to cooperate with rules not aware of proto, we could have created -a separate rule delegating the library creation to the anonymous -target and then simply reflecting the values of that target. -In fact, we could simply use an empty library with a public ~proto~ -dependency for this purpose. - -#+SRCNAME: TARGETS -#+BEGIN_SRC js -... -, "address proto library": - {"type": ["@", "rules", "CC", "library"], "proto": ["address"]} -... -#+END_SRC - -#+BEGIN_SRC sh -$ just-mr analyse 'address proto library' -... -INFO: Requested target is [["@","","","address proto library"],{}] -INFO: Result of target [["@","","","address proto library"],{}]: { - "artifacts": { - }, - "provides": { - ... - "compile-deps": { - "addressbook.pb.h": {"data":{"id":"6d70cd10fabcbc7591cd82aae2f100cca39d3879","path":"work/addressbook.pb.h"},"type":"ACTION"}, - ... - }, - "link-args": [ - "libaddressbook.a", - ... - ], - "link-deps": { - "libaddressbook.a": {"data":{"id":"753073bd026b6470138c47e004469dd1d3df08d4","path":"libaddressbook.a"},"type":"ACTION"}, - ... - }, - ... - }, - "runfiles": { - } - } -$ -#+END_SRC - -** Adding a test - -Finally, let's add a test. As we use the ~protobuf~ repository as -workspace root, we add the test script ad hoc into a targets file, -using the ~"file_gen"~ rule. For debugging a potentially failing -test, we also keep the intermediate files the test generates. -Create a top-level ~TARGETS~ file with the following content: - -#+SRCNAME: TARGETS -#+BEGIN_SRC js -... -, "test.sh": - { "type": "file_gen" - , "name": "test.sh" - , "data": - { "type": "join" - , "separator": "\n" - , "$1": - [ "set -e" - , "(echo 12345; echo 'John Doe'; echo 'jdoe@example.org'; echo) | ./add_person addressbook.data" - , "./list_people addressbook.data > out.txt" - , "grep Doe out.txt" - ] - } - } -, "test": - { "type": ["@", "rules", "shell/test", "script"] - , "name": ["read-write-test"] - , "test": ["test.sh"] - , "deps": ["add_person", "list_people"] - , "keep": ["addressbook.data", "out.txt"] - } -... -#+END_SRC - -That example also shows why it is important that the generation -of the language bindings is delegated to an anonymous target: we -want to analyse only once how the ~C++~ bindings are generated. -Nevertheless, many targets can depend (directly or indirectly) on -the same proto library. And, indeed, analysing the test, we get -the expected additional targets and the one anonymous target is -reused by both binaries. - -#+BEGIN_SRC sh -$ just-mr analyse test --dump-targets - -INFO: Requested target is [["@","","","test"],{}] -INFO: Result of target [["@","","","test"],{}]: { - "artifacts": { - "result": {"data":{"id":"20967787c42a289f5598249e696f851dde50065c","path":"result"},"type":"ACTION"}, - "stderr": {"data":{"id":"20967787c42a289f5598249e696f851dde50065c","path":"stderr"},"type":"ACTION"}, - "stdout": {"data":{"id":"20967787c42a289f5598249e696f851dde50065c","path":"stdout"},"type":"ACTION"}, - "time-start": {"data":{"id":"20967787c42a289f5598249e696f851dde50065c","path":"time-start"},"type":"ACTION"}, - "time-stop": {"data":{"id":"20967787c42a289f5598249e696f851dde50065c","path":"time-stop"},"type":"ACTION"}, - "work/addressbook.data": {"data":{"id":"20967787c42a289f5598249e696f851dde50065c","path":"work/addressbook.data"},"type":"ACTION"}, - "work/out.txt": {"data":{"id":"20967787c42a289f5598249e696f851dde50065c","path":"work/out.txt"},"type":"ACTION"} - }, - "provides": { - }, - "runfiles": { - "read-write-test": {"data":{"id":"c9d7bfc5bc8448bfef25b4e73e4494560bf6c350"},"type":"TREE"} - } - } -INFO: List of analysed targets: -{ - "#": { - "eda46ea21de25033ff7250e6a4cdc0b2c24be0c7": { - "2a483a2de7f25c1bc066e47245f55ec9a2d4a719": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}] - } - }, - "@": { - "": { - "": { - "add_person": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}], - "address": [{}], - "list_people": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}], - "test": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"RUNS_PER_TEST":null,"TEST_ENV":null}], - "test.sh": [{}] - } - }, - "rules-cc": { - "CC": { - "defaults": [{}] - } - }, - "rules-cc/just/protobuf": { - "": { - "C++ runtime": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}], - "protoc": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}], - "well_known_protos": [{}] - } - } - } -} -INFO: Target tainted ["test"]. -$ -#+END_SRC - -Finally, the test passes and the output is as expected. - -#+BEGIN_SRC sh -$ just-mr build test -Pwork/out.txt -INFO: Requested target is [["@","","","test"],{}] -INFO: Analysed target [["@","","","test"],{}] -INFO: Export targets found: 3 cached, 0 uncached, 0 not eligible for caching -INFO: Target tainted ["test"]. -INFO: Discovered 8 actions, 4 trees, 1 blobs -INFO: Building [["@","","","test"],{}]. -INFO: Processed 8 actions, 5 cache hits. -INFO: Artifacts built, logical paths are: - result [7ef22e9a431ad0272713b71fdc8794016c8ef12f:5:f] - stderr [e69de29bb2d1d6434b8b29ae775ad8c2e48c5391:0:f] - stdout [7fab9dd1ee66a1e76a3697a27524f905600afbd0:196:f] - time-start [7ac216a2a98b7739ae5304d96cdfa6f0b0ed87b6:11:f] - time-stop [7ac216a2a98b7739ae5304d96cdfa6f0b0ed87b6:11:f] - work/addressbook.data [baa6f28731ff6d93fbef9fcc5f7e8ae900da5ba5:41:f] - work/out.txt [7fb178dd66ecf24fdb786a0f96ae5969b55442da:101:f] - (1 runfiles omitted.) -Person ID: 12345 - Name: John Doe - E-mail address: jdoe@example.org - Updated: 2022-12-14T18:08:36Z -INFO: Target tainted ["test"]. -$ -#+END_SRC diff --git a/doc/tutorial/rebuild.md b/doc/tutorial/rebuild.md new file mode 100644 index 00000000..3f1ddd88 --- /dev/null +++ b/doc/tutorial/rebuild.md @@ -0,0 +1,227 @@ +Ensuring reproducibility of the build +===================================== + +Software builds should be +[reproducible](https://reproducible-builds.org/). The `just` tool, +supports this goal in local builds by isolating individual actions, +setting permissions and file time stamps to canonical values, etc; most +remote execution systems take even further measures to ensure the +environment always looks the same to every action. Nevertheless, it is +always possible to break reproducibility by bad actions, both coming +from rules not carefully written, as well as from ad-hoc actions added +by the `generic` target. + +``` jsonc +... +, "version.h": + { "type": "generic" + , "cmds": + ["echo '#define VERSION \"0.0.0.'`date +%Y%m%d%H%M%S`'\"' > version.h"] + , "outs": ["version.h"] + } +... +``` + +Besides time stamps there are many other sources of nondeterminism, like +properties of the build machine (name, number of CPUs available, etc), +but also subtle ones like `readdir` order. Often, those non-reproducible +parts get buried deeply in a final artifact (like the version string +embedded in a binary contained in a compressed installation archive); +and, as long as the non-reproducible action stays in cache, it does not +even result in bad incrementality. Still, others won't be able to +reproduce the exact artifact. + +There are tools like [diffoscope](https://diffoscope.org/) to deeply +compare archives and other container formats. Nevertheless, it is +desirable to find the root causes, i.e., the first (in topological +order) actions that yield a different output. + +Rebuilding +---------- + +For the remainder of this section, we will consider the following +example project with the C++ source file `hello.cpp`: + +``` {.cpp srcname="hello.cpp"} +#include +#include "version.h" + +int main(int argc, const char* argv[]) { + if (argc > 1 && std::string{argv[1]} == "-v") { + std::cout << VERSION << std::endl; + } + return 0; +} +``` + +and the following `TARGETS` file: + +``` {.jsonc srcname="TARGETS"} +{ "": + { "type": "install" + , "files": + { "bin/hello": "hello" + , "share/hello/version.txt": "version.txt" + , "share/hello/OUT.txt": "OUT.txt" + } + } +, "hello": + { "type": ["@", "rules", "CC", "binary"] + , "name": ["hello"] + , "srcs": ["hello.cpp"] + , "private-hdrs": ["version.h"] + } +, "version.h": + { "type": "generic" + , "cmds": + ["echo '#define VERSION \"0.0.0.'`date +%Y%m%d%H%M%S`'\"' > version.h"] + , "outs": ["version.h"] + } +, "version.txt": + { "type": "generic" + , "outs": ["version.txt"] + , "cmds": ["./hello -v > version.txt"] + , "deps": ["hello"] + } +, "out.txt": + { "type": "generic" + , "outs": ["out.txt"] + , "cmds": ["./hello > out.txt"] + , "deps": ["hello"] + } +, "OUT.txt": + { "type": "generic" + , "outs": ["OUT.txt"] + , "cmds": ["tr a-z A-Z > OUT.txt < out.txt"] + , "deps": ["out.txt"] + } +} +``` + +To search for the root cause of non-reproducibility, `just` has a +subcommand `rebuild`. It builds the specified target again, requesting +that every action be executed again (but target-level cache is still +active); then the result of every action is compared to the one in the +action cache, if present with the same inputs. So, you typically would +first `build` and then `rebuild`. Note that a repeated `build` simply +takes the action result from cache. + +``` sh +$ just-mr build +INFO: Requested target is [["@","tutorial","",""],{}] +INFO: Analysed target [["@","tutorial","",""],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 6 actions, 1 trees, 0 blobs +INFO: Building [["@","tutorial","",""],{}]. +INFO: Processed 6 actions, 0 cache hits. +INFO: Artifacts built, logical paths are: + bin/hello [59f7af154b3b7beac4a6cab40499cb3b388220c4:16608:x] + share/hello/OUT.txt [428b97b82b6c59cad7488b24e6b618ebbcd819bc:13:f] + share/hello/version.txt [088ae5a8a57f62016392bdf124a9b8dfc0288763:39:f] +$ just-mr build +INFO: Requested target is [["@","tutorial","",""],{}] +INFO: Analysed target [["@","tutorial","",""],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 6 actions, 1 trees, 0 blobs +INFO: Building [["@","tutorial","",""],{}]. +INFO: Processed 6 actions, 6 cache hits. +INFO: Artifacts built, logical paths are: + bin/hello [59f7af154b3b7beac4a6cab40499cb3b388220c4:16608:x] + share/hello/OUT.txt [428b97b82b6c59cad7488b24e6b618ebbcd819bc:13:f] + share/hello/version.txt [088ae5a8a57f62016392bdf124a9b8dfc0288763:39:f] +$ just-mr rebuild +INFO: Requested target is [["@","tutorial","",""],{}] +INFO: Analysed target [["@","tutorial","",""],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 6 actions, 1 trees, 0 blobs +INFO: Rebuilding [["@","tutorial","",""],{}]. +WARN: Found flaky action: + - id: c854a382ea26628e1a5b8d4af00d6d0cef433436 + - cmd: ["sh","-c","echo '#define VERSION \"0.0.0.'`date +%Y%m%d%H%M%S`'\"' > version.h\n"] + - output 'version.h' differs: + - [6aac3477e22cd57e8c98ded78562d3c017e5d611:39:f] (rebuilt) + - [789a29f39b6aa966f91776bfe092e247614e6acd:39:f] (cached) +INFO: 2 actions compared with cache, 1 flaky actions found (0 of which tainted), no cache entry found for 4 actions. +INFO: Artifacts built, logical paths are: + bin/hello [73994ff43ec1161aba96708f277e8c88feab0386:16608:x] + share/hello/OUT.txt [428b97b82b6c59cad7488b24e6b618ebbcd819bc:13:f] + share/hello/version.txt [8dd65747395c0feab30891eab9e11d4a9dd0c715:39:f] +$ +``` + +In the example, the second action compared to cache is the upper casing +of the output. Even though the generation of `out.txt` depends on the +non-reproducible `hello`, the file itself is reproducible. Therefore, +the follow-up actions are checked as well. + +For this simple example, reading the console output is enough to +understand what's going on. However, checking for reproducibility +usually is part of a larger, quality-assurance process. To support the +automation of such processes, the findings can also be reported in +machine-readable form. + +``` sh +$ just-mr rebuild --dump-flaky flakes.json --dump-graph actions.json +[...] +$ cat flakes.json +{ + "cache misses": [ + "ca45feffd2bab3bdbf752db7c89c451ce99d4803", + "eff0117276a0ad65a331eb29a2393a9b6e106e4b", + "1eef99333e887f2aa78d2eaee0a7c172db66009c", + "72a337628fa6187513818d5d8d00d36fbdb6923e" + ], + "flaky actions": { + "c854a382ea26628e1a5b8d4af00d6d0cef433436": { + "version.h": { + "cached": { + "file_type": "f", + "id": "789a29f39b6aa966f91776bfe092e247614e6acd", + "size": 39 + }, + "rebuilt": { + "file_type": "f", + "id": "4b5e64f324745d628c893f9d7e29ce7febdfdd0c", + "size": 39 + } + } + } + } +}$ +``` + +The file reports the flaky actions together with the non-reproducible +artifacts they generated, reporting both, the cached and the newly +generated output. The files themselves can be obtained via `just +install-cas` as usual, allowing deeper comparison of the outputs. The +full definitions of the actions can be found in the action graph, in the +example dumped as well as `actions.json`; this definition also includes +the origins for each action, i.e., the configured targets that requested +the respective action. + +Comparing build environments +---------------------------- + +Simply rebuilding on the same machine is good way to detect embedded +time stamps of sufficiently small granularity; for other sources of +non-reproducibility, however, more modifications of the environment are +necessary. + +A simple, but effective, way for modifying the build environment is the +option `-L` to set the local launcher, a list of strings the argument +vector is prefixed with before the action is executed. The default +`["env", "--"]` simply resolves the program to be executed in the +current value of `PATH`, but a different value for the launcher can +obviously be used to set environment variables like `LD_PRELOAD`. +Relevant libraries and tools include +[libfaketime](https://github.com/wolfcw/libfaketime), +[fakehostname](https://github.com/dtcooper/fakehostname), and +[disorderfs](https://salsa.debian.org/reproducible-builds/disorderfs). + +More variation can be achieved by comparing remote execution builds, +either for two different remote-execution end points or comparing one +remote-execution end point to the local build. The latter is also a good +way to find out where a build that "works on my machine" differs. The +endpoint on which the rebuild is executed can be set, in the same way as +for build with the `-r` option; the cache end point to compare against +can be set via the `--vs` option. diff --git a/doc/tutorial/rebuild.org b/doc/tutorial/rebuild.org deleted file mode 100644 index 80aafb6f..00000000 --- a/doc/tutorial/rebuild.org +++ /dev/null @@ -1,225 +0,0 @@ -* Ensuring reproducibility of the build - -Software builds should be [[https://reproducible-builds.org/][reproducible]]. -The ~just~ tool, supports this goal in local builds by isolating -individual actions, setting permissions and file time stamps to -canonical values, etc; most remote execution systems take even further -measures to ensure the environment always looks the same to every -action. Nevertheless, it is always possible to break reproducibility -by bad actions, both coming from rules not carefully written, as -well as from ad-hoc actions added by the ~generic~ target. - -#+BEGIN_SRC js -... -, "version.h": - { "type": "generic" - , "cmds": - ["echo '#define VERSION \"0.0.0.'`date +%Y%m%d%H%M%S`'\"' > version.h"] - , "outs": ["version.h"] - } -... -#+END_SRC - -Besides time stamps there are many other sources of nondeterminism, -like properties of the build machine (name, number of CPUs available, -etc), but also subtle ones like ~readdir~ order. Often, those -non-reproducible parts get buried deeply in a final artifact (like -the version string embedded in a binary contained in a compressed -installation archive); and, as long as the non-reproducible action -stays in cache, it does not even result in bad incrementality. -Still, others won't be able to reproduce the exact artifact. - -There are tools like [[https://diffoscope.org/][diffoscope]] to deeply -compare archives and other container formats. Nevertheless, it is -desirable to find the root causes, i.e., the first (in topological -order) actions that yield a different output. - -** Rebuilding - -For the remainder of this section, we will consider the following example -project with the C++ source file ~hello.cpp~: - -#+SRCNAME: hello.cpp -#+BEGIN_SRC cpp -#include -#include "version.h" - -int main(int argc, const char* argv[]) { - if (argc > 1 && std::string{argv[1]} == "-v") { - std::cout << VERSION << std::endl; - } - return 0; -} -#+END_SRC - -and the following ~TARGETS~ file: - -#+SRCNAME: TARGETS -#+BEGIN_SRC js -{ "": - { "type": "install" - , "files": - { "bin/hello": "hello" - , "share/hello/version.txt": "version.txt" - , "share/hello/OUT.txt": "OUT.txt" - } - } -, "hello": - { "type": ["@", "rules", "CC", "binary"] - , "name": ["hello"] - , "srcs": ["hello.cpp"] - , "private-hdrs": ["version.h"] - } -, "version.h": - { "type": "generic" - , "cmds": - ["echo '#define VERSION \"0.0.0.'`date +%Y%m%d%H%M%S`'\"' > version.h"] - , "outs": ["version.h"] - } -, "version.txt": - { "type": "generic" - , "outs": ["version.txt"] - , "cmds": ["./hello -v > version.txt"] - , "deps": ["hello"] - } -, "out.txt": - { "type": "generic" - , "outs": ["out.txt"] - , "cmds": ["./hello > out.txt"] - , "deps": ["hello"] - } -, "OUT.txt": - { "type": "generic" - , "outs": ["OUT.txt"] - , "cmds": ["tr a-z A-Z > OUT.txt < out.txt"] - , "deps": ["out.txt"] - } -} -#+END_SRC - -To search for the root cause of non-reproducibility, ~just~ has -a subcommand ~rebuild~. It builds the specified target again, requesting -that every action be executed again (but target-level cache is still -active); then the result of every action is compared to the one in the -action cache, if present with the same inputs. So, you typically would -first ~build~ and then ~rebuild~. Note that a repeated ~build~ simply -takes the action result from cache. - -#+BEGIN_SRC sh -$ just-mr build -INFO: Requested target is [["@","tutorial","",""],{}] -INFO: Analysed target [["@","tutorial","",""],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 6 actions, 1 trees, 0 blobs -INFO: Building [["@","tutorial","",""],{}]. -INFO: Processed 6 actions, 0 cache hits. -INFO: Artifacts built, logical paths are: - bin/hello [59f7af154b3b7beac4a6cab40499cb3b388220c4:16608:x] - share/hello/OUT.txt [428b97b82b6c59cad7488b24e6b618ebbcd819bc:13:f] - share/hello/version.txt [088ae5a8a57f62016392bdf124a9b8dfc0288763:39:f] -$ just-mr build -INFO: Requested target is [["@","tutorial","",""],{}] -INFO: Analysed target [["@","tutorial","",""],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 6 actions, 1 trees, 0 blobs -INFO: Building [["@","tutorial","",""],{}]. -INFO: Processed 6 actions, 6 cache hits. -INFO: Artifacts built, logical paths are: - bin/hello [59f7af154b3b7beac4a6cab40499cb3b388220c4:16608:x] - share/hello/OUT.txt [428b97b82b6c59cad7488b24e6b618ebbcd819bc:13:f] - share/hello/version.txt [088ae5a8a57f62016392bdf124a9b8dfc0288763:39:f] -$ just-mr rebuild -INFO: Requested target is [["@","tutorial","",""],{}] -INFO: Analysed target [["@","tutorial","",""],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 6 actions, 1 trees, 0 blobs -INFO: Rebuilding [["@","tutorial","",""],{}]. -WARN: Found flaky action: - - id: c854a382ea26628e1a5b8d4af00d6d0cef433436 - - cmd: ["sh","-c","echo '#define VERSION \"0.0.0.'`date +%Y%m%d%H%M%S`'\"' > version.h\n"] - - output 'version.h' differs: - - [6aac3477e22cd57e8c98ded78562d3c017e5d611:39:f] (rebuilt) - - [789a29f39b6aa966f91776bfe092e247614e6acd:39:f] (cached) -INFO: 2 actions compared with cache, 1 flaky actions found (0 of which tainted), no cache entry found for 4 actions. -INFO: Artifacts built, logical paths are: - bin/hello [73994ff43ec1161aba96708f277e8c88feab0386:16608:x] - share/hello/OUT.txt [428b97b82b6c59cad7488b24e6b618ebbcd819bc:13:f] - share/hello/version.txt [8dd65747395c0feab30891eab9e11d4a9dd0c715:39:f] -$ -#+END_SRC - -In the example, the second action compared to cache is the upper -casing of the output. Even though the generation of ~out.txt~ depends -on the non-reproducible ~hello~, the file itself is reproducible. -Therefore, the follow-up actions are checked as well. - -For this simple example, reading the console output is enough to understand -what's going on. However, checking for reproducibility usually is part -of a larger, quality-assurance process. To support the automation of such -processes, the findings can also be reported in machine-readable form. - -#+BEGIN_SRC sh -$ just-mr rebuild --dump-flaky flakes.json --dump-graph actions.json -[...] -$ cat flakes.json -{ - "cache misses": [ - "ca45feffd2bab3bdbf752db7c89c451ce99d4803", - "eff0117276a0ad65a331eb29a2393a9b6e106e4b", - "1eef99333e887f2aa78d2eaee0a7c172db66009c", - "72a337628fa6187513818d5d8d00d36fbdb6923e" - ], - "flaky actions": { - "c854a382ea26628e1a5b8d4af00d6d0cef433436": { - "version.h": { - "cached": { - "file_type": "f", - "id": "789a29f39b6aa966f91776bfe092e247614e6acd", - "size": 39 - }, - "rebuilt": { - "file_type": "f", - "id": "4b5e64f324745d628c893f9d7e29ce7febdfdd0c", - "size": 39 - } - } - } - } -}$ -#+END_SRC - -The file reports the flaky actions together with the non-reproducible -artifacts they generated, reporting both, the cached and the newly -generated output. The files themselves can be obtained via ~just -install-cas~ as usual, allowing deeper comparison of the outputs. -The full definitions of the actions can be found in the action graph, -in the example dumped as well as ~actions.json~; this definition -also includes the origins for each action, i.e., the configured -targets that requested the respective action. - - -** Comparing build environments - -Simply rebuilding on the same machine is good way to detect embedded -time stamps of sufficiently small granularity; for other sources of -non-reproducibility, however, more modifications of the environment -are necessary. - -A simple, but effective, way for modifying the build environment -is the option ~-L~ to set the local launcher, a list of -strings the argument vector is prefixed with before the action is -executed. The default ~["env", "--"]~ simply resolves the program -to be executed in the current value of ~PATH~, but a different -value for the launcher can obviously be used to set environment -variables like ~LD_PRELOAD~. Relevant libraries and tools -include [[https://github.com/wolfcw/libfaketime][libfaketime]], -[[https://github.com/dtcooper/fakehostname][fakehostname]], -and [[https://salsa.debian.org/reproducible-builds/disorderfs][disorderfs]]. - -More variation can be achieved by comparing remote execution builds, -either for two different remote-execution end points or comparing -one remote-execution end point to the local build. The latter is -also a good way to find out where a build that "works on my machine" -differs. The endpoint on which the rebuild is executed can be set, -in the same way as for build with the ~-r~ option; the cache end -point to compare against can be set via the ~--vs~ option. diff --git a/doc/tutorial/target-file-glob-tree.md b/doc/tutorial/target-file-glob-tree.md new file mode 100644 index 00000000..524cf358 --- /dev/null +++ b/doc/tutorial/target-file-glob-tree.md @@ -0,0 +1,430 @@ +Target versus `FILE`, `GLOB`, and `TREE` +======================================== + +So far, we referred to defined targets as well as source files by their +name and it just worked. When considering third-party software we +already saw the `TREE` reference. In this section, we will highlight in +more detail the ways to refer to sources, as well as the difference +between defined and source targets. The latter is used, e.g., when +third-party software has to be patched. + +As example for this section we use gnu `units` where we want to patch +into the standard units definition add two units of area popular in +German news. + +Repository Config for `units` with patches +------------------------------------------ + +Before we begin, we first need to declare where the root of our +workspace is located by creating the empty file `ROOT`: + +``` sh +$ touch ROOT +``` + +The sources are an archive available on the web. As upstream uses a +different build system, we have to provide our own build description; we +take the top-level directory as layer for this. As we also want to patch +the definition file, we add the subdirectory `files` as logical +repository for the patches. Hence we create a file `repos.json` with the +following content. + +``` {.jsonc srcname="repos.json"} +{ "main": "units" +, "repositories": + { "rules-cc": + { "repository": + { "type": "git" + , "branch": "master" + , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" + , "repository": "https://github.com/just-buildsystem/rules-cc.git" + , "subdir": "rules" + } + } + , "import targets": {"repository": {"type": "file", "path": "."}} + , "patches": {"repository": {"type": "file", "path": "files"}} + , "units": + { "repository": + { "type": "archive" + , "content": "9781174d42bd593d3bab6c6decfdcae60e3ce328" + , "fetch": "https://ftp.gnu.org/gnu/units/units-2.21.tar.gz" + , "subdir": "units-2.21" + } + , "target_root": "import targets" + , "target_file_name": "TARGETS.units" + , "bindings": {"rules": "rules-cc", "patches": "patches"} + } + } +} +``` + +The repository to set up is `units` and, as usual, we can use `just-mr` +to fetch the archive and obtain the resulting multi-repository +configuration. + +``` sh +$ just-mr setup units +``` + +Patching a file: targets versus `FILE` +-------------------------------------- + +Let's start by patching the source file `definitions.units`. While, +conceptionally, we want to patch a third-party source file, we do *not* +modify the sources. The workspace root is a git tree and stay like this. +Instead, we remember that we specify *targets* and the definition of a +target is looked up in the targets file; only if not defined there, it +is implicitly considered a source target and taken from the target root. +So we will define a *target* named `definitions.units` to replace the +original source file. + +Let's first generate the patch. As we're already referring to source +files as targets, we have to provide a targets file already; we start +with the empty object and refine it later. + +``` sh +$ echo {} > TARGETS.units +$ just-mr install -o . definitions.units +INFO: Requested target is [["@","units","","definitions.units"],{}] +INFO: Analysed target [["@","units","","definitions.units"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 0 actions, 0 trees, 0 blobs +INFO: Building [["@","units","","definitions.units"],{}]. +INFO: Processed 0 actions, 0 cache hits. +INFO: Artifacts can be found in: + /tmp/work-2022-08-22/definitions.units [0f24a321694aab5c1d3676e22d01fc73492bee42:342718:f] +$ cp definitions.units definitions.units.orig +$ # interactively edit definitions.units +$ echo -e "/German units\n+2a\narea_soccerfield 105 m * 68 m\narea_saarland 2570 km^2\n.\nw\nq" | ed definitions.units +342718 +# A few German units as currently in use. +342772 +$ mkdir files +$ echo {} > files/TARGETS +$ diff -u definitions.units.orig definitions.units > files/definitions.units.diff +$ rm definitions.units* +``` + +Our rules conveniently contain a rule `["patch", "file"]` to patch a +single file, and we already created the patch. The only other input +missing is the source file. So far, we could refer to it as +`"definitions.units"` because there was no target of that name, but now +we're about to define a target with that very name. Fortunately, in +target files, we can use a special syntax to explicitly refer to a +source file of the current module, even if there is a target with the +same name: `["FILE", null, "definition.units"]`. The syntax requires the +explicit `null` value for the current module, despite the fact that +explicit file references are only allowed for the current module; in +this way, the name is a list of length more than two and cannot be +confused with a top-level module called `FILE`. So we add this target +and obtain as `TARGETS.units` the following. + +``` {.jsonc srcname="TARGETS.units"} +{ "definitions.units": + { "type": ["@", "rules", "patch", "file"] + , "src": [["FILE", ".", "definitions.units"]] + , "patch": [["@", "patches", "", "definitions.units.diff"]] + } +} +``` + +Analysing `"definitions.units"` we find our defined target which +contains an action output. Still, it looks like a patched source file; +the new artifact is staged to the original location. Staging is also +used in the action definition, to avoid magic names (like file names +starting with `-`), in-place operations (all actions must not modify +their inputs) and, in fact, have a fixed command line. + +``` sh +$ just-mr analyse definitions.units --dump-actions - +INFO: Requested target is [["@","units","","definitions.units"],{}] +INFO: Result of target [["@","units","","definitions.units"],{}]: { + "artifacts": { + "definitions.units": {"data":{"id":"98e3c7758f5dd433c6aa7b327040be676faf6f34","path":"patched"},"type":"ACTION"} + }, + "provides": { + }, + "runfiles": { + "definitions.units": {"data":{"id":"98e3c7758f5dd433c6aa7b327040be676faf6f34","path":"patched"},"type":"ACTION"} + } + } +INFO: Actions for target [["@","units","","definitions.units"],{}]: +[ + { + "command": ["patch","-s","--read-only=ignore","--follow-symlinks","-o","patched","orig","patch"], + "input": { + "orig": { + "data": { + "file_type": "f", + "id": "0f24a321694aab5c1d3676e22d01fc73492bee42", + "size": 342718 + }, + "type": "KNOWN" + }, + "patch": { + "data": { + "path": "definitions.units.diff", + "repository": "patches" + }, + "type": "LOCAL" + } + }, + "output": ["patched"] + } +] +$ +``` + +Building `"definitions.units"` we find out patch applied correctly. + +``` sh +$ just-mr build definitions.units -P definitions.units | grep -A 5 'German units' +INFO: Requested target is [["@","units","","definitions.units"],{}] +INFO: Analysed target [["@","units","","definitions.units"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 1 actions, 0 trees, 1 blobs +INFO: Building [["@","units","","definitions.units"],{}]. +INFO: Processed 1 actions, 0 cache hits. +INFO: Artifacts built, logical paths are: + definitions.units [763f3289422c296057e142f61be190ee6bef049a:342772:f] +# A few German units as currently in use. +# + +area_soccerfield 105 m * 68 m +area_saarland 2570 km^2 +zentner 50 kg +$ +``` + +Globbing source files: `"GLOB"` +------------------------------- + +Next, we collect all `.units` files. We could simply do this by +enumerating them in a target. + +``` {.jsonc srcname="TARGETS.units"} +... +, "data-draft": { "type": "install", "deps": ["definitions.units", "currency.units"]} +... +``` + +In this way, we get the desired collection of one unmodified source file +and the output of the patch action. + +``` sh +$ just-mr analyse data-draft +INFO: Requested target is [["@","units","","data-draft"],{}] +INFO: Result of target [["@","units","","data-draft"],{}]: { + "artifacts": { + "currency.units": {"data":{"file_type":"f","id":"ac6da8afaac0f34e114e123e4ab3a41e59121b10","size":14707},"type":"KNOWN"}, + "definitions.units": {"data":{"id":"98e3c7758f5dd433c6aa7b327040be676faf6f34","path":"patched"},"type":"ACTION"} + }, + "provides": { + }, + "runfiles": { + "currency.units": {"data":{"file_type":"f","id":"ac6da8afaac0f34e114e123e4ab3a41e59121b10","size":14707},"type":"KNOWN"}, + "definitions.units": {"data":{"id":"98e3c7758f5dd433c6aa7b327040be676faf6f34","path":"patched"},"type":"ACTION"} + } + } +$ +``` + +The disadvantage, however, that we might miss newly added `.units` files +if we update and upstream added new files. So we want all source files +that have the respective ending. The corresponding source reference is +`"GLOB"`. A glob expands to the *collection* of all *sources* that are +*files* in the *top-level* directory of the current module and that +match the given pattern. It is important to understand this in detail +and the rational behind it. + + - First of all, the artifact (and runfiles) map has an entry for each + file that matches. In particular, targets have the option to define + individual actions for each file, like `["CC", "binary"]` does for + the source files. This is different from `"TREE"` where the artifact + map contains a single artifact that happens to be a directory. The + tree behaviour is preferable when the internals of the directory + only matter for the execution of actions and not for analysis; then + there are less entries to carry around during analysis and + action-key computation, and the whole directory is "reserved" for + that tree avoid staging conflicts when latter adding entries there. + - As a source reference, a glob expands to explicit source files; + targets having the same name as a source file are not taken into + account. In our example, `["GLOB", null, "*.units"]` therefore + contains the unpatched source file `definitions.units`. In this way, + we avoid any surprises in the expansion of a glob when a new source + file is added with a name equal to an already existing target. + - Only files are considered for matching the glob. Directories are + ignored. + - Matches are only considered at the top-level directory. In this way, + only one directory has to be read during analysis; allowing deeper + globs would require traversal of subdirectories requiring larger + cost. While the explicit `"TREE"` reference allows recursive + traversal, in the typical use case of the respective workspace root + being a `git` root, it is actually cheap; we can look up the `git` + tree identifier without traversing the tree. Such a quick look up + would not be possible if matches had to be selected. + +So, `["GLOB", null, "*.units"]` expands to all the relevant source +files; but we still want to keep the patching. Most rules, like +`"install"`, disallow staging conflicts to avoid accidentally ignoring a +file due to conflicting name. In our case, however, the dropping of the +source file in favour of the patched one is deliberate. For this, there +is the rule `["data", "overlay"]` taking the union of the artifacts of +the specified targets, accepting conflicts and resolving them in a +latest-wins fashion. Keep in mind, that our target fields are list, not +sets. Looking at the definition of the rule, one finds that it is simply +a `"map_union"`. Hence we refine our `"data"` target. + +``` {.jsonc srcname="TARGETS.units"} +... +, "data": + { "type": ["@", "rules", "data", "overlay"] + , "deps": [["GLOB", null, "*.units"], "definitions.units"] + } +... +``` + +The result of the analysis, of course, still is the same. + +Finishing the example: binaries from globbed sources +---------------------------------------------------- + +The source-code organisation of units is pretty simple. All source and +header files are in the top-level directory. As the header files are not +in a directory of their own, we can't use a tree, so we use a glob, +which is fine for the private headers of a binary. For the source files, +we have to have them individually anyway. So our first attempt of +defining the binary is as follows. + +``` {.jsonc srcname="TARGETS.units"} +... +, "units-draft": + { "type": ["@", "rules", "CC", "binary"] + , "name": ["units"] + , "private-ldflags": ["-lm"] + , "pure C": ["YES"] + , "srcs": [["GLOB", null, "*.c"]] + , "private-hdrs": [["GLOB", null, "*.h"]] + } +... +``` + +The result basically work and shows that we have 5 source files in +total, giving 5 compile and one link action. + +``` sh +$ just-mr build units-draft +INFO: Requested target is [["@","units","","units-draft"],{}] +INFO: Analysed target [["@","units","","units-draft"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 6 actions, 1 trees, 0 blobs +INFO: Building [["@","units","","units-draft"],{}]. +INFO (action:12af248ce5737be492f7f5909284d4e3b6488807): + Stderr of command: ["cc","-I","work","-isystem","include","-c","work/strfunc.c","-o","work/strfunc.o"] + work/strfunc.c:109:8: warning: extra tokens at end of #endif directive [-Wendif-labels] + 109 | #endif NO_STRSPN + | ^~~~~~~~~ +INFO: Processed 6 actions, 0 cache hits. +INFO: Artifacts built, logical paths are: + units [718cb1489bd006082f966ea73e3fba3dd072d084:124488:x] +$ +``` + +To keep the build clean, we want to get rid of the warning. Of course, +we could simply set an appropriate compiler flag, but let's do things +properly and patch away the underlying reason. To do so, we first create +a patch. + +``` sh +$ just-mr install -o . strfunc.c +INFO: Requested target is [["@","units","","strfunc.c"],{}] +INFO: Analysed target [["@","units","","strfunc.c"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 0 actions, 0 trees, 0 blobs +INFO: Building [["@","units","","strfunc.c"],{}]. +INFO: Processed 0 actions, 0 cache hits. +INFO: Artifacts can be found in: + /tmp/work-2022-08-22/strfunc.c [e2aab4b825fa2822ccf33746d467a4944212abb9:2201:f] +$ cp strfunc.c strfunc.c.orig +$ echo -e "109\ns|N|// N\nw\nq" | ed strfunc.c +2201 +#endif NO_STRSPN +#endif // NO_STRSPN +2204 +$ diff strfunc.c.orig strfunc.c > files/strfunc.c.diff +$ rm strfunc.c* +$ +``` + +Then we amend our `"units"` target. + +``` {.jsonc srcname="TARGETS.units"} +... +, "units": + { "type": ["@", "rules", "CC", "binary"] + , "name": ["units"] + , "private-ldflags": ["-lm"] + , "pure C": ["YES"] + , "srcs": ["patched srcs"] + , "private-hdrs": [["GLOB", null, "*.h"]] + } +, "patched srcs": + { "type": ["@", "rules", "data", "overlay"] + , "deps": [["GLOB", null, "*.c"], "strfunc.c"] + } +, "strfunc.c": + { "type": ["@", "rules", "patch", "file"] + , "src": [["FILE", ".", "strfunc.c"]] + , "patch": [["@", "patches", "", "strfunc.c.diff"]] + } +... +``` + +Building the new target, 2 actions have to be executed: the patching, +and the compiling of the patched source file. As the patched file still +generates the same object file as the unpatched file (after all, we only +wanted to get rid of a warning), the linking step can be taken from +cache. + +``` sh +$ just-mr build units +INFO: Requested target is [["@","units","","units"],{}] +INFO: Analysed target [["@","units","","units"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 7 actions, 1 trees, 1 blobs +INFO: Building [["@","units","","units"],{}]. +INFO: Processed 7 actions, 5 cache hits. +INFO: Artifacts built, logical paths are: + units [718cb1489bd006082f966ea73e3fba3dd072d084:124488:x] +$ +``` + +To finish the example, we also add a default target (using that, if no +target is specified, `just` builds the lexicographically first target), +staging artifacts according to the usual conventions. + +``` {.jsonc srcname="TARGETS.units"} +... +, "": {"type": "install", "dirs": [["units", "bin"], ["data", "share/units"]]} +... +``` + +Then things work as expected + +``` sh +$ just-mr install -o /tmp/testinstall +INFO: Requested target is [["@","units","",""],{}] +INFO: Analysed target [["@","units","",""],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 8 actions, 1 trees, 1 blobs +INFO: Building [["@","units","",""],{}]. +INFO: Processed 8 actions, 8 cache hits. +INFO: Artifacts can be found in: + /tmp/testinstall/bin/units [718cb1489bd006082f966ea73e3fba3dd072d084:124488:x] + /tmp/testinstall/share/units/currency.units [ac6da8afaac0f34e114e123e4ab3a41e59121b10:14707:f] + /tmp/testinstall/share/units/definitions.units [763f3289422c296057e142f61be190ee6bef049a:342772:f] +$ /tmp/testinstall/bin/units 'area_saarland' 'area_soccerfield' + * 359943.98 + / 2.7782101e-06 +$ +``` diff --git a/doc/tutorial/target-file-glob-tree.org b/doc/tutorial/target-file-glob-tree.org deleted file mode 100644 index 58e9c725..00000000 --- a/doc/tutorial/target-file-glob-tree.org +++ /dev/null @@ -1,430 +0,0 @@ -* Target versus ~FILE~, ~GLOB~, and ~TREE~ - -So far, we referred to defined targets as well as source files -by their name and it just worked. When considering third-party -software we already saw the ~TREE~ reference. In this section, we -will highlight in more detail the ways to refer to sources, as well -as the difference between defined and source targets. The latter -is used, e.g., when third-party software has to be patched. - -As example for this section we use gnu ~units~ where we want to -patch into the standard units definition add two units of area -popular in German news. - -** Repository Config for ~units~ with patches - -Before we begin, we first need to declare where the root of our workspace is -located by creating the empty file ~ROOT~: - -#+BEGIN_SRC sh -$ touch ROOT -#+END_SRC - -The sources are an archive available on the web. As upstream uses a -different build system, we have to provide our own build description; -we take the top-level directory as layer for this. As we also want -to patch the definition file, we add the subdirectory ~files~ as -logical repository for the patches. Hence we create a file ~repos.json~ -with the following content. - -#+SRCNAME: repos.json -#+BEGIN_SRC js -{ "main": "units" -, "repositories": - { "rules-cc": - { "repository": - { "type": "git" - , "branch": "master" - , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" - , "repository": "https://github.com/just-buildsystem/rules-cc.git" - , "subdir": "rules" - } - } - , "import targets": {"repository": {"type": "file", "path": "."}} - , "patches": {"repository": {"type": "file", "path": "files"}} - , "units": - { "repository": - { "type": "archive" - , "content": "9781174d42bd593d3bab6c6decfdcae60e3ce328" - , "fetch": "https://ftp.gnu.org/gnu/units/units-2.21.tar.gz" - , "subdir": "units-2.21" - } - , "target_root": "import targets" - , "target_file_name": "TARGETS.units" - , "bindings": {"rules": "rules-cc", "patches": "patches"} - } - } -} -#+END_SRC - -The repository to set up is ~units~ and, as usual, we can use ~just-mr~ to -fetch the archive and obtain the resulting multi-repository configuration. - -#+BEGIN_SRC sh -$ just-mr setup units -#+END_SRC - -** Patching a file: targets versus ~FILE~ - -Let's start by patching the source file ~definitions.units~. While, -conceptionally, we want to patch a third-party source file, we do /not/ -modify the sources. The workspace root is a git tree and stay like this. -Instead, we remember that we specify /targets/ and the definition of a -target is looked up in the targets file; only if not defined there, it -is implicitly considered a source target and taken from the target root. -So we will define a /target/ named ~definitions.units~ to replace the -original source file. - -Let's first generate the patch. As we're already referring to source files -as targets, we have to provide a targets file already; we start with the -empty object and refine it later. - -#+BEGIN_SRC sh -$ echo {} > TARGETS.units -$ just-mr install -o . definitions.units -INFO: Requested target is [["@","units","","definitions.units"],{}] -INFO: Analysed target [["@","units","","definitions.units"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 0 actions, 0 trees, 0 blobs -INFO: Building [["@","units","","definitions.units"],{}]. -INFO: Processed 0 actions, 0 cache hits. -INFO: Artifacts can be found in: - /tmp/work-2022-08-22/definitions.units [0f24a321694aab5c1d3676e22d01fc73492bee42:342718:f] -$ cp definitions.units definitions.units.orig -$ # interactively edit definitions.units -$ echo -e "/German units\n+2a\narea_soccerfield 105 m * 68 m\narea_saarland 2570 km^2\n.\nw\nq" | ed definitions.units -342718 -# A few German units as currently in use. -342772 -$ mkdir files -$ echo {} > files/TARGETS -$ diff -u definitions.units.orig definitions.units > files/definitions.units.diff -$ rm definitions.units* -#+END_SRC - -Our rules conveniently contain a rule ~["patch", "file"]~ to patch -a single file, and we already created the patch. The only other -input missing is the source file. So far, we could refer to it as -~"definitions.units"~ because there was no target of that name, but -now we're about to define a target with that very name. Fortunately, -in target files, we can use a special syntax to explicitly refer to -a source file of the current module, even if there is a target with -the same name: ~["FILE", null, "definition.units"]~. The syntax -requires the explicit ~null~ value for the current module, despite -the fact that explicit file references are only allowed for the -current module; in this way, the name is a list of length more than -two and cannot be confused with a top-level module called ~FILE~. -So we add this target and obtain as ~TARGETS.units~ the following. - -#+SRCNAME: TARGETS.units -#+BEGIN_SRC js -{ "definitions.units": - { "type": ["@", "rules", "patch", "file"] - , "src": [["FILE", ".", "definitions.units"]] - , "patch": [["@", "patches", "", "definitions.units.diff"]] - } -} -#+END_SRC - -Analysing ~"definitions.units"~ we find our defined target which -contains an action output. Still, it looks like a patched source -file; the new artifact is staged to the original location. Staging -is also used in the action definition, to avoid magic names (like -file names starting with ~-~), in-place operations (all actions -must not modify their inputs) and, in fact, have a -fixed command line. - -#+BEGIN_SRC sh -$ just-mr analyse definitions.units --dump-actions - -INFO: Requested target is [["@","units","","definitions.units"],{}] -INFO: Result of target [["@","units","","definitions.units"],{}]: { - "artifacts": { - "definitions.units": {"data":{"id":"98e3c7758f5dd433c6aa7b327040be676faf6f34","path":"patched"},"type":"ACTION"} - }, - "provides": { - }, - "runfiles": { - "definitions.units": {"data":{"id":"98e3c7758f5dd433c6aa7b327040be676faf6f34","path":"patched"},"type":"ACTION"} - } - } -INFO: Actions for target [["@","units","","definitions.units"],{}]: -[ - { - "command": ["patch","-s","--read-only=ignore","--follow-symlinks","-o","patched","orig","patch"], - "input": { - "orig": { - "data": { - "file_type": "f", - "id": "0f24a321694aab5c1d3676e22d01fc73492bee42", - "size": 342718 - }, - "type": "KNOWN" - }, - "patch": { - "data": { - "path": "definitions.units.diff", - "repository": "patches" - }, - "type": "LOCAL" - } - }, - "output": ["patched"] - } -] -$ -#+END_SRC - -Building ~"definitions.units"~ we find out patch applied correctly. - -#+BEGIN_SRC sh -$ just-mr build definitions.units -P definitions.units | grep -A 5 'German units' -INFO: Requested target is [["@","units","","definitions.units"],{}] -INFO: Analysed target [["@","units","","definitions.units"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 1 actions, 0 trees, 1 blobs -INFO: Building [["@","units","","definitions.units"],{}]. -INFO: Processed 1 actions, 0 cache hits. -INFO: Artifacts built, logical paths are: - definitions.units [763f3289422c296057e142f61be190ee6bef049a:342772:f] -# A few German units as currently in use. -# - -area_soccerfield 105 m * 68 m -area_saarland 2570 km^2 -zentner 50 kg -$ -#+END_SRC - -** Globbing source files: ~"GLOB"~ - -Next, we collect all ~.units~ files. We could simply do this by enumerating -them in a target. - -#+SRCNAME: TARGETS.units -#+BEGIN_SRC js -... -, "data-draft": { "type": "install", "deps": ["definitions.units", "currency.units"]} -... -#+END_SRC - -In this way, we get the desired collection of one unmodified source file and -the output of the patch action. - -#+BEGIN_SRC sh -$ just-mr analyse data-draft -INFO: Requested target is [["@","units","","data-draft"],{}] -INFO: Result of target [["@","units","","data-draft"],{}]: { - "artifacts": { - "currency.units": {"data":{"file_type":"f","id":"ac6da8afaac0f34e114e123e4ab3a41e59121b10","size":14707},"type":"KNOWN"}, - "definitions.units": {"data":{"id":"98e3c7758f5dd433c6aa7b327040be676faf6f34","path":"patched"},"type":"ACTION"} - }, - "provides": { - }, - "runfiles": { - "currency.units": {"data":{"file_type":"f","id":"ac6da8afaac0f34e114e123e4ab3a41e59121b10","size":14707},"type":"KNOWN"}, - "definitions.units": {"data":{"id":"98e3c7758f5dd433c6aa7b327040be676faf6f34","path":"patched"},"type":"ACTION"} - } - } -$ -#+END_SRC - -The disadvantage, however, that we might miss newly added ~.units~ -files if we update and upstream added new files. So we want all -source files that have the respective ending. The corresponding -source reference is ~"GLOB"~. A glob expands to the /collection/ -of all /sources/ that are /files/ in the /top-level/ directory of -the current module and that match the given pattern. It is important -to understand this in detail and the rational behind it. -- First of all, the artifact (and runfiles) map has an entry for - each file that matches. In particular, targets have the option to - define individual actions for each file, like ~["CC", "binary"]~ - does for the source files. This is different from ~"TREE"~ where - the artifact map contains a single artifact that happens to be a - directory. The tree behaviour is preferable when the internals - of the directory only matter for the execution of actions and not - for analysis; then there are less entries to carry around during - analysis and action-key computation, and the whole directory - is "reserved" for that tree avoid staging conflicts when latter - adding entries there. -- As a source reference, a glob expands to explicit source files; - targets having the same name as a source file are not taken into - account. In our example, ~["GLOB", null, "*.units"]~ therefore - contains the unpatched source file ~definitions.units~. In this - way, we avoid any surprises in the expansion of a glob when a new - source file is added with a name equal to an already existing target. -- Only files are considered for matching the glob. Directories - are ignored. -- Matches are only considered at the top-level directory. In this - way, only one directory has to be read during analysis; allowing - deeper globs would require traversal of subdirectories requiring - larger cost. While the explicit ~"TREE"~ reference allows recursive - traversal, in the typical use case of the respective workspace root - being a ~git~ root, it is actually cheap; we can look up the - ~git~ tree identifier without traversing the tree. Such a quick - look up would not be possible if matches had to be selected. - -So, ~["GLOB", null, "*.units"]~ expands to all the relevant source -files; but we still want to keep the patching. Most rules, like ~"install"~, -disallow staging conflicts to avoid accidentally ignoring a file due -to conflicting name. In our case, however, the dropping of the source -file in favour of the patched one is deliberate. For this, there is -the rule ~["data", "overlay"]~ taking the union of the artifacts of -the specified targets, accepting conflicts and resolving them in a -latest-wins fashion. Keep in mind, that our target fields are list, -not sets. Looking at the definition of the rule, one finds that -it is simply a ~"map_union"~. Hence we refine our ~"data"~ target. - -#+SRCNAME: TARGETS.units -#+BEGIN_SRC js -... -, "data": - { "type": ["@", "rules", "data", "overlay"] - , "deps": [["GLOB", null, "*.units"], "definitions.units"] - } -... -#+END_SRC - -The result of the analysis, of course, still is the same. - -** Finishing the example: binaries from globbed sources - -The source-code organisation of units is pretty simple. All source -and header files are in the top-level directory. As the header files -are not in a directory of their own, we can't use a tree, so we use -a glob, which is fine for the private headers of a binary. For the -source files, we have to have them individually anyway. So our first -attempt of defining the binary is as follows. - -#+SRCNAME: TARGETS.units -#+BEGIN_SRC js -... -, "units-draft": - { "type": ["@", "rules", "CC", "binary"] - , "name": ["units"] - , "private-ldflags": ["-lm"] - , "pure C": ["YES"] - , "srcs": [["GLOB", null, "*.c"]] - , "private-hdrs": [["GLOB", null, "*.h"]] - } -... -#+END_SRC - -The result basically work and shows that we have 5 source files in total, -giving 5 compile and one link action. - -#+BEGIN_SRC sh -$ just-mr build units-draft -INFO: Requested target is [["@","units","","units-draft"],{}] -INFO: Analysed target [["@","units","","units-draft"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 6 actions, 1 trees, 0 blobs -INFO: Building [["@","units","","units-draft"],{}]. -INFO (action:12af248ce5737be492f7f5909284d4e3b6488807): - Stderr of command: ["cc","-I","work","-isystem","include","-c","work/strfunc.c","-o","work/strfunc.o"] - work/strfunc.c:109:8: warning: extra tokens at end of #endif directive [-Wendif-labels] - 109 | #endif NO_STRSPN - | ^~~~~~~~~ -INFO: Processed 6 actions, 0 cache hits. -INFO: Artifacts built, logical paths are: - units [718cb1489bd006082f966ea73e3fba3dd072d084:124488:x] -$ -#+END_SRC - -To keep the build clean, we want to get rid of the warning. Of course, we could -simply set an appropriate compiler flag, but let's do things properly and patch -away the underlying reason. To do so, we first create a patch. - -#+BEGIN_SRC sh -$ just-mr install -o . strfunc.c -INFO: Requested target is [["@","units","","strfunc.c"],{}] -INFO: Analysed target [["@","units","","strfunc.c"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 0 actions, 0 trees, 0 blobs -INFO: Building [["@","units","","strfunc.c"],{}]. -INFO: Processed 0 actions, 0 cache hits. -INFO: Artifacts can be found in: - /tmp/work-2022-08-22/strfunc.c [e2aab4b825fa2822ccf33746d467a4944212abb9:2201:f] -$ cp strfunc.c strfunc.c.orig -$ echo -e "109\ns|N|// N\nw\nq" | ed strfunc.c -2201 -#endif NO_STRSPN -#endif // NO_STRSPN -2204 -$ diff strfunc.c.orig strfunc.c > files/strfunc.c.diff -$ rm strfunc.c* -$ -#+END_SRC - -Then we amend our ~"units"~ target. - -#+SRCNAME: TARGETS.units -#+BEGIN_SRC js -... -, "units": - { "type": ["@", "rules", "CC", "binary"] - , "name": ["units"] - , "private-ldflags": ["-lm"] - , "pure C": ["YES"] - , "srcs": ["patched srcs"] - , "private-hdrs": [["GLOB", null, "*.h"]] - } -, "patched srcs": - { "type": ["@", "rules", "data", "overlay"] - , "deps": [["GLOB", null, "*.c"], "strfunc.c"] - } -, "strfunc.c": - { "type": ["@", "rules", "patch", "file"] - , "src": [["FILE", ".", "strfunc.c"]] - , "patch": [["@", "patches", "", "strfunc.c.diff"]] - } -... -#+END_SRC - -Building the new target, 2 actions have to be executed: the patching, and -the compiling of the patched source file. As the patched file still generates -the same object file as the unpatched file (after all, we only wanted to get -rid of a warning), the linking step can be taken from cache. - -#+BEGIN_SRC sh -$ just-mr build units -INFO: Requested target is [["@","units","","units"],{}] -INFO: Analysed target [["@","units","","units"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 7 actions, 1 trees, 1 blobs -INFO: Building [["@","units","","units"],{}]. -INFO: Processed 7 actions, 5 cache hits. -INFO: Artifacts built, logical paths are: - units [718cb1489bd006082f966ea73e3fba3dd072d084:124488:x] -$ -#+END_SRC - -To finish the example, we also add a default target (using that, if no target -is specified, ~just~ builds the lexicographically first target), staging -artifacts according to the usual conventions. - -#+SRCNAME: TARGETS.units -#+BEGIN_SRC js -... -, "": {"type": "install", "dirs": [["units", "bin"], ["data", "share/units"]]} -... -#+END_SRC - -Then things work as expected - -#+BEGIN_SRC sh -$ just-mr install -o /tmp/testinstall -INFO: Requested target is [["@","units","",""],{}] -INFO: Analysed target [["@","units","",""],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 8 actions, 1 trees, 1 blobs -INFO: Building [["@","units","",""],{}]. -INFO: Processed 8 actions, 8 cache hits. -INFO: Artifacts can be found in: - /tmp/testinstall/bin/units [718cb1489bd006082f966ea73e3fba3dd072d084:124488:x] - /tmp/testinstall/share/units/currency.units [ac6da8afaac0f34e114e123e4ab3a41e59121b10:14707:f] - /tmp/testinstall/share/units/definitions.units [763f3289422c296057e142f61be190ee6bef049a:342772:f] -$ /tmp/testinstall/bin/units 'area_saarland' 'area_soccerfield' - * 359943.98 - / 2.7782101e-06 -$ -#+END_SRC diff --git a/doc/tutorial/tests.md b/doc/tutorial/tests.md new file mode 100644 index 00000000..138769b1 --- /dev/null +++ b/doc/tutorial/tests.md @@ -0,0 +1,339 @@ +Creating Tests +============== + +To run tests with justbuild, we do *not* have a dedicated `test` +subcommand. Instead, we consider tests being a specific action that +generates a test report. Consequently, we use the `build` subcommand to +build the test report, and thereby run the test action. Test actions, +however, are slightly different from normal actions in that we don't +want the build of the test report to be aborted if a test action fails +(but still, we want only successfully actions taken from cache). Rules +defining targets containing such special actions have to identify +themselves as *tainted* by specifying a string explaining why such +special actions are justified; in our case, the string is `"test"`. +Besides the implicit marking by using a tainted rule, those tainting +strings can also be explicitly assigned by the user in the definition of +a target, e.g., to mark test data. Any target has to be tainted with (at +least) all the strings any of its dependencies is tainted with. In this +way, it is ensured that no test target will end up in a production +build. + +For the remainder of this section, we expect to have the project files +available resulting from successfully completing the tutorial section on +*Building C++ Hello World*. We will demonstrate how to write a test +binary for the `greet` library and a shell test for the `helloworld` +binary. + +Creating a C++ test binary +-------------------------- + +First, we will create a C++ test binary for testing the correct +functionality of the `greet` library. Therefore, we need to provide a +C++ source file that performs the actual testing and returns non-`0` on +failure. For simplicity reasons, we do not use a testing framework for +this tutorial. A simple test that captures standard output and verifies +it with the expected output should be provided in the file +`tests/greet.test.cpp`: + +``` {.cpp srcname="tests/greet.test.cpp"} +#include +#include +#include +#include +#include "greet/greet.hpp" + +template +auto capture_stdout(std::function const& func) -> std::string { + int fd[2]; + if (pipe(fd) < 0) return {}; + int fd_stdout = dup(fileno(stdout)); + fflush(stdout); + dup2(fd[1], fileno(stdout)); + + func(); + fflush(stdout); + + std::string buf(kMaxBufSize, '\0'); + auto n = read(fd[0], &(*buf.begin()), kMaxBufSize); + close(fd[0]); + close(fd[1]); + dup2(fd_stdout, fileno(stdout)); + return buf.substr(0, n); +} + +auto test_greet(std::string const& name) -> bool { + auto expect = std::string{"Hello "} + name + "!\n"; + auto result = capture_stdout([&name] { greet(name); }); + std::cout << "greet output: " << result << std::endl; + return result == expect; +} + +int main() { + return test_greet("World") && test_greet("Universe") ? 0 : 1; +} +``` + +Next, a new test target needs to be created in module `greet`. This +target uses the rule `["@", "rules", "CC/test", "test"]` and needs to +depend on the `["greet", "greet"]` target. To create the test target, +add the following to `tests/TARGETS`: + +``` {.jsonc srcname="tests/TARGETS"} +{ "greet": + { "type": ["@", "rules", "CC/test", "test"] + , "name": ["test_greet"] + , "srcs": ["greet.test.cpp"] + , "private-deps": [["greet", "greet"]] + } +} +``` + +Before we can run the test, a proper default module for `CC/test` must +be provided. By specifying the appropriate target in this module the +default test runner can be overwritten by a different test runner fom +the rule's workspace root. Moreover, all test targets share runner +infrastructure from `shell/test`, e.g., summarizing multiple runs per +test (to detect flakyness) if the configuration variable `RUNS_PER_TEST` +is set. + +However, in our case, we want to use the default runner and therefore it +is sufficient to create an empty module. To do so, create the file +`tutorial-defaults/CC/test/TARGETS` with content + +``` {.jsonc srcname="tutorial-defaults/CC/test/TARGETS"} +{} +``` + +as well as the file `tutorial-defaults/shell/test/TARGETS` with content + +``` {.jsonc srcname="tutorial-defaults/shell/test/TARGETS"} +{} +``` + +Now we can run the test (i.e., build the test result): + +``` sh +$ just-mr build tests greet +INFO: Requested target is [["@","tutorial","tests","greet"],{}] +INFO: Analysed target [["@","tutorial","tests","greet"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Target tainted ["test"]. +INFO: Discovered 5 actions, 3 trees, 1 blobs +INFO: Building [["@","tutorial","tests","greet"],{}]. +INFO: Processed 5 actions, 2 cache hits. +INFO: Artifacts built, logical paths are: + result [7ef22e9a431ad0272713b71fdc8794016c8ef12f:5:f] + stderr [8b137891791fe96927ad78e64b0aad7bded08bdc:1:f] + stdout [ae6c6813755da67954a4a562f6d2ef01578c3e89:60:f] + time-start [8e7a68af8d5d7a6d0036c1126ff5c16a5045ae95:11:f] + time-stop [8e7a68af8d5d7a6d0036c1126ff5c16a5045ae95:11:f] + (1 runfiles omitted.) +INFO: Target tainted ["test"]. +$ +``` + +Note that the target is correctly reported as tainted with `"test"`. It +will produce 3 additional actions for compiling, linking and running the +test binary. + +The result of the test target are 5 artifacts: `result` (containing +`UNKNOWN`, `PASS`, or `FAIL`), `stderr`, `stdout`, `time-start`, and +`time-stop`, and a single runfile (omitted in the output above), which +is a tree artifact with the name `test_greet` that contains all of the +above artifacts. The test was run successfully as otherwise all reported +artifacts would have been reported as `FAILED` in the output, and +justbuild would have returned the exit code `2`. + +To immediately print the standard output produced by the test binary on +the command line, the `-P` option can be used. Argument to this option +is the name of the artifact that should be printed on the command line, +in our case `stdout`: + +``` sh +$ just-mr build tests greet --log-limit 1 -P stdout +greet output: Hello World! + +greet output: Hello Universe! + +$ +``` + +Note that `--log-limit 1` was just added to omit justbuild's `INFO:` +prints. + +Our test binary does not have any useful options for directly +interacting with it. When working with test frameworks, it sometimes can +be desirable to get hold of the test binary itself for manual +interaction. The running of the test binary is the last action +associated with the test and the test binary is, of course, one of its +inputs. + +``` sh +$ just-mr analyse --request-action-input -1 tests greet +INFO: Requested target is [["@","tutorial","tests","greet"],{}] +INFO: Request is input of action #-1 +INFO: Result of input of action #-1 of target [["@","tutorial","tests","greet"],{}]: { + "artifacts": { + "runner": {"data":{"file_type":"x","id":"0647621fba9b22f0727fbef98104f3e398496e2f","size":1876},"type":"KNOWN"}, + "test": {"data":{"id":"465b690f0b006553c15fb059e2293011c20f74d4","path":"test_greet"},"type":"ACTION"}, + "test-args.json": {"data":{"file_type":"f","id":"0637a088a01e8ddab3bf3fa98dbe804cbde1a0dc","size":2},"type":"KNOWN"}, + "test-launcher.json": {"data":{"file_type":"f","id":"0637a088a01e8ddab3bf3fa98dbe804cbde1a0dc","size":2},"type":"KNOWN"} + }, + "provides": { + "cmd": [ + "./runner" + ], + "env": { + }, + "may_fail": "CC test /test_greet failed", + "output": [ + "result", + "stderr", + "stdout", + "time-start", + "time-stop" + ], + "output_dirs": [ + ] + }, + "runfiles": { + } + } +INFO: Target tainted ["test"]. +$ +``` + +The provided data also shows us the precise description of the action +for which we request the input. This allows us to manually rerun the +action. Or we can simply interact with the test binary manually after +installing the inputs to this action. Requesting the inputs of an action +can also be useful when debugging a build failure. + +``` sh +$ just-mr install -o work --request-action-input -1 tests greet +INFO: Requested target is [["@","tutorial","tests","greet"],{}] +INFO: Request is input of action #-1 +INFO: Analysed target [["@","tutorial","tests","greet"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Target tainted ["test"]. +INFO: Discovered 5 actions, 3 trees, 1 blobs +INFO: Building input of action #-1 of [["@","tutorial","tests","greet"],{}]. +INFO: Processed 4 actions, 4 cache hits. +INFO: Artifacts can be found in: + /tmp/tutorial/work/runner [0647621fba9b22f0727fbef98104f3e398496e2f:1876:x] + /tmp/tutorial/work/test [00458536b165abdee1802e5fb7b0922e04c81491:20352:x] + /tmp/tutorial/work/test-args.json [0637a088a01e8ddab3bf3fa98dbe804cbde1a0dc:2:f] + /tmp/tutorial/work/test-launcher.json [0637a088a01e8ddab3bf3fa98dbe804cbde1a0dc:2:f] +INFO: Target tainted ["test"]. +$ cd work/ +$ ./test +greet output: Hello World! + +greet output: Hello Universe! + +$ echo $? +0 +$ cd .. +$ rm -rf work +``` + +Creating a shell test +--------------------- + +Similarly, to create a shell test for testing the `helloworld` binary, a +test script `tests/test_helloworld.sh` must be provided: + +``` {.sh srcname="tests/test_helloworld.sh"} +set -e +[ "$(./helloworld)" = "Hello Universe!" ] +``` + +The test target for this shell tests uses the rule +`["@", "rules", "shell/test", "script"]` and must depend on the +`"helloworld"` target. To create the test target, add the following to +the `tests/TARGETS` file: + +``` {.jsonc srcname="tests/TARGETS"} +... +, "helloworld": + { "type": ["@", "rules", "shell/test", "script"] + , "name": ["test_helloworld"] + , "test": ["test_helloworld.sh"] + , "deps": [["", "helloworld"]] + } +... +``` + +Now we can run the shell test (i.e., build the test result): + +``` sh +$ just-mr build tests helloworld +INFO: Requested target is [["@","tutorial","tests","helloworld"],{}] +INFO: Analysed target [["@","tutorial","tests","helloworld"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Target tainted ["test"]. +INFO: Discovered 5 actions, 4 trees, 0 blobs +INFO: Building [["@","tutorial","tests","helloworld"],{}]. +INFO: Processed 5 actions, 4 cache hits. +INFO: Artifacts built, logical paths are: + result [7ef22e9a431ad0272713b71fdc8794016c8ef12f:5:f] + stderr [e69de29bb2d1d6434b8b29ae775ad8c2e48c5391:0:f] + stdout [e69de29bb2d1d6434b8b29ae775ad8c2e48c5391:0:f] + time-start [c31e92e6b16dacec4ee95beefcc6a688dbffee2d:11:f] + time-stop [c31e92e6b16dacec4ee95beefcc6a688dbffee2d:11:f] + (1 runfiles omitted.) +INFO: Target tainted ["test"]. +$ +``` + +The result is also similar, containing also the 5 artifacts and a single +runfile (omitted in the output above), which is a tree artifact with the +name `test_helloworld` that contains all of the above artifacts. + +Creating a compound test target +------------------------------- + +As most people probably do not want to call every test target by hand, +it is desirable to compound test target that triggers the build of +multiple test reports. To do so, an `"install"` target can be used. The +field `"deps"` of an install target is a list of targets for which the +runfiles are collected. As for the tests the runfiles happen to be tree +artifacts named the same way as the test and containing all test +results, this is precisely what we need. Furthermore, as the dependent +test targets are tainted by `"test"`, also the compound test target must +be tainted by the same string. To create the compound test target +combining the two tests above (the tests `"greet"` and `"helloworld"` +from module `"tests"`), add the following to the `tests/TARGETS` file: + +``` {.jsonc srcname="tests/TARGETS"} +... +, "ALL": + { "type": "install" + , "tainted": ["test"] + , "deps": ["greet", "helloworld"] + } +... +``` + +Now we can run all tests at once by just building the compound test +target `"ALL"`: + +``` sh +$ just-mr build tests ALL +INFO: Requested target is [["@","tutorial","tests","ALL"],{}] +INFO: Analysed target [["@","tutorial","tests","ALL"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Target tainted ["test"]. +INFO: Discovered 8 actions, 5 trees, 1 blobs +INFO: Building [["@","tutorial","tests","ALL"],{}]. +INFO: Processed 8 actions, 8 cache hits. +INFO: Artifacts built, logical paths are: + test_greet [251ba2038ccdb8ba1ae2f4e963751b9230b36646:177:t] + test_helloworld [63fa5954161b52b275b05c270e1626feaa8e178b:177:t] +INFO: Target tainted ["test"]. +$ +``` + +As a result it reports the runfiles (result directories) of both tests +as artifacts. Both tests ran successfully as none of those artifacts in +this output above are tagged as `FAILED`. diff --git a/doc/tutorial/tests.org b/doc/tutorial/tests.org deleted file mode 100644 index d6842ab2..00000000 --- a/doc/tutorial/tests.org +++ /dev/null @@ -1,337 +0,0 @@ -* Creating Tests - -To run tests with justbuild, we do /not/ have a dedicated ~test~ -subcommand. Instead, we consider tests being a specific action that -generates a test report. Consequently, we use the ~build~ subcommand -to build the test report, and thereby run the test action. Test -actions, however, are slightly different from normal actions in -that we don't want the build of the test report to be aborted if -a test action fails (but still, we want only successfully actions -taken from cache). Rules defining targets containing such special -actions have to identify themselves as /tainted/ by specifying -a string explaining why such special actions are justified; in -our case, the string is ~"test"~. Besides the implicit marking by -using a tainted rule, those tainting strings can also be explicitly -assigned by the user in the definition of a target, e.g., to mark -test data. Any target has to be tainted with (at least) all the -strings any of its dependencies is tainted with. In this way, it -is ensured that no test target will end up in a production build. - -For the remainder of this section, we expect to have the project files available -resulting from successfully completing the tutorial section on /Building C++ -Hello World/. We will demonstrate how to write a test binary for the ~greet~ -library and a shell test for the ~helloworld~ binary. - -** Creating a C++ test binary - -First, we will create a C++ test binary for testing the correct functionality of -the ~greet~ library. Therefore, we need to provide a C++ source file that performs -the actual testing and returns non-~0~ on failure. For simplicity reasons, we do -not use a testing framework for this tutorial. A simple test that captures -standard output and verifies it with the expected output should be provided in -the file ~tests/greet.test.cpp~: - -#+SRCNAME: tests/greet.test.cpp -#+BEGIN_SRC cpp -#include -#include -#include -#include -#include "greet/greet.hpp" - -template -auto capture_stdout(std::function const& func) -> std::string { - int fd[2]; - if (pipe(fd) < 0) return {}; - int fd_stdout = dup(fileno(stdout)); - fflush(stdout); - dup2(fd[1], fileno(stdout)); - - func(); - fflush(stdout); - - std::string buf(kMaxBufSize, '\0'); - auto n = read(fd[0], &(*buf.begin()), kMaxBufSize); - close(fd[0]); - close(fd[1]); - dup2(fd_stdout, fileno(stdout)); - return buf.substr(0, n); -} - -auto test_greet(std::string const& name) -> bool { - auto expect = std::string{"Hello "} + name + "!\n"; - auto result = capture_stdout([&name] { greet(name); }); - std::cout << "greet output: " << result << std::endl; - return result == expect; -} - -int main() { - return test_greet("World") && test_greet("Universe") ? 0 : 1; -} -#+END_SRC - -Next, a new test target needs to be created in module ~greet~. This target uses -the rule ~["@", "rules", "CC/test", "test"]~ and needs to depend on the -~["greet", "greet"]~ target. To create the test target, add the following to -~tests/TARGETS~: - -#+SRCNAME: tests/TARGETS -#+BEGIN_SRC js -{ "greet": - { "type": ["@", "rules", "CC/test", "test"] - , "name": ["test_greet"] - , "srcs": ["greet.test.cpp"] - , "private-deps": [["greet", "greet"]] - } -} -#+END_SRC - -Before we can run the test, a proper default module for ~CC/test~ must be -provided. By specifying the appropriate target in this module the default test -runner can be overwritten by a different test runner fom the rule's workspace -root. Moreover, all test targets share runner infrastructure from ~shell/test~, -e.g., summarizing multiple runs per test (to detect flakyness) if the configuration -variable ~RUNS_PER_TEST~ is set. - -However, in our case, we want to use the default runner and therefore it -is sufficient to create an empty module. To do so, create the file -~tutorial-defaults/CC/test/TARGETS~ with content - -#+SRCNAME: tutorial-defaults/CC/test/TARGETS -#+BEGIN_SRC js -{} -#+END_SRC - -as well as the file ~tutorial-defaults/shell/test/TARGETS~ with content - -#+SRCNAME: tutorial-defaults/shell/test/TARGETS -#+BEGIN_SRC js -{} -#+END_SRC - - -Now we can run the test (i.e., build the test result): - -#+BEGIN_SRC sh -$ just-mr build tests greet -INFO: Requested target is [["@","tutorial","tests","greet"],{}] -INFO: Analysed target [["@","tutorial","tests","greet"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Target tainted ["test"]. -INFO: Discovered 5 actions, 3 trees, 1 blobs -INFO: Building [["@","tutorial","tests","greet"],{}]. -INFO: Processed 5 actions, 2 cache hits. -INFO: Artifacts built, logical paths are: - result [7ef22e9a431ad0272713b71fdc8794016c8ef12f:5:f] - stderr [8b137891791fe96927ad78e64b0aad7bded08bdc:1:f] - stdout [ae6c6813755da67954a4a562f6d2ef01578c3e89:60:f] - time-start [8e7a68af8d5d7a6d0036c1126ff5c16a5045ae95:11:f] - time-stop [8e7a68af8d5d7a6d0036c1126ff5c16a5045ae95:11:f] - (1 runfiles omitted.) -INFO: Target tainted ["test"]. -$ -#+END_SRC - -Note that the target is correctly reported as tainted with ~"test"~. It will -produce 3 additional actions for compiling, linking and running the test binary. - -The result of the test target are 5 artifacts: ~result~ (containing ~UNKNOWN~, -~PASS~, or ~FAIL~), ~stderr~, ~stdout~, ~time-start~, and ~time-stop~, and a -single runfile (omitted in the output above), which is a tree artifact with the -name ~test_greet~ that contains all of the above artifacts. The test was run -successfully as otherwise all reported artifacts would have been reported as -~FAILED~ in the output, and justbuild would have returned the exit code ~2~. - -To immediately print the standard output produced by the test binary on the -command line, the ~-P~ option can be used. Argument to this option is the name -of the artifact that should be printed on the command line, in our case -~stdout~: - -#+BEGIN_SRC sh -$ just-mr build tests greet --log-limit 1 -P stdout -greet output: Hello World! - -greet output: Hello Universe! - -$ -#+END_SRC - -Note that ~--log-limit 1~ was just added to omit justbuild's ~INFO:~ prints. - -Our test binary does not have any useful options for directly interacting -with it. When working with test frameworks, it sometimes can be desirable to -get hold of the test binary itself for manual interaction. The running of -the test binary is the last action associated with the test and the test -binary is, of course, one of its inputs. - -#+BEGIN_SRC sh -$ just-mr analyse --request-action-input -1 tests greet -INFO: Requested target is [["@","tutorial","tests","greet"],{}] -INFO: Request is input of action #-1 -INFO: Result of input of action #-1 of target [["@","tutorial","tests","greet"],{}]: { - "artifacts": { - "runner": {"data":{"file_type":"x","id":"0647621fba9b22f0727fbef98104f3e398496e2f","size":1876},"type":"KNOWN"}, - "test": {"data":{"id":"465b690f0b006553c15fb059e2293011c20f74d4","path":"test_greet"},"type":"ACTION"}, - "test-args.json": {"data":{"file_type":"f","id":"0637a088a01e8ddab3bf3fa98dbe804cbde1a0dc","size":2},"type":"KNOWN"}, - "test-launcher.json": {"data":{"file_type":"f","id":"0637a088a01e8ddab3bf3fa98dbe804cbde1a0dc","size":2},"type":"KNOWN"} - }, - "provides": { - "cmd": [ - "./runner" - ], - "env": { - }, - "may_fail": "CC test /test_greet failed", - "output": [ - "result", - "stderr", - "stdout", - "time-start", - "time-stop" - ], - "output_dirs": [ - ] - }, - "runfiles": { - } - } -INFO: Target tainted ["test"]. -$ -#+END_SRC - -The provided data also shows us the precise description of the action -for which we request the input. This allows us to manually rerun -the action. Or we can simply interact with the test binary manually -after installing the inputs to this action. Requesting the inputs -of an action can also be useful when debugging a build failure. - -#+BEGIN_SRC sh -$ just-mr install -o work --request-action-input -1 tests greet -INFO: Requested target is [["@","tutorial","tests","greet"],{}] -INFO: Request is input of action #-1 -INFO: Analysed target [["@","tutorial","tests","greet"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Target tainted ["test"]. -INFO: Discovered 5 actions, 3 trees, 1 blobs -INFO: Building input of action #-1 of [["@","tutorial","tests","greet"],{}]. -INFO: Processed 4 actions, 4 cache hits. -INFO: Artifacts can be found in: - /tmp/tutorial/work/runner [0647621fba9b22f0727fbef98104f3e398496e2f:1876:x] - /tmp/tutorial/work/test [00458536b165abdee1802e5fb7b0922e04c81491:20352:x] - /tmp/tutorial/work/test-args.json [0637a088a01e8ddab3bf3fa98dbe804cbde1a0dc:2:f] - /tmp/tutorial/work/test-launcher.json [0637a088a01e8ddab3bf3fa98dbe804cbde1a0dc:2:f] -INFO: Target tainted ["test"]. -$ cd work/ -$ ./test -greet output: Hello World! - -greet output: Hello Universe! - -$ echo $? -0 -$ cd .. -$ rm -rf work -#+END_SRC - -** Creating a shell test - -Similarly, to create a shell test for testing the ~helloworld~ binary, a test -script ~tests/test_helloworld.sh~ must be provided: - -#+SRCNAME: tests/test_helloworld.sh -#+BEGIN_SRC sh -set -e -[ "$(./helloworld)" = "Hello Universe!" ] -#+END_SRC - -The test target for this shell tests uses the rule -~["@", "rules", "shell/test", "script"]~ and must depend on the ~"helloworld"~ -target. To create the test target, add the following to the ~tests/TARGETS~ -file: - -#+SRCNAME: tests/TARGETS -#+BEGIN_SRC js -... -, "helloworld": - { "type": ["@", "rules", "shell/test", "script"] - , "name": ["test_helloworld"] - , "test": ["test_helloworld.sh"] - , "deps": [["", "helloworld"]] - } -... -#+END_SRC - -Now we can run the shell test (i.e., build the test result): - -#+BEGIN_SRC sh -$ just-mr build tests helloworld -INFO: Requested target is [["@","tutorial","tests","helloworld"],{}] -INFO: Analysed target [["@","tutorial","tests","helloworld"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Target tainted ["test"]. -INFO: Discovered 5 actions, 4 trees, 0 blobs -INFO: Building [["@","tutorial","tests","helloworld"],{}]. -INFO: Processed 5 actions, 4 cache hits. -INFO: Artifacts built, logical paths are: - result [7ef22e9a431ad0272713b71fdc8794016c8ef12f:5:f] - stderr [e69de29bb2d1d6434b8b29ae775ad8c2e48c5391:0:f] - stdout [e69de29bb2d1d6434b8b29ae775ad8c2e48c5391:0:f] - time-start [c31e92e6b16dacec4ee95beefcc6a688dbffee2d:11:f] - time-stop [c31e92e6b16dacec4ee95beefcc6a688dbffee2d:11:f] - (1 runfiles omitted.) -INFO: Target tainted ["test"]. -$ -#+END_SRC - -The result is also similar, containing also the 5 artifacts and a single runfile -(omitted in the output above), which is a tree artifact with the name -~test_helloworld~ that contains all of the above artifacts. - -** Creating a compound test target - -As most people probably do not want to call every test target by hand, it is -desirable to compound test target that triggers the build of multiple test -reports. To do so, an ~"install"~ target can be used. The field ~"deps"~ of -an install target is a list of targets for which the runfiles are collected. -As for the tests the runfiles happen to be -tree artifacts named the same way as the test and containing all test results, -this is precisely what we need. -Furthermore, as the dependent test targets are tainted by ~"test"~, also the -compound test target must be tainted by the same string. To create the compound -test target combining the two tests above (the tests ~"greet"~ and -~"helloworld"~ from module ~"tests"~), add the following to the ~tests/TARGETS~ -file: - -#+SRCNAME: tests/TARGETS -#+BEGIN_SRC js -... -, "ALL": - { "type": "install" - , "tainted": ["test"] - , "deps": ["greet", "helloworld"] - } -... -#+END_SRC - -Now we can run all tests at once by just building the compound test target -~"ALL"~: - -#+BEGIN_SRC sh -$ just-mr build tests ALL -INFO: Requested target is [["@","tutorial","tests","ALL"],{}] -INFO: Analysed target [["@","tutorial","tests","ALL"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Target tainted ["test"]. -INFO: Discovered 8 actions, 5 trees, 1 blobs -INFO: Building [["@","tutorial","tests","ALL"],{}]. -INFO: Processed 8 actions, 8 cache hits. -INFO: Artifacts built, logical paths are: - test_greet [251ba2038ccdb8ba1ae2f4e963751b9230b36646:177:t] - test_helloworld [63fa5954161b52b275b05c270e1626feaa8e178b:177:t] -INFO: Target tainted ["test"]. -$ -#+END_SRC - -As a result it reports the runfiles (result directories) of both tests as -artifacts. Both tests ran successfully as none of those artifacts in this output -above are tagged as ~FAILED~. diff --git a/doc/tutorial/third-party-software.md b/doc/tutorial/third-party-software.md new file mode 100644 index 00000000..daaf5b2d --- /dev/null +++ b/doc/tutorial/third-party-software.md @@ -0,0 +1,473 @@ +Building Third-party Software +============================= + +Third-party projects usually ship with their own build description, +which often happens to be not compatible with justbuild. Nevertheless, +it is highly desireable to include external projects via their source +code base, instead of relying on the integration of out-of-band binary +distributions. justbuild offers a flexible approach to provide the +required build description via an overlay layer without the need to +touch the original code base. + +For the remainder of this section, we expect to have the project files +available resulting from successfully completing the tutorial section on +*Building C++ Hello World*. We will demonstrate how to use the +open-source project [fmtlib](https://github.com/fmtlib/fmt) as an +example for integrating third-party software to a justbuild project. + +Creating the target overlay layer for fmtlib +-------------------------------------------- + +Before we construct the overlay layer for fmtlib, we need to determine +its file structure ([tag +8.1.1](https://github.com/fmtlib/fmt/tree/8.1.1)). The relevant header +and source files are structured as follows: + + fmt + | + +--include + | +--fmt + | +--*.h + | + +--src + +--format.cc + +--os.cc + +The public headers can be found in `include/fmt`, while the library's +source files are located in `src`. For the overlay layer, the `TARGETS` +files should be placed in a tree structure that resembles the original +code base's structure. It is also good practice to provide a top-level +`TARGETS` file, leading to the following structure for the overlay: + + fmt-layer + | + +--TARGETS + +--include + | +--fmt + | +--TARGETS + | + +--src + +--TARGETS + +Let's create the overlay structure: + +``` sh +$ mkdir -p fmt-layer/include/fmt +$ mkdir -p fmt-layer/src +``` + +The directory `include/fmt` contains only header files. As we want all +files in this directory to be included in the `"hdrs"` target, we can +safely use the explicit `TREE` reference[^1], which collects, in a +single artifact (describing a directory) *all* directory contents from +`"."` of the workspace root. Note that the `TARGETS` file is only part +of the overlay, and therefore will not be part of this tree. +Furthermore, this tree should be staged to `"fmt"`, so that any consumer +can include those headers via ``. The resulting header +directory target `"hdrs"` in `include/fmt/TARGETS` should be described +as: + +``` {.jsonc srcname="fmt-layer/include/fmt/TARGETS"} +{ "hdrs": + { "type": ["@", "rules", "data", "staged"] + , "srcs": [["TREE", null, "."]] + , "stage": ["fmt"] + } +} +``` + +The actual library target is defined in the directory `src`. For the +public headers, it refers to the previously created `"hdrs"` target via +its fully-qualified target name (`["include/fmt", "hdrs"]`). Source +files are the two local files `format.cc`, and `os.cc`. The final target +description in `src/TARGETS` will look like this: + +``` {.jsonc srcname="fmt-layer/src/TARGETS"} +{ "fmt": + { "type": ["@", "rules", "CC", "library"] + , "name": ["fmt"] + , "hdrs": [["include/fmt", "hdrs"]] + , "srcs": ["format.cc", "os.cc"] + } +} +``` + +Finally, the top-level `TARGETS` file can be created. While it is +technically not strictly required, it is considered good practice to +*export* every target that may be used by another project. Exported +targets are subject to high-level target caching, which allows to skip +the analysis and traversal of entire subgraphs in the action graph. +Therefore, we create an export target that exports the target +`["src", "fmt"]`, with only the variables in the field +`"flexible_config"` being propagated. The top-level `TARGETS` file +contains the following content: + +``` {.jsonc srcname="fmt-layer/TARGETS"} +{ "fmt": + { "type": "export" + , "target": ["src", "fmt"] + , "flexible_config": ["CXX", "CXXFLAGS", "ADD_CXXFLAGS", "AR", "ENV"] + } +} +``` + +After adding the library to the multi-repository configuration (next +step), the list of configuration variables a target, like `["src", +"fmt"]`, actually depends on can be obtained using the `--dump-vars` +option of the `analyse` subcommand. In this way, an informed decision +can be taken when deciding which variables of the export target to make +tunable for the consumer. + +Adding fmtlib to the Multi-Repository Configuration +--------------------------------------------------- + +Based on the *hello world* tutorial, we can extend the existing +`repos.json` by the layer definition `"fmt-targets-layer"` and the +repository `"fmtlib"`, which is based on the Git repository with its +target root being overlayed. Furthermore, we want to use `"fmtlib"` in +the repository `"tutorial"`, and therefore need to introduce an +additional binding `"format"` for it: + +``` {.jsonc srcname="repos.json"} +{ "main": "tutorial" +, "repositories": + { "rules-cc": + { "repository": + { "type": "git" + , "branch": "master" + , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" + , "repository": "https://github.com/just-buildsystem/rules-cc.git" + , "subdir": "rules" + } + , "target_root": "tutorial-defaults" + , "rule_root": "rules-cc" + } + , "tutorial": + { "repository": {"type": "file", "path": "."} + , "bindings": {"rules": "rules-cc", "format": "fmtlib"} + } + , "tutorial-defaults": + { "repository": {"type": "file", "path": "./tutorial-defaults"} + } + , "fmt-targets-layer": + { "repository": {"type": "file", "path": "./fmt-layer"} + } + , "fmtlib": + { "repository": + { "type": "git" + , "branch": "8.1.1" + , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9" + , "repository": "https://github.com/fmtlib/fmt.git" + } + , "target_root": "fmt-targets-layer" + , "bindings": {"rules": "rules-cc"} + } + } +} +``` + +This `"format"` binding can you be used to add a new private dependency +in `greet/TARGETS`: + +``` {.jsonc srcname="greet/TARGETS"} +{ "greet": + { "type": ["@", "rules", "CC", "library"] + , "name": ["greet"] + , "hdrs": ["greet.hpp"] + , "srcs": ["greet.cpp"] + , "stage": ["greet"] + , "private-deps": [["@", "format", "", "fmt"]] + } +} +``` + +Consequently, the `fmtlib` library can now be used by `greet/greet.cpp`: + +``` {.cpp srcname="greet/greet.cpp"} +#include "greet.hpp" +#include + +void greet(std::string const& s) { + fmt::print("Hello {}!\n", s); +} +``` + +Due to changes made to `repos.json`, building this tutorial requires to +rerun `just-mr`, which will fetch the necessary sources for the external +repositories: + +``` sh +$ just-mr build helloworld +INFO: Requested target is [["@","tutorial","","helloworld"],{}] +INFO: Analysed target [["@","tutorial","","helloworld"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 1 not eligible for caching +INFO: Discovered 7 actions, 3 trees, 0 blobs +INFO: Building [["@","tutorial","","helloworld"],{}]. +INFO: Processed 7 actions, 1 cache hits. +INFO: Artifacts built, logical paths are: + helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x] +$ +``` + +Note to build the `fmt` target alone, its containing repository `fmtlib` +must be specified via the `--main` option: + +``` sh +$ just-mr --main fmtlib build fmt +INFO: Requested target is [["@","fmtlib","","fmt"],{}] +INFO: Analysed target [["@","fmtlib","","fmt"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 1 not eligible for caching +INFO: Discovered 3 actions, 1 trees, 0 blobs +INFO: Building [["@","fmtlib","","fmt"],{}]. +INFO: Processed 3 actions, 3 cache hits. +INFO: Artifacts built, logical paths are: + libfmt.a [513b2ac17c557675fc841f3ebf279003ff5a73ae:240914:f] + (1 runfiles omitted.) +$ +``` + +Employing high-level target caching +----------------------------------- + +The make use of high-level target caching for exported targets, we need +to ensure that all inputs to an export target are transitively +content-fixed. This is automatically the case for `"type":"git"` +repositories. However, the `libfmt` repository also depends on +`"rules-cc"`, `"tutorial-defaults"`, and `"fmt-target-layer"`. As the +latter two are `"type":"file"` repositories, they must be put under Git +versioning first: + +``` sh +$ git init . +$ git add tutorial-defaults fmt-layer +$ git commit -m"fix compile flags and fmt targets layer" +``` + +Note that `rules-cc` already is under Git versioning. + +Now, to instruct `just-mr` to use the content-fixed, committed source +trees of those `"type":"file"` repositories the pragma `"to_git"` must +be set for them in `repos.json`: + +``` {.jsonc srcname="repos.json"} +{ "main": "tutorial" +, "repositories": + { "rules-cc": + { "repository": + { "type": "git" + , "branch": "master" + , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" + , "repository": "https://github.com/just-buildsystem/rules-cc.git" + , "subdir": "rules" + } + , "target_root": "tutorial-defaults" + , "rule_root": "rules-cc" + } + , "tutorial": + { "repository": {"type": "file", "path": "."} + , "bindings": {"rules": "rules-cc", "format": "fmtlib"} + } + , "tutorial-defaults": + { "repository": + { "type": "file" + , "path": "./tutorial-defaults" + , "pragma": {"to_git": true} + } + } + , "fmt-targets-layer": + { "repository": + { "type": "file" + , "path": "./fmt-layer" + , "pragma": {"to_git": true} + } + } + , "fmtlib": + { "repository": + { "type": "git" + , "branch": "master" + , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9" + , "repository": "https://github.com/fmtlib/fmt.git" + } + , "target_root": "fmt-targets-layer" + , "bindings": {"rules": "rules-cc"} + } + } +} +``` + +Due to changes in the repository configuration, we need to rebuild and +the benefits of the target cache should be visible on the second build: + +``` sh +$ just-mr build helloworld +INFO: Requested target is [["@","tutorial","","helloworld"],{}] +INFO: Analysed target [["@","tutorial","","helloworld"],{}] +INFO: Export targets found: 0 cached, 1 uncached, 0 not eligible for caching +INFO: Discovered 7 actions, 3 trees, 0 blobs +INFO: Building [["@","tutorial","","helloworld"],{}]. +INFO: Processed 7 actions, 7 cache hits. +INFO: Artifacts built, logical paths are: + helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x] +$ +$ just-mr build helloworld +INFO: Requested target is [["@","tutorial","","helloworld"],{}] +INFO: Analysed target [["@","tutorial","","helloworld"],{}] +INFO: Export targets found: 1 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 4 actions, 2 trees, 0 blobs +INFO: Building [["@","tutorial","","helloworld"],{}]. +INFO: Processed 4 actions, 4 cache hits. +INFO: Artifacts built, logical paths are: + helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x] +$ +``` + +Note that in the second run the export target `"fmt"` was taken from +cache and its 3 actions were eliminated, as their result has been +recorded to the high-level target cache during the first run. + +Combining overlay layers for multiple projects +---------------------------------------------- + +Projects typically depend on multiple external repositories. Creating an +overlay layer for each external repository might unnecessarily clutter +up the repository configuration and the file structure of your +repository. One solution to mitigate this issue is to combine the +`TARGETS` files of multiple external repositories in a single overlay +layer. To avoid conflicts, the `TARGETS` files can be assigned different +file names per repository. As an example, imagine a common overlay layer +with the files `TARGETS.fmt` and `TARGETS.gsl` for the repositories +`"fmtlib"` and `"gsl-lite"`, respectively: + + common-layer + | + +--TARGETS.fmt + +--TARGETS.gsl + +--include + | +--fmt + | | +--TARGETS.fmt + | +--gsl + | +--TARGETS.gsl + | + +--src + +--TARGETS.fmt + +Such a common overlay layer can be used as the target root for both +repositories with only one difference: the `"target_file_name"` field. +By specifying this field, the dispatch where to find the respective +target description for each repository is implemented. For the given +example, the following `repos.json` defines the overlay +`"common-targets-layer"`, which is used by `"fmtlib"` and `"gsl-lite"`: + +``` {.jsonc srcname="repos.json"} +{ "main": "tutorial" +, "repositories": + { "rules-cc": + { "repository": + { "type": "git" + , "branch": "master" + , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" + , "repository": "https://github.com/just-buildsystem/rules-cc.git" + , "subdir": "rules" + } + , "target_root": "tutorial-defaults" + , "rule_root": "rules-cc" + } + , "tutorial": + { "repository": {"type": "file", "path": "."} + , "bindings": {"rules": "rules-cc", "format": "fmtlib"} + } + , "tutorial-defaults": + { "repository": + { "type": "file" + , "path": "./tutorial-defaults" + , "pragma": {"to_git": true} + } + } + , "common-targets-layer": + { "repository": + { "type": "file" + , "path": "./common-layer" + , "pragma": {"to_git": true} + } + } + , "fmtlib": + { "repository": + { "type": "git" + , "branch": "8.1.1" + , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9" + , "repository": "https://github.com/fmtlib/fmt.git" + } + , "target_root": "common-targets-layer" + , "target_file_name": "TARGETS.fmt" + , "bindings": {"rules": "rules-cc"} + } + , "gsl-lite": + { "repository": + { "type": "git" + , "branch": "v0.40.0" + , "commit": "d6c8af99a1d95b3db36f26b4f22dc3bad89952de" + , "repository": "https://github.com/gsl-lite/gsl-lite.git" + } + , "target_root": "common-targets-layer" + , "target_file_name": "TARGETS.gsl" + , "bindings": {"rules": "rules-cc"} + } + } +} +``` + +Using pre-built dependencies +---------------------------- + +While building external dependencies from source brings advantages, most +prominently the flexibility to quickly and seamlessly switch to a +different build configuration (production, debug, instrumented for +performance analysis; cross-compiling for a different target +architecture), there are also legitimate reasons to use pre-built +dependencies. The most prominent one is if your project is packaged as +part of a larger distribution. For that reason, just also has (in +`etc/import.prebuilt`) target files for all its dependencies assuming +they are pre-installed. The reason why target files are used at all for +this situation is twofold. + + - On the one hand, having a target allows the remaining targets to not + care about where their dependencies come from, or if it is a build + against pre-installed dependencies or not. Also, the top-level + binary does not have to know the linking requirements of its + transitive dependencies. In other words, information stays where it + belongs to and if one target acquires a new dependency, the + information is automatically propagated to all targets using it. + - Still some information is needed to use a pre-installed library and, + as explained, a target describing the pre-installed library is the + right place to collect this information. + - The public header files of the library. By having this explicit, + we do not accumulate directories in the include search path and + hence also properly detect include conflicts. + - The information on how to link the library itself (i.e., + basically its base name). + - Any dependencies on other libraries that the library might have. + This information is used to obtain the correct linking order and + complete transitive linking arguments while keeping the + description maintainable, as each target still only declares its + direct dependencies. + +The target description for a pre-built version of the format library +that was used as an example in this section is shown next; with our +staging mechanism the logical repository it belongs to is rooted in the +`fmt` subdirectory of the `include` directory of the ambient system. + +``` {.jsonc srcname="etc/import.prebuilt/TARGETS.fmt"} +{ "fmt": + { "type": ["@", "rules", "CC", "library"] + , "name": ["fmt"] + , "stage": ["fmt"] + , "hdrs": [["TREE", null, "."]] + , "private-ldflags": ["-lfmt"] + } +} +``` + +[^1]: Explicit `TREE` references are always a list of length 3, to + distinguish them from target references of length 2 (module and + target name). Furthermore, the second list element is always `null` + as we only want to allow tree references from the current module. diff --git a/doc/tutorial/third-party-software.org b/doc/tutorial/third-party-software.org deleted file mode 100644 index d1712cc8..00000000 --- a/doc/tutorial/third-party-software.org +++ /dev/null @@ -1,475 +0,0 @@ -* Building Third-party Software - -Third-party projects usually ship with their own build description, which often -happens to be not compatible with justbuild. Nevertheless, it is highly -desireable to include external projects via their source code base, instead of -relying on the integration of out-of-band binary distributions. justbuild offers -a flexible approach to provide the required build description via an overlay -layer without the need to touch the original code base. - -For the remainder of this section, we expect to have the project files available -resulting from successfully completing the tutorial section on /Building C++ -Hello World/. We will demonstrate how to use the open-source project -[[https://github.com/fmtlib/fmt][fmtlib]] as an example for integrating -third-party software to a justbuild project. - -** Creating the target overlay layer for fmtlib - -Before we construct the overlay layer for fmtlib, we need to determine its file -structure ([[https://github.com/fmtlib/fmt/tree/8.1.1][tag 8.1.1]]). The -relevant header and source files are structured as follows: - -#+BEGIN_SRC - fmt - | - +--include - | +--fmt - | +--*.h - | - +--src - +--format.cc - +--os.cc -#+END_SRC - -The public headers can be found in ~include/fmt~, while the library's source -files are located in ~src~. For the overlay layer, the ~TARGETS~ files should be -placed in a tree structure that resembles the original code base's structure. -It is also good practice to provide a top-level ~TARGETS~ file, leading to the -following structure for the overlay: - -#+BEGIN_SRC - fmt-layer - | - +--TARGETS - +--include - | +--fmt - | +--TARGETS - | - +--src - +--TARGETS -#+END_SRC - -Let's create the overlay structure: - -#+BEGIN_SRC sh -$ mkdir -p fmt-layer/include/fmt -$ mkdir -p fmt-layer/src -#+END_SRC - -The directory ~include/fmt~ contains only header files. As we want all files in -this directory to be included in the ~"hdrs"~ target, we can safely -use the explicit ~TREE~ reference[fn:1], which collects, in a single -artifact (describing a directory) /all/ directory contents -from ~"."~ of the workspace root. Note that the ~TARGETS~ file is only part of -the overlay, and -therefore will not be part of this tree. Furthermore, this tree should be staged -to ~"fmt"~, so that any consumer can include those headers via ~~. The -resulting header directory target ~"hdrs"~ in ~include/fmt/TARGETS~ should be -described as: - -[fn:1] Explicit ~TREE~ references are always a list of length 3, to distinguish -them from target references of length 2 (module and target name). Furthermore, -the second list element is always ~null~ as we only want to allow tree -references from the current module. - - -#+SRCNAME: fmt-layer/include/fmt/TARGETS -#+BEGIN_SRC js -{ "hdrs": - { "type": ["@", "rules", "data", "staged"] - , "srcs": [["TREE", null, "."]] - , "stage": ["fmt"] - } -} -#+END_SRC - -The actual library target is defined in the directory ~src~. For the public -headers, it refers to the previously created ~"hdrs"~ target via its -fully-qualified target name (~["include/fmt", "hdrs"]~). Source files are the -two local files ~format.cc~, and ~os.cc~. The final target description in -~src/TARGETS~ will look like this: - -#+SRCNAME: fmt-layer/src/TARGETS -#+BEGIN_SRC js -{ "fmt": - { "type": ["@", "rules", "CC", "library"] - , "name": ["fmt"] - , "hdrs": [["include/fmt", "hdrs"]] - , "srcs": ["format.cc", "os.cc"] - } -} -#+END_SRC - -Finally, the top-level ~TARGETS~ file can be created. While it is technically -not strictly required, it is considered good practice to /export/ every target -that may be used by another project. Exported targets are subject to high-level -target caching, which allows to skip the analysis and traversal of entire -subgraphs in the action graph. Therefore, we create an export target that -exports the target ~["src", "fmt"]~, with only the variables in the field -~"flexible_config"~ being propagated. The top-level ~TARGETS~ file contains the -following content: - -#+SRCNAME: fmt-layer/TARGETS -#+BEGIN_SRC js -{ "fmt": - { "type": "export" - , "target": ["src", "fmt"] - , "flexible_config": ["CXX", "CXXFLAGS", "ADD_CXXFLAGS", "AR", "ENV"] - } -} -#+END_SRC - -After adding the library to the multi-repository configuration (next -step), the list of configuration variables a target, like ~["src", -"fmt"]~, actually depends on can be obtained using the ~--dump-vars~ -option of the ~analyse~ subcommand. In this way, an informed decision -can be taken when deciding which variables of the export target to -make tunable for the consumer. - -** Adding fmtlib to the Multi-Repository Configuration - -Based on the /hello world/ tutorial, we can extend the existing ~repos.json~ by -the layer definition ~"fmt-targets-layer"~ and the repository ~"fmtlib"~, which -is based on the Git repository with its target root being overlayed. -Furthermore, we want to use ~"fmtlib"~ in the repository ~"tutorial"~, and -therefore need to introduce an additional binding ~"format"~ for it: - -#+SRCNAME: repos.json -#+BEGIN_SRC js -{ "main": "tutorial" -, "repositories": - { "rules-cc": - { "repository": - { "type": "git" - , "branch": "master" - , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" - , "repository": "https://github.com/just-buildsystem/rules-cc.git" - , "subdir": "rules" - } - , "target_root": "tutorial-defaults" - , "rule_root": "rules-cc" - } - , "tutorial": - { "repository": {"type": "file", "path": "."} - , "bindings": {"rules": "rules-cc", "format": "fmtlib"} - } - , "tutorial-defaults": - { "repository": {"type": "file", "path": "./tutorial-defaults"} - } - , "fmt-targets-layer": - { "repository": {"type": "file", "path": "./fmt-layer"} - } - , "fmtlib": - { "repository": - { "type": "git" - , "branch": "8.1.1" - , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9" - , "repository": "https://github.com/fmtlib/fmt.git" - } - , "target_root": "fmt-targets-layer" - , "bindings": {"rules": "rules-cc"} - } - } -} -#+END_SRC - -This ~"format"~ binding can you be used to add a new private dependency in -~greet/TARGETS~: - -#+SRCNAME: greet/TARGETS -#+BEGIN_SRC js -{ "greet": - { "type": ["@", "rules", "CC", "library"] - , "name": ["greet"] - , "hdrs": ["greet.hpp"] - , "srcs": ["greet.cpp"] - , "stage": ["greet"] - , "private-deps": [["@", "format", "", "fmt"]] - } -} -#+END_SRC - -Consequently, the ~fmtlib~ library can now be used by ~greet/greet.cpp~: - -#+SRCNAME: greet/greet.cpp -#+BEGIN_SRC cpp -#include "greet.hpp" -#include - -void greet(std::string const& s) { - fmt::print("Hello {}!\n", s); -} -#+END_SRC - -Due to changes made to ~repos.json~, building this tutorial requires to rerun -~just-mr~, which will fetch the necessary sources for the external repositories: - -#+BEGIN_SRC sh -$ just-mr build helloworld -INFO: Requested target is [["@","tutorial","","helloworld"],{}] -INFO: Analysed target [["@","tutorial","","helloworld"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 1 not eligible for caching -INFO: Discovered 7 actions, 3 trees, 0 blobs -INFO: Building [["@","tutorial","","helloworld"],{}]. -INFO: Processed 7 actions, 1 cache hits. -INFO: Artifacts built, logical paths are: - helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x] -$ -#+END_SRC - -Note to build the ~fmt~ target alone, its containing repository ~fmtlib~ must be -specified via the ~--main~ option: -#+BEGIN_SRC sh -$ just-mr --main fmtlib build fmt -INFO: Requested target is [["@","fmtlib","","fmt"],{}] -INFO: Analysed target [["@","fmtlib","","fmt"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 1 not eligible for caching -INFO: Discovered 3 actions, 1 trees, 0 blobs -INFO: Building [["@","fmtlib","","fmt"],{}]. -INFO: Processed 3 actions, 3 cache hits. -INFO: Artifacts built, logical paths are: - libfmt.a [513b2ac17c557675fc841f3ebf279003ff5a73ae:240914:f] - (1 runfiles omitted.) -$ -#+END_SRC - -** Employing high-level target caching - -The make use of high-level target caching for exported targets, we need to -ensure that all inputs to an export target are transitively content-fixed. This -is automatically the case for ~"type":"git"~ repositories. However, the ~libfmt~ -repository also depends on ~"rules-cc"~, ~"tutorial-defaults"~, and -~"fmt-target-layer"~. As the latter two are ~"type":"file"~ repositories, they -must be put under Git versioning first: - -#+BEGIN_SRC sh -$ git init . -$ git add tutorial-defaults fmt-layer -$ git commit -m"fix compile flags and fmt targets layer" -#+END_SRC - -Note that ~rules-cc~ already is under Git versioning. - -Now, to instruct ~just-mr~ to use the content-fixed, committed source trees of -those ~"type":"file"~ repositories the pragma ~"to_git"~ must be set for them in -~repos.json~: - -#+SRCNAME: repos.json -#+BEGIN_SRC js -{ "main": "tutorial" -, "repositories": - { "rules-cc": - { "repository": - { "type": "git" - , "branch": "master" - , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" - , "repository": "https://github.com/just-buildsystem/rules-cc.git" - , "subdir": "rules" - } - , "target_root": "tutorial-defaults" - , "rule_root": "rules-cc" - } - , "tutorial": - { "repository": {"type": "file", "path": "."} - , "bindings": {"rules": "rules-cc", "format": "fmtlib"} - } - , "tutorial-defaults": - { "repository": - { "type": "file" - , "path": "./tutorial-defaults" - , "pragma": {"to_git": true} - } - } - , "fmt-targets-layer": - { "repository": - { "type": "file" - , "path": "./fmt-layer" - , "pragma": {"to_git": true} - } - } - , "fmtlib": - { "repository": - { "type": "git" - , "branch": "master" - , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9" - , "repository": "https://github.com/fmtlib/fmt.git" - } - , "target_root": "fmt-targets-layer" - , "bindings": {"rules": "rules-cc"} - } - } -} -#+END_SRC - -Due to changes in the repository configuration, we need to rebuild and the -benefits of the target cache should be visible on the second build: - -#+BEGIN_SRC sh -$ just-mr build helloworld -INFO: Requested target is [["@","tutorial","","helloworld"],{}] -INFO: Analysed target [["@","tutorial","","helloworld"],{}] -INFO: Export targets found: 0 cached, 1 uncached, 0 not eligible for caching -INFO: Discovered 7 actions, 3 trees, 0 blobs -INFO: Building [["@","tutorial","","helloworld"],{}]. -INFO: Processed 7 actions, 7 cache hits. -INFO: Artifacts built, logical paths are: - helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x] -$ -$ just-mr build helloworld -INFO: Requested target is [["@","tutorial","","helloworld"],{}] -INFO: Analysed target [["@","tutorial","","helloworld"],{}] -INFO: Export targets found: 1 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 4 actions, 2 trees, 0 blobs -INFO: Building [["@","tutorial","","helloworld"],{}]. -INFO: Processed 4 actions, 4 cache hits. -INFO: Artifacts built, logical paths are: - helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x] -$ -#+END_SRC - -Note that in the second run the export target ~"fmt"~ was taken from cache and -its 3 actions were eliminated, as their result has been recorded to the -high-level target cache during the first run. - -** Combining overlay layers for multiple projects - -Projects typically depend on multiple external repositories. Creating an overlay -layer for each external repository might unnecessarily clutter up the repository -configuration and the file structure of your repository. One solution to -mitigate this issue is to combine the ~TARGETS~ files of multiple external -repositories in a single overlay layer. To avoid conflicts, the ~TARGETS~ files -can be assigned different file names per repository. As an example, imagine a -common overlay layer with the files ~TARGETS.fmt~ and ~TARGETS.gsl~ for the -repositories ~"fmtlib"~ and ~"gsl-lite"~, respectively: - -#+BEGIN_SRC - common-layer - | - +--TARGETS.fmt - +--TARGETS.gsl - +--include - | +--fmt - | | +--TARGETS.fmt - | +--gsl - | +--TARGETS.gsl - | - +--src - +--TARGETS.fmt -#+END_SRC - -Such a common overlay layer can be used as the target root for both repositories -with only one difference: the ~"target_file_name"~ field. By specifying this -field, the dispatch where to find the respective target description for each -repository is implemented. For the given example, the following ~repos.json~ -defines the overlay ~"common-targets-layer"~, which is used by ~"fmtlib"~ and -~"gsl-lite"~: - -#+SRCNAME: repos.json -#+BEGIN_SRC js -{ "main": "tutorial" -, "repositories": - { "rules-cc": - { "repository": - { "type": "git" - , "branch": "master" - , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" - , "repository": "https://github.com/just-buildsystem/rules-cc.git" - , "subdir": "rules" - } - , "target_root": "tutorial-defaults" - , "rule_root": "rules-cc" - } - , "tutorial": - { "repository": {"type": "file", "path": "."} - , "bindings": {"rules": "rules-cc", "format": "fmtlib"} - } - , "tutorial-defaults": - { "repository": - { "type": "file" - , "path": "./tutorial-defaults" - , "pragma": {"to_git": true} - } - } - , "common-targets-layer": - { "repository": - { "type": "file" - , "path": "./common-layer" - , "pragma": {"to_git": true} - } - } - , "fmtlib": - { "repository": - { "type": "git" - , "branch": "8.1.1" - , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9" - , "repository": "https://github.com/fmtlib/fmt.git" - } - , "target_root": "common-targets-layer" - , "target_file_name": "TARGETS.fmt" - , "bindings": {"rules": "rules-cc"} - } - , "gsl-lite": - { "repository": - { "type": "git" - , "branch": "v0.40.0" - , "commit": "d6c8af99a1d95b3db36f26b4f22dc3bad89952de" - , "repository": "https://github.com/gsl-lite/gsl-lite.git" - } - , "target_root": "common-targets-layer" - , "target_file_name": "TARGETS.gsl" - , "bindings": {"rules": "rules-cc"} - } - } -} -#+END_SRC - -** Using pre-built dependencies - -While building external dependencies from source brings advantages, -most prominently the flexibility to quickly and seamlessly switch -to a different build configuration (production, debug, instrumented -for performance analysis; cross-compiling for a different target -architecture), there are also legitimate reasons to use pre-built -dependencies. The most prominent one is if your project is packaged -as part of a larger distribution. For that reason, just also has (in -~etc/import.prebuilt~) target files for all its dependencies assuming -they are pre-installed. The reason why target files are used at -all for this situation is twofold. -- On the one hand, having a target allows the remaining targets - to not care about where their dependencies come from, or if it - is a build against pre-installed dependencies or not. Also, the - top-level binary does not have to know the linking requirements - of its transitive dependencies. In other words, information stays - where it belongs to and if one target acquires a new dependency, - the information is automatically propagated to all targets using it. -- Still some information is needed to use a pre-installed library - and, as explained, a target describing the pre-installed library - is the right place to collect this information. - - The public header files of the library. By having this explicit, - we do not accumulate directories in the include search path - and hence also properly detect include conflicts. - - The information on how to link the library itself (i.e., - basically its base name). - - Any dependencies on other libraries that the library might have. - This information is used to obtain the correct linking order - and complete transitive linking arguments while keeping the - description maintainable, as each target still only declares - its direct dependencies. - -The target description for a pre-built version of the format -library that was used as an example in this section is shown next; -with our staging mechanism the logical repository it belongs to is -rooted in the ~fmt~ subdirectory of the ~include~ directory of the -ambient system. - -#+SRCNAME: etc/import.prebuilt/TARGETS.fmt -#+BEGIN_SRC js -{ "fmt": - { "type": ["@", "rules", "CC", "library"] - , "name": ["fmt"] - , "stage": ["fmt"] - , "hdrs": [["TREE", null, "."]] - , "private-ldflags": ["-lfmt"] - } -} -#+END_SRC -- cgit v1.2.3