diff options
author | Oliver Reiche <oliver.reiche@huawei.com> | 2023-06-01 13:36:32 +0200 |
---|---|---|
committer | Oliver Reiche <oliver.reiche@huawei.com> | 2023-06-12 16:29:05 +0200 |
commit | b66a7359fbbff35af630c88c56598bbc06b393e1 (patch) | |
tree | d866802c4b44c13cbd90f9919cc7fc472091be0c /doc/tutorial | |
parent | 144b2c619f28c91663936cd445251ca28af45f88 (diff) | |
download | justbuild-b66a7359fbbff35af630c88c56598bbc06b393e1.tar.gz |
doc: Convert orgmode files to markdown
Diffstat (limited to 'doc/tutorial')
-rw-r--r-- | doc/tutorial/getting-started.md | 217 | ||||
-rw-r--r-- | doc/tutorial/getting-started.org | 212 | ||||
-rw-r--r-- | doc/tutorial/hello-world.md | 379 | ||||
-rw-r--r-- | doc/tutorial/hello-world.org | 370 | ||||
-rw-r--r-- | doc/tutorial/proto.md (renamed from doc/tutorial/proto.org) | 245 | ||||
-rw-r--r-- | doc/tutorial/rebuild.md (renamed from doc/tutorial/rebuild.org) | 164 | ||||
-rw-r--r-- | doc/tutorial/target-file-glob-tree.md (renamed from doc/tutorial/target-file-glob-tree.org) | 326 | ||||
-rw-r--r-- | doc/tutorial/tests.md (renamed from doc/tutorial/tests.org) | 270 | ||||
-rw-r--r-- | doc/tutorial/third-party-software.md | 473 | ||||
-rw-r--r-- | doc/tutorial/third-party-software.org | 475 |
10 files changed, 1573 insertions, 1558 deletions
diff --git a/doc/tutorial/getting-started.md b/doc/tutorial/getting-started.md new file mode 100644 index 00000000..36a57d26 --- /dev/null +++ b/doc/tutorial/getting-started.md @@ -0,0 +1,217 @@ +Getting Started +=============== + +In order to use *justbuild*, first make sure that `just`, `just-mr`, and +`just-import-git` are available in your `PATH`. + +Creating a new project +---------------------- + +*justbuild* needs to know the root of the project worked on. By default, +it searches upwards from the current directory till it finds a marker. +Currently, we support three different markers: the files `ROOT` and +`WORKSPACE` or the directory `.git`. Lets create a new project by +creating one of those markers: + +``` sh +$ touch ROOT +``` + +Creating a generic target +------------------------- + +By default, targets are described in `TARGETS` files. These files +contain a `JSON` object with the target name as key and the target +description as value. A target description is an object with at least a +single mandatory field: `"type"`. This field specifies which rule +(built-in or user-defined) to apply for this target. + +A simple target that only executes commands can be created using the +built-in `"generic"` rule, which requires at least one command and one +output file or directory. To create such a target, create the file +`TARGETS` with the following content: + +``` {.jsonc srcname="TARGETS"} +{ "greeter": + { "type": "generic" + , "cmds": ["echo -n 'Hello ' > out.txt", "cat name.txt >> out.txt"] + , "outs": ["out.txt"] + , "deps": ["name.txt"] + } +} +``` + +In this example, the `"greeter"` target will run two commands to produce +the output file `out.txt`. The second command depends on the input file +`name.txt` that we need to create as well: + +``` sh +$ echo World > name.txt +``` + +Building a generic target +------------------------- + +To build a target, we need to run `just` with the subcommand `build`: + +``` sh +$ just build greeter +INFO: Requested target is [["@","","","greeter"],{}] +INFO: Analysed target [["@","","","greeter"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 1 actions, 0 trees, 0 blobs +INFO: Building [["@","","","greeter"],{}]. +INFO: Processed 1 actions, 0 cache hits. +INFO: Artifacts built, logical paths are: + out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] +$ +``` + +The subcommand `build` just builds the artifact but does not stage it to +any user-defined location on the file system. Instead it reports a +description of the artifact consisting of `git` blob identifier, size, +and type (in this case `f` for non-executable file). To also stage the +produced artifact to the working directory, use the `install` subcommand +and specify the output directory: + +``` sh +$ just install greeter -o . +INFO: Requested target is [["@","","","greeter"],{}] +INFO: Analysed target [["@","","","greeter"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 1 actions, 0 trees, 0 blobs +INFO: Building [["@","","","greeter"],{}]. +INFO: Processed 1 actions, 1 cache hits. +INFO: Artifacts can be found in: + /tmp/tutorial/out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] +$ cat out.txt +Hello World +$ +``` + +Note that the `install` subcommand initiates the build a second time, +without executing any actions as all actions are being served from +cache. The produced artifact is identical, which is indicated by the +same hash/size/type. + +If one is only interested in a single final artifact, one can also +request via the `-P` option that this artifact be written to standard +output after the build. As all messages are reported to standard error, +this can be used for both, interactively reading a text file, as well as +for piping the artifact to another program. + +``` sh +$ just build greeter -Pout.txt +INFO: Requested target is [["@","","","greeter"],{}] +INFO: Analysed target [["@","","","greeter"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 1 actions, 0 trees, 0 blobs +INFO: Building [["@","","","greeter"],{}]. +INFO: Processed 1 actions, 1 cache hits. +INFO: Artifacts built, logical paths are: + out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] +Hello World +$ +``` + +Alternatively, we could also directly request the artifact `out.txt` +from *justbuild*'s CAS (content-addressable storage) and print it on +the command line via: + +``` sh +$ just install-cas [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] +Hello World +$ +``` + +The canonical way of requesting an object from the CAS is, as just +shown, to specify the full triple of hash, size, and type, separated by +colons and enclosed in square brackets. To simplify usage, the brackets +can be omitted and the size and type fields have the default values `0` +and `f`, respectively. While the default value for the size is wrong for +all but one string, the hash still determines the content of the file +and hence the local CAS is still able to retrieve the file. So the +typical invocation would simply specify the hash. + +``` sh +$ just install-cas 557db03de997c86a4a028e1ebd3a1ceb225be238 +Hello World +$ +``` + +Targets versus Files: The Stage +------------------------------- + +When invoking the `build` command, we had to specify the target +`greeter`, not the output file `out.txt`. While other build systems +allow requests specifying an output file, for *justbuild* this would +conflict with a fundamental design principle: staging; each target has +its own logical output space, the "stage", where it can put its +artifacts. We can, without any problem, add a second target also +generating a file `out.txt`. + +``` {.jsonc srcname="TARGETS"} +... +, "upper": + { "type": "generic" + , "cmds": ["cat name.txt | tr a-z A-Z > out.txt"] + , "outs": ["out.txt"] + , "deps": ["name.txt"] + } +... +``` + +As we only request targets, no conflicts arise. + +``` sh +$ just build upper -P out.txt +INFO: Requested target is [["@","","","upper"],{}] +INFO: Analysed target [["@","","","upper"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 1 actions, 0 trees, 0 blobs +INFO: Building [["@","","","upper"],{}]. +INFO: Processed 1 actions, 0 cache hits. +INFO: Artifacts built, logical paths are: + out.txt [83cf24cdfb4891a36bee93421930dd220766299a:6:f] +WORLD +$ just build greeter -P out.txt +INFO: Requested target is [["@","","","greeter"],{}] +INFO: Analysed target [["@","","","greeter"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 1 actions, 0 trees, 0 blobs +INFO: Building [["@","","","greeter"],{}]. +INFO: Processed 1 actions, 1 cache hits. +INFO: Artifacts built, logical paths are: + out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] +Hello World +$ +``` + +While one normally tries to design targets in such a way that they +don't have conflicting files if they should be used together, it is up +to the receiving target to decide what to do with those artifacts. A +built-in rule allowing to rearrange artifacts is `"install"`; a detailed +description of this rule can be found in the documentation. In the +simple case of a target producing precisely one file, the argument +`"files"` can be used to map that file to a new location. + +``` {.jsonc srcname="TARGETS"} +... +, "both": + {"type": "install", "files": {"hello.txt": "greeter", "upper.txt": "upper"}} +... +``` + +``` sh +$ just build both +INFO: Requested target is [["@","","","both"],{}] +INFO: Analysed target [["@","","","both"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 2 actions, 0 trees, 0 blobs +INFO: Building [["@","","","both"],{}]. +INFO: Processed 2 actions, 2 cache hits. +INFO: Artifacts built, logical paths are: + hello.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] + upper.txt [83cf24cdfb4891a36bee93421930dd220766299a:6:f] +$ +``` diff --git a/doc/tutorial/getting-started.org b/doc/tutorial/getting-started.org deleted file mode 100644 index 5a041397..00000000 --- a/doc/tutorial/getting-started.org +++ /dev/null @@ -1,212 +0,0 @@ -* Getting Started - -In order to use /justbuild/, first make sure that ~just~, ~just-mr~, and -~just-import-git~ are available in your ~PATH~. - -** Creating a new project - -/justbuild/ needs to know the root of the project worked on. By default, it -searches upwards from the current directory till it finds a marker. Currently, -we support three different markers: the files ~ROOT~ and ~WORKSPACE~ or the -directory ~.git~. Lets create a new project by creating one of those markers: - -#+BEGIN_SRC sh -$ touch ROOT -#+END_SRC - -** Creating a generic target - -By default, targets are described in ~TARGETS~ files. These files contain a -~JSON~ object with the target name as key and the target description as value. A -target description is an object with at least a single mandatory field: -~"type"~. This field specifies which rule (built-in or user-defined) to apply -for this target. - -A simple target that only executes commands can be created using the built-in -~"generic"~ rule, which requires at least one command and one output file or -directory. To create such a target, create the file ~TARGETS~ with the following -content: - -#+SRCNAME: TARGETS -#+BEGIN_SRC js -{ "greeter": - { "type": "generic" - , "cmds": ["echo -n 'Hello ' > out.txt", "cat name.txt >> out.txt"] - , "outs": ["out.txt"] - , "deps": ["name.txt"] - } -} -#+END_SRC - -In this example, the ~"greeter"~ target will run two commands to produce the -output file ~out.txt~. The second command depends on the input file ~name.txt~ -that we need to create as well: - -#+BEGIN_SRC sh -$ echo World > name.txt -#+END_SRC - -** Building a generic target - -To build a target, we need to run ~just~ with the subcommand ~build~: - -#+BEGIN_SRC sh -$ just build greeter -INFO: Requested target is [["@","","","greeter"],{}] -INFO: Analysed target [["@","","","greeter"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 1 actions, 0 trees, 0 blobs -INFO: Building [["@","","","greeter"],{}]. -INFO: Processed 1 actions, 0 cache hits. -INFO: Artifacts built, logical paths are: - out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] -$ -#+END_SRC - -The subcommand ~build~ just builds the artifact but does not stage it to any -user-defined location on the file system. Instead it reports a description -of the artifact consisting of ~git~ blob identifier, size, and type (in -this case ~f~ for non-executable file). To also stage the produced artifact to -the working directory, use the ~install~ subcommand and specify the output -directory: - -#+BEGIN_SRC sh -$ just install greeter -o . -INFO: Requested target is [["@","","","greeter"],{}] -INFO: Analysed target [["@","","","greeter"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 1 actions, 0 trees, 0 blobs -INFO: Building [["@","","","greeter"],{}]. -INFO: Processed 1 actions, 1 cache hits. -INFO: Artifacts can be found in: - /tmp/tutorial/out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] -$ cat out.txt -Hello World -$ -#+END_SRC - -Note that the ~install~ subcommand initiates the build a second time, without -executing any actions as all actions are being served from cache. The produced -artifact is identical, which is indicated by the same hash/size/type. - -If one is only interested in a single final artifact, one can -also request via the ~-P~ option that this artifact be written to -standard output after the build. As all messages are reported to -standard error, this can be used for both, interactively reading a -text file, as well as for piping the artifact to another program. - -#+BEGIN_SRC sh -$ just build greeter -Pout.txt -INFO: Requested target is [["@","","","greeter"],{}] -INFO: Analysed target [["@","","","greeter"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 1 actions, 0 trees, 0 blobs -INFO: Building [["@","","","greeter"],{}]. -INFO: Processed 1 actions, 1 cache hits. -INFO: Artifacts built, logical paths are: - out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] -Hello World -$ -#+END_SRC - -Alternatively, we could also directly request the artifact ~out.txt~ from -/justbuild/'s CAS (content-addressable storage) and print it on the command line -via: - -#+BEGIN_SRC sh -$ just install-cas [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] -Hello World -$ -#+END_SRC - -The canonical way of requesting an object from the CAS is, as just shown, to -specify the full triple of hash, size, and type, separated by colons and -enclosed in square brackets. To simplify usage, the brackets can be omitted -and the size and type fields have the default values ~0~ and ~f~, respectively. -While the default value for the size is wrong for all but one string, the hash -still determines the content of the file and hence the local CAS is still -able to retrieve the file. So the typical invocation would simply specify the -hash. - -#+BEGIN_SRC sh -$ just install-cas 557db03de997c86a4a028e1ebd3a1ceb225be238 -Hello World -$ -#+END_SRC - -** Targets versus Files: The Stage - -When invoking the ~build~ command, we had to specify the target ~greeter~, -not the output file ~out.txt~. While other build systems allow requests -specifying an output file, for /justbuild/ this would conflict with a -fundamental design principle: staging; each target has its own logical -output space, the "stage", where it can put its artifacts. We can, without -any problem, add a second target also generating a file ~out.txt~. - -#+SRCNAME: TARGETS -#+BEGIN_SRC js -... -, "upper": - { "type": "generic" - , "cmds": ["cat name.txt | tr a-z A-Z > out.txt"] - , "outs": ["out.txt"] - , "deps": ["name.txt"] - } -... -#+END_SRC - -As we only request targets, no conflicts arise. - -#+BEGIN_SRC sh -$ just build upper -P out.txt -INFO: Requested target is [["@","","","upper"],{}] -INFO: Analysed target [["@","","","upper"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 1 actions, 0 trees, 0 blobs -INFO: Building [["@","","","upper"],{}]. -INFO: Processed 1 actions, 0 cache hits. -INFO: Artifacts built, logical paths are: - out.txt [83cf24cdfb4891a36bee93421930dd220766299a:6:f] -WORLD -$ just build greeter -P out.txt -INFO: Requested target is [["@","","","greeter"],{}] -INFO: Analysed target [["@","","","greeter"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 1 actions, 0 trees, 0 blobs -INFO: Building [["@","","","greeter"],{}]. -INFO: Processed 1 actions, 1 cache hits. -INFO: Artifacts built, logical paths are: - out.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] -Hello World -$ -#+END_SRC - -While one normally tries to design targets in such a way that they -don't have conflicting files if they should be used together, it is -up to the receiving target to decide what to do with those artifacts. -A built-in rule allowing to rearrange artifacts is ~"install"~; a -detailed description of this rule can be found in the documentation. -In the simple case of a target producing precisely one file, the -argument ~"files"~ can be used to map that file to a new location. - -#+SRCNAME: TARGETS -#+BEGIN_SRC js -... -, "both": - {"type": "install", "files": {"hello.txt": "greeter", "upper.txt": "upper"}} -... -#+END_SRC - -#+BEGIN_SRC sh -$ just build both -INFO: Requested target is [["@","","","both"],{}] -INFO: Analysed target [["@","","","both"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 2 actions, 0 trees, 0 blobs -INFO: Building [["@","","","both"],{}]. -INFO: Processed 2 actions, 2 cache hits. -INFO: Artifacts built, logical paths are: - hello.txt [557db03de997c86a4a028e1ebd3a1ceb225be238:12:f] - upper.txt [83cf24cdfb4891a36bee93421930dd220766299a:6:f] -$ -#+END_SRC diff --git a/doc/tutorial/hello-world.md b/doc/tutorial/hello-world.md new file mode 100644 index 00000000..9af68f07 --- /dev/null +++ b/doc/tutorial/hello-world.md @@ -0,0 +1,379 @@ +Building C++ Hello World +======================== + +*justbuild* is a true language-agnostic (there are no more-equal +languages) and multi-repository build system. As a consequence, +high-level concepts (e.g., C++ binaries, C++ libraries, etc.) are not +hardcoded built-ins of the tool, but rather provided via a set of rules. +These rules can be specified as a true dependency to your project like +any other external repository your project might depend on. + +Setting up the Multi-Repository Configuration +--------------------------------------------- + +To build a project with multi-repository dependencies, we first need to +provide a configuration that declares the required repositories. Before +we begin, we need to declare where the root of our workspace is located +by creating an empty file `ROOT`: + +``` sh +$ touch ROOT +``` + +Second, we also need to create the multi-repository configuration +`repos.json` in the workspace root: + +``` {.jsonc srcname="repos.json"} +{ "main": "tutorial" +, "repositories": + { "rules-cc": + { "repository": + { "type": "git" + , "branch": "master" + , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" + , "repository": "https://github.com/just-buildsystem/rules-cc.git" + , "subdir": "rules" + } + } + , "tutorial": + { "repository": {"type": "file", "path": "."} + , "bindings": {"rules": "rules-cc"} + } + } +} +``` + +In that configuration, two repositories are defined: + +1. The `"rules-cc"` repository located in the subdirectory `rules` of + [just-buildsystem/rules-cc:123d8b03bf2440052626151c14c54abce2726e6f](https://github.com/just-buildsystem/rules-cc/tree/123d8b03bf2440052626151c14c54abce2726e6f), + which contains the high-level concepts for building C/C++ binaries + and libraries. + +2. The `"tutorial"` repository located at `.`, which contains the + targets that we want to build. It has a single dependency, which is + the *rules* that are needed to build the target. These rules are + bound via the open name `"rules"` to the just created repository + `"rules-cc"`. In this way, the entities provided by `"rules-cc"` can + be accessed from within the `"tutorial"` repository via the + fully-qualified name `["@", "rules", "<module>", "<name>"]`; + fully-qualified names (for rules, targets to build (like libraries, + binaries), etc) are given by a repository name, a path specifying a + directory within that repository (the "module") where the + specification file is located, and a symbolic name (i.e., an + arbitrary string that is used as key in the specification). + +The final repository configuration contains a single `JSON` object with +the key `"repositories"` referring to an object of repository names as +keys and repository descriptions as values. For convenience, the main +repository to pick is set to `"tutorial"`. + +Description of the helloworld target +------------------------------------ + +For this tutorial, we want to create a target `helloworld` that produces +a binary from the C++ source `main.cpp`. To define such a target, create +a `TARGETS` file with the following content: + +``` {.jsonc srcname="TARGETS"} +{ "helloworld": + { "type": ["@", "rules", "CC", "binary"] + , "name": ["helloworld"] + , "srcs": ["main.cpp"] + } +} +``` + +The `"type"` field refers to the rule `"binary"` from the module `"CC"` +of the `"rules"` repository. This rule additionally requires the string +field `"name"`, which specifies the name of the binary to produce; as +the generic interface of rules is to have fields either take a list of +strings or a list of targets, we have to specify the name as a list +(this rule will simply concatenate all strings given in this field). +Furthermore, at least one input to the binary is required, which can be +specified via the target fields `"srcs"` or `"deps"`. In our case, the +former is used, which contains our single source file (files are +considered targets). + +Now, the last file that is missing is the actual source file `main.cpp`: + +``` {.cpp srcname="main.cpp"} +#include <iostream> + +int main() { + std::cout << "Hello world!\n"; + return 0; +} +``` + +Building the helloworld target +------------------------------ + +To build the `helloworld` target, we need specify it on the `just-mr` +command line: + +``` sh +$ just-mr build helloworld +INFO: Requested target is [["@","tutorial","","helloworld"],{}] +INFO: Analysed target [["@","tutorial","",helloworld"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 2 actions, 1 trees, 0 blobs +INFO: Building [["@","helloworld","","helloworld"],{}]. +INFO: Processed 2 actions, 0 cache hits. +INFO: Artifacts built, logical paths are: + helloworld [b5cfca8b810adc4686f5cac00258a137c5d4a3ba:17088:x] +$ +``` + +Note that the target is taken from the `tutorial` repository, as it +specified as the main repository in `repos.json`. If targets from other +repositories should be build, the repository to use must be specified +via the `--main` option. + +`just-mr` reads the repository configuration, fetches externals (if +any), generates the actual build configuration, and stores it in its +cache directory (by default under `$HOME/.cache/just`). Afterwards, the +generated configuration is used to call the `just` binary, which +performs the actual build. + +Note that these two programs, `just-mr` and `just`, can also be run +individually. To do so, first run `just-mr` with `setup` and capture the +path to the generated build configuration from stdout by assigning it to +a shell variable (e.g., `CONF`). Afterwards, `just` can be called to +perform the actual build by explicitly specifying the configuration file +via `-C`: + +``` sh +$ CONF=$(just-mr setup tutorial) +$ just build -C $CONF helloworld +``` + +Note that `just-mr` only needs to be run the very first time and only +once again whenever the `repos.json` file is modified. + +By default, the BSD-default compiler front-ends (which are also defined +for most Linux distributions) `cc` and `c++` are used for C and C++ +(variables `"CC"` and `"CXX"`). If you want to temporarily use different +defaults, you can use `-D` to provide a JSON object that sets different +default variables. For instance, to use Clang as C++ compiler for a +single build invocation, you can use the following command to provide an +object that sets `"CXX"` to `"clang++"`: + +``` sh +$ just-mr build helloworld -D'{"CXX":"clang++"}' +INFO: Requested target is [["@","tutorial","","helloworld"],{"CXX":"clang++"}] +INFO: Analysed target [["@","tutorial","","helloworld"],{"CXX":"clang++"}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 2 actions, 1 trees, 0 blobs +INFO: Building [["@","tutorial","","helloworld"],{"CXX":"clang++"}]. +INFO: Processed 2 actions, 0 cache hits. +INFO: Artifacts built, logical paths are: + helloworld [b8cf7b8579d9dc7172b61660139e2c14521cedae:16944:x] +$ +``` + +Defining project defaults +------------------------- + +To define a custom set of defaults (toolchain and compile flags) for +your project, you need to create a separate file root for providing +required `TARGETS` file, which contains the `"defaults"` target that +should be used by the rules. This file root is then used as the *target +root* for the rules, i.e., the search path for `TARGETS` files. In this +way, the description of the `"defaults"` target is provided in a +separate file root, to keep the rules repository independent of these +definitions. + +We will call the new file root `tutorial-defaults` and need to create a +module directory `CC` in it: + +``` sh +$ mkdir -p ./tutorial-defaults/CC +``` + +In that module, we need to create the file +`tutorial-defaults/CC/TARGETS` that contains the target `"defaults"` and +specifies which toolchain and compile flags to use; it has to specify +the complete toolchain, but can specify a `"base"` toolchain to inherit +from. In our case, we don't use any base, but specify all the required +fields directly. + +``` {.jsonc srcname="tutorial-defaults/CC/TARGETS"} +{ "defaults": + { "type": ["CC", "defaults"] + , "CC": ["cc"] + , "CXX": ["c++"] + , "CFLAGS": ["-O2", "-Wall"] + , "CXXFLAGS": ["-O2", "-Wall"] + , "AR": ["ar"] + , "PATH": ["/bin", "/usr/bin"] + } +} +``` + +To use the project defaults, modify the existing `repos.json` to reflect +the following content: + +``` {.jsonc srcname="repos.json"} +{ "main": "tutorial" +, "repositories": + { "rules-cc": + { "repository": + { "type": "git" + , "branch": "master" + , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" + , "repository": "https://github.com/just-buildsystem/rules-cc.git" + , "subdir": "rules" + } + , "target_root": "tutorial-defaults" + , "rule_root": "rules-cc" + } + , "tutorial": + { "repository": {"type": "file", "path": "."} + , "bindings": {"rules": "rules-cc"} + } + , "tutorial-defaults": + { "repository": {"type": "file", "path": "./tutorial-defaults"} + } + } +} +``` + +Note that the `"defaults"` target uses the rule `["CC", "defaults"]` +without specifying any external repository (e.g., +`["@", "rules", ...]`). This is because `"tutorial-defaults"` is not a +full-fledged repository but merely a file root that is considered local +to the `"rules-cc"` repository. In fact, the `"rules-cc"` repository +cannot refer to any external repository as it does not have any defined +bindings. + +To rebuild the project, we need to rerun `just-mr` (note that due to +configuration changes, rerunning only `just` would not suffice): + +``` sh +$ just-mr build helloworld +INFO: Requested target is [["@","tutorial","","helloworld"],{}] +INFO: Analysed target [["@","tutorial","","helloworld"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 2 actions, 1 trees, 0 blobs +INFO: Building [["@","tutorial","","helloworld"],{}]. +INFO: Processed 2 actions, 0 cache hits. +INFO: Artifacts built, logical paths are: + helloworld [487dc9e47b978877ed2f7d80b3395ce84b23be92:16992:x] +$ +``` + +Note that the output binary may have changed due to different defaults. + +Modeling target dependencies +---------------------------- + +For demonstration purposes, we will separate the print statements into a +static library `greet`, which will become a dependency to our binary. +Therefore, we create a new subdirectory `greet` with the files +`greet/greet.hpp`: + +``` {.cpp srcname="greet/greet.hpp"} +#include <string> + +void greet(std::string const& s); +``` + +and `greet/greet.cpp`: + +``` {.cpp srcname="greet/greet.cpp"} +#include "greet.hpp" +#include <iostream> + +void greet(std::string const& s) { + std::cout << "Hello " << s << "!\n"; +} +``` + +These files can now be used to create a static library `libgreet.a`. To +do so, we need to create the following target description in +`greet/TARGETS`: + +``` {.jsonc srcname="greet/TARGETS"} +{ "greet": + { "type": ["@", "rules", "CC", "library"] + , "name": ["greet"] + , "hdrs": ["greet.hpp"] + , "srcs": ["greet.cpp"] + , "stage": ["greet"] + } +} +``` + +Similar to `"binary"`, we have to provide a name and source file. +Additionally, a library has public headers defined via `"hdrs"` and an +optional staging directory `"stage"` (default value `"."`). The staging +directory specifies where the consumer of this library can expect to +find the library's artifacts. Note that this does not need to reflect +the location on the file system (i.e., a full-qualified path like +`["com", "example", "utils", "greet"]` could be used to distinguish it +from greeting libraries of other projects). The staging directory does +not only affect the main artifact `libgreet.a` but also it's +*runfiles*, a second set of artifacts, usually those a consumer needs to +make proper use the actual artifact; in the case of a library, the +runfiles are its public headers. Hence, the public header will be staged +to `"greet/greet.hpp"`. With that knowledge, we can now perform the +necessary modifications to `main.cpp`: + +``` {.cpp srcname="main.cpp"} +#include "greet/greet.hpp" + +int main() { + greet("Universe"); + return 0; +} +``` + +The target `"helloworld"` will have a direct dependency to the target +`"greet"` of the module `"greet"` in the top-level `TARGETS` file: + +``` {.jsonc srcname="TARGETS"} +{ "helloworld": + { "type": ["@", "rules", "CC", "binary"] + , "name": ["helloworld"] + , "srcs": ["main.cpp"] + , "private-deps": [["greet", "greet"]] + } +} +``` + +Note that there is no need to explicitly specify `"greet"`'s public +headers here as the appropriate artifacts of dependencies are +automatically added to the inputs of compile and link actions. The new +binary can be built with the same command as before (no need to rerun +`just-mr`): + +``` sh +$ just-mr build helloworld +INFO: Requested target is [["@","tutorial","","helloworld"],{}] +INFO: Analysed target [["@","tutorial","","helloworld"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 4 actions, 2 trees, 0 blobs +INFO: Building [["@","tutorial","","helloworld"],{}]. +INFO: Processed 4 actions, 0 cache hits. +INFO: Artifacts built, logical paths are: + helloworld [2b81e3177afc382452a2df9f294d3df90a9ccaf0:17664:x] +$ +``` + +To only build the static library target `"greet"` from module `"greet"`, +run the following command: + +``` sh +$ just-mr build greet greet +INFO: Requested target is [["@","tutorial","greet","greet"],{}] +INFO: Analysed target [["@","tutorial","greet","greet"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 2 actions, 1 trees, 0 blobs +INFO: Building [["@","tutorial","greet","greet"],{}]. +INFO: Processed 2 actions, 2 cache hits. +INFO: Artifacts built, logical paths are: + greet/libgreet.a [83ed406e21f285337b0c9bd5011f56f656bba683:2992:f] + (1 runfiles omitted.) +$ +``` diff --git a/doc/tutorial/hello-world.org b/doc/tutorial/hello-world.org deleted file mode 100644 index 342eaf82..00000000 --- a/doc/tutorial/hello-world.org +++ /dev/null @@ -1,370 +0,0 @@ -* Building C++ Hello World - -/justbuild/ is a true language-agnostic (there are no more-equal languages) and -multi-repository build system. As a consequence, high-level concepts (e.g., C++ -binaries, C++ libraries, etc.) are not hardcoded built-ins of the tool, but -rather provided via a set of rules. These rules can be specified as a true -dependency to your project like any other external repository your project might -depend on. - -** Setting up the Multi-Repository Configuration - -To build a project with multi-repository dependencies, we first need to provide -a configuration that declares the required repositories. Before we begin, we -need to declare where the root of our workspace is located by creating an empty -file ~ROOT~: - -#+BEGIN_SRC sh -$ touch ROOT -#+END_SRC - -Second, we also need to create the multi-repository configuration ~repos.json~ -in the workspace root: - -#+SRCNAME: repos.json -#+BEGIN_SRC js -{ "main": "tutorial" -, "repositories": - { "rules-cc": - { "repository": - { "type": "git" - , "branch": "master" - , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" - , "repository": "https://github.com/just-buildsystem/rules-cc.git" - , "subdir": "rules" - } - } - , "tutorial": - { "repository": {"type": "file", "path": "."} - , "bindings": {"rules": "rules-cc"} - } - } -} -#+END_SRC - -In that configuration, two repositories are defined: - - 1. The ~"rules-cc"~ repository located in the subdirectory ~rules~ of - [[https://github.com/just-buildsystem/rules-cc/tree/123d8b03bf2440052626151c14c54abce2726e6f][just-buildsystem/rules-cc:123d8b03bf2440052626151c14c54abce2726e6f]], - which contains the high-level concepts for building C/C++ binaries and - libraries. - - 2. The ~"tutorial"~ repository located at ~.~, which contains the targets that - we want to build. It has a single dependency, which is the /rules/ that are - needed to build the target. These rules are bound via the open name - ~"rules"~ to the just created repository ~"rules-cc"~. In this way, the - entities provided by ~"rules-cc"~ can be accessed from within the - ~"tutorial"~ repository via the fully-qualified name - ~["@", "rules", "<module>", "<name>"]~; fully-qualified - names (for rules, targets to build (like libraries, binaries), - etc) are given by a repository name, a path specifying a - directory within that repository (the "module") where the - specification file is located, and a symbolic name (i.e., an - arbitrary string that is used as key in the specification). - -The final repository configuration contains a single ~JSON~ object with the key -~"repositories"~ referring to an object of repository names as keys and -repository descriptions as values. For convenience, the main repository to pick -is set to ~"tutorial"~. - -** Description of the helloworld target - -For this tutorial, we want to create a target ~helloworld~ that produces a -binary from the C++ source ~main.cpp~. To define such a target, create a -~TARGETS~ file with the following content: - -#+SRCNAME: TARGETS -#+BEGIN_SRC js -{ "helloworld": - { "type": ["@", "rules", "CC", "binary"] - , "name": ["helloworld"] - , "srcs": ["main.cpp"] - } -} -#+END_SRC - -The ~"type"~ field refers to the rule ~"binary"~ from the module ~"CC"~ of the -~"rules"~ repository. This rule additionally requires the string field ~"name"~, -which specifies the name of the binary to produce; as the generic interface of -rules is to have fields either take a list of strings or a list of targets, -we have to specify the name as a list (this rule will simply concatenate all -strings given in this field). Furthermore, at least one -input to the binary is required, which can be specified via the target fields -~"srcs"~ or ~"deps"~. In our case, the former is used, which contains our single -source file (files are considered targets). - -Now, the last file that is missing is the actual source file ~main.cpp~: - -#+SRCNAME: main.cpp -#+BEGIN_SRC cpp -#include <iostream> - -int main() { - std::cout << "Hello world!\n"; - return 0; -} -#+END_SRC - -** Building the helloworld target - -To build the ~helloworld~ target, we need specify it on the ~just-mr~ command -line: - -#+BEGIN_SRC sh -$ just-mr build helloworld -INFO: Requested target is [["@","tutorial","","helloworld"],{}] -INFO: Analysed target [["@","tutorial","",helloworld"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 2 actions, 1 trees, 0 blobs -INFO: Building [["@","helloworld","","helloworld"],{}]. -INFO: Processed 2 actions, 0 cache hits. -INFO: Artifacts built, logical paths are: - helloworld [b5cfca8b810adc4686f5cac00258a137c5d4a3ba:17088:x] -$ -#+END_SRC - -Note that the target is taken from the ~tutorial~ repository, as it specified as -the main repository in ~repos.json~. If targets from other repositories should -be build, the repository to use must be specified via the ~--main~ option. - -~just-mr~ reads the repository configuration, fetches externals (if any), -generates the actual build configuration, and stores it in its cache directory -(by default under ~$HOME/.cache/just~). Afterwards, the generated configuration -is used to call the ~just~ binary, which performs the actual build. - -Note that these two programs, ~just-mr~ and ~just~, can also be run -individually. To do so, first run ~just-mr~ with ~setup~ and capture the path to -the generated build configuration from stdout by assigning it to a shell -variable (e.g., ~CONF~). Afterwards, ~just~ can be called to perform the actual -build by explicitly specifying the configuration file via ~-C~: - -#+BEGIN_SRC sh -$ CONF=$(just-mr setup tutorial) -$ just build -C $CONF helloworld -#+END_SRC - -Note that ~just-mr~ only needs to be run the very first time and only once again -whenever the ~repos.json~ file is modified. - -By default, the BSD-default compiler front-ends (which are also defined for most -Linux distributions) ~cc~ and ~c++~ are used for C and C++ (variables ~"CC"~ and -~"CXX"~). If you want to temporarily use different defaults, you can use ~-D~ to -provide a JSON object that sets different default variables. For instance, to -use Clang as C++ compiler for a single build invocation, you can use the -following command to provide an object that sets ~"CXX"~ to ~"clang++"~: - -#+BEGIN_SRC sh -$ just-mr build helloworld -D'{"CXX":"clang++"}' -INFO: Requested target is [["@","tutorial","","helloworld"],{"CXX":"clang++"}] -INFO: Analysed target [["@","tutorial","","helloworld"],{"CXX":"clang++"}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 2 actions, 1 trees, 0 blobs -INFO: Building [["@","tutorial","","helloworld"],{"CXX":"clang++"}]. -INFO: Processed 2 actions, 0 cache hits. -INFO: Artifacts built, logical paths are: - helloworld [b8cf7b8579d9dc7172b61660139e2c14521cedae:16944:x] -$ -#+END_SRC - -** Defining project defaults - -To define a custom set of defaults (toolchain and compile flags) for your -project, you need to create a separate file root for providing required -~TARGETS~ file, which contains the ~"defaults"~ target that should be used by -the rules. This file root is then used as the /target root/ for the rules, i.e., -the search path for ~TARGETS~ files. In this way, the description of the -~"defaults"~ target is provided in a separate file root, to keep the rules -repository independent of these definitions. - -We will call the new file root ~tutorial-defaults~ and need to create a module -directory ~CC~ in it: - -#+BEGIN_SRC sh -$ mkdir -p ./tutorial-defaults/CC -#+END_SRC - -In that module, we need to create the file ~tutorial-defaults/CC/TARGETS~ that -contains the target ~"defaults"~ and specifies which toolchain and compile flags -to use; it has to specify the complete toolchain, but can specify a ~"base"~ -toolchain to inherit from. In our case, we don't use any base, but specify all -the required fields directly. - -#+SRCNAME: tutorial-defaults/CC/TARGETS -#+BEGIN_SRC js -{ "defaults": - { "type": ["CC", "defaults"] - , "CC": ["cc"] - , "CXX": ["c++"] - , "CFLAGS": ["-O2", "-Wall"] - , "CXXFLAGS": ["-O2", "-Wall"] - , "AR": ["ar"] - , "PATH": ["/bin", "/usr/bin"] - } -} -#+END_SRC - -To use the project defaults, modify the existing ~repos.json~ to reflect the -following content: - -#+SRCNAME: repos.json -#+BEGIN_SRC js -{ "main": "tutorial" -, "repositories": - { "rules-cc": - { "repository": - { "type": "git" - , "branch": "master" - , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" - , "repository": "https://github.com/just-buildsystem/rules-cc.git" - , "subdir": "rules" - } - , "target_root": "tutorial-defaults" - , "rule_root": "rules-cc" - } - , "tutorial": - { "repository": {"type": "file", "path": "."} - , "bindings": {"rules": "rules-cc"} - } - , "tutorial-defaults": - { "repository": {"type": "file", "path": "./tutorial-defaults"} - } - } -} -#+END_SRC - -Note that the ~"defaults"~ target uses the rule ~["CC", "defaults"]~ without -specifying any external repository (e.g., ~["@", "rules", ...]~). This is -because ~"tutorial-defaults"~ is not a full-fledged repository but merely a file -root that is considered local to the ~"rules-cc"~ repository. In fact, the -~"rules-cc"~ repository cannot refer to any external repository as it does not -have any defined bindings. - -To rebuild the project, we need to rerun ~just-mr~ (note that due to -configuration changes, rerunning only ~just~ would not suffice): - -#+BEGIN_SRC sh -$ just-mr build helloworld -INFO: Requested target is [["@","tutorial","","helloworld"],{}] -INFO: Analysed target [["@","tutorial","","helloworld"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 2 actions, 1 trees, 0 blobs -INFO: Building [["@","tutorial","","helloworld"],{}]. -INFO: Processed 2 actions, 0 cache hits. -INFO: Artifacts built, logical paths are: - helloworld [487dc9e47b978877ed2f7d80b3395ce84b23be92:16992:x] -$ -#+END_SRC - -Note that the output binary may have changed due to different defaults. - -** Modeling target dependencies - -For demonstration purposes, we will separate the print statements into a static -library ~greet~, which will become a dependency to our binary. Therefore, we -create a new subdirectory ~greet~ with the files ~greet/greet.hpp~: - -#+SRCNAME: greet/greet.hpp -#+BEGIN_SRC cpp -#include <string> - -void greet(std::string const& s); -#+END_SRC - -and ~greet/greet.cpp~: - -#+SRCNAME: greet/greet.cpp -#+BEGIN_SRC cpp -#include "greet.hpp" -#include <iostream> - -void greet(std::string const& s) { - std::cout << "Hello " << s << "!\n"; -} -#+END_SRC - -These files can now be used to create a static library ~libgreet.a~. To do so, -we need to create the following target description in ~greet/TARGETS~: - -#+SRCNAME: greet/TARGETS -#+BEGIN_SRC js -{ "greet": - { "type": ["@", "rules", "CC", "library"] - , "name": ["greet"] - , "hdrs": ["greet.hpp"] - , "srcs": ["greet.cpp"] - , "stage": ["greet"] - } -} -#+END_SRC - -Similar to ~"binary"~, we have to provide a name and source file. Additionally, -a library has public headers defined via ~"hdrs"~ and an optional staging -directory ~"stage"~ (default value ~"."~). The staging directory specifies where -the consumer of this library can expect to find the library's artifacts. Note -that this does not need to reflect the location on the file system (i.e., a -full-qualified path like ~["com", "example", "utils", "greet"]~ could be used to -distinguish it from greeting libraries of other projects). The staging directory -does not only affect the main artifact ~libgreet.a~ but also it's /runfiles/, -a second set of artifacts, usually those a consumer needs to make proper use the -actual artifact; in the case of a library, the runfiles are its public headers. -Hence, the public header will be staged to ~"greet/greet.hpp"~. With that -knowledge, we can now perform the necessary modifications to ~main.cpp~: - -#+SRCNAME: main.cpp -#+BEGIN_SRC cpp -#include "greet/greet.hpp" - -int main() { - greet("Universe"); - return 0; -} -#+END_SRC - -The target ~"helloworld"~ will have a direct dependency to the target ~"greet"~ -of the module ~"greet"~ in the top-level ~TARGETS~ file: - -#+SRCNAME: TARGETS -#+BEGIN_SRC js -{ "helloworld": - { "type": ["@", "rules", "CC", "binary"] - , "name": ["helloworld"] - , "srcs": ["main.cpp"] - , "private-deps": [["greet", "greet"]] - } -} -#+END_SRC - -Note that there is no need to explicitly specify ~"greet"~'s public headers here -as the appropriate artifacts of dependencies are automatically added to the -inputs of compile and link actions. The new binary can be built with the same -command as before (no need to rerun ~just-mr~): - -#+BEGIN_SRC sh -$ just-mr build helloworld -INFO: Requested target is [["@","tutorial","","helloworld"],{}] -INFO: Analysed target [["@","tutorial","","helloworld"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 4 actions, 2 trees, 0 blobs -INFO: Building [["@","tutorial","","helloworld"],{}]. -INFO: Processed 4 actions, 0 cache hits. -INFO: Artifacts built, logical paths are: - helloworld [2b81e3177afc382452a2df9f294d3df90a9ccaf0:17664:x] -$ -#+END_SRC - -To only build the static library target ~"greet"~ from module ~"greet"~, run the -following command: - -#+BEGIN_SRC sh -$ just-mr build greet greet -INFO: Requested target is [["@","tutorial","greet","greet"],{}] -INFO: Analysed target [["@","tutorial","greet","greet"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 2 actions, 1 trees, 0 blobs -INFO: Building [["@","tutorial","greet","greet"],{}]. -INFO: Processed 2 actions, 2 cache hits. -INFO: Artifacts built, logical paths are: - greet/libgreet.a [83ed406e21f285337b0c9bd5011f56f656bba683:2992:f] - (1 runfiles omitted.) -$ -#+END_SRC diff --git a/doc/tutorial/proto.org b/doc/tutorial/proto.md index b4a02d48..8a04e373 100644 --- a/doc/tutorial/proto.org +++ b/doc/tutorial/proto.md @@ -1,27 +1,28 @@ -* Using protocol buffers +Using protocol buffers +====================== -The rules /justbuild/ uses for itself also support protocol -buffers. This tutorial shows how to use those rules and the targets -associated with them. It is not a tutorial on protocol buffers -itself; rather, it is assumed that the reader has some knowledge on -[[https://developers.google.com/protocol-buffers/][protocol buffers]]. +The rules *justbuild* uses for itself also support protocol buffers. +This tutorial shows how to use those rules and the targets associated +with them. It is not a tutorial on protocol buffers itself; rather, it +is assumed that the reader has some knowledge on [protocol +buffers](https://developers.google.com/protocol-buffers/). -** Setting up the repository configuration +Setting up the repository configuration +--------------------------------------- -Before we begin, we first need to declare where the root of our workspace is -located by creating the empty file ~ROOT~: +Before we begin, we first need to declare where the root of our +workspace is located by creating the empty file `ROOT`: -#+BEGIN_SRC sh +``` sh $ touch ROOT -#+END_SRC +``` -The ~protobuf~ repository conveniently contains an -[[https://github.com/protocolbuffers/protobuf/tree/v3.12.4/examples][example]], -so we can use this and just add our own target files. We create -file ~repos.template.json~ as follows. +The `protobuf` repository conveniently contains an +[example](https://github.com/protocolbuffers/protobuf/tree/v3.12.4/examples), +so we can use this and just add our own target files. We create file +`repos.template.json` as follows. -#+SRCNAME: repos.template.json -#+BEGIN_SRC js +``` {.jsonc srcname="repos.template.json"} { "repositories": { "": { "repository": @@ -36,45 +37,45 @@ file ~repos.template.json~ as follows. , "tutorial": {"repository": {"type": "file", "path": "."}} } } -#+END_SRC +``` -The missing entry ~"rules-cc"~ refers to our C/C++ build rules provided -[[https://github.com/just-buildsystem/rules-cc][online]]. These rules support -protobuf if the dependency ~"protoc"~ is provided. To import this rule -repository including the required transitive dependencies for protobuf, the -~bin/just-import-git~ script with option ~--as rules-cc~ can be used to -generate the actual ~repos.json~: +The missing entry `"rules-cc"` refers to our C/C++ build rules provided +[online](https://github.com/just-buildsystem/rules-cc). These rules +support protobuf if the dependency `"protoc"` is provided. To import +this rule repository including the required transitive dependencies for +protobuf, the `bin/just-import-git` script with option `--as rules-cc` +can be used to generate the actual `repos.json`: -#+BEGIN_SRC sh +``` sh $ just-import-git -C repos.template.json -b master --as rules-cc https://github.com/just-buildsystem/rules-cc > repos.json -#+END_SRC +``` -To build the example with ~just~, the only task is to write targets files. As -that contains a couple of new concepts, we will do this step by step. +To build the example with `just`, the only task is to write targets +files. As that contains a couple of new concepts, we will do this step +by step. -** The proto library +The proto library +----------------- First, we have to declare the proto library. In this case, it only -contains the file ~addressbook.proto~ and has no dependencies. To -declare the library, create a ~TARGETS~ file with the following -content: +contains the file `addressbook.proto` and has no dependencies. To +declare the library, create a `TARGETS` file with the following content: -#+SRCNAME: TARGETS -#+BEGIN_SRC js +``` {.jsonc srcname="TARGETS"} { "address": { "type": ["@", "rules", "proto", "library"] , "name": ["addressbook"] , "srcs": ["addressbook.proto"] } } -#+END_SRC +``` In general, proto libraries could also depend on other proto libraries; -those would be added to the ~"deps"~ field. +those would be added to the `"deps"` field. When building the library, there's very little to do. -#+BEGIN_SRC sh +``` sh $ just-mr build address INFO: Requested target is [["@","","","address"],{}] INFO: Analysed target [["@","","","address"],{}] @@ -84,20 +85,21 @@ INFO: Building [["@","","","address"],{}]. INFO: Processed 0 actions, 0 cache hits. INFO: Artifacts built, logical paths are: $ -#+END_SRC +``` On the other hand, what did we expect? A proto library is an abstract description of a protocol, so, as long as we don't specify for which language we want to have bindings, there is nothing to generate. -Nevertheless, a proto library target is not empty. In fact, it can't be empty, -as other targets can only access the values of a target and have no -insights into its definitions. We already relied on this design principle -implicitly, when we exploited target-level caching for our external dependencies -and did not even construct the dependency graph for that target. A proto -library simply provides the dependency structure of the ~.proto~ files. +Nevertheless, a proto library target is not empty. In fact, it can't be +empty, as other targets can only access the values of a target and have +no insights into its definitions. We already relied on this design +principle implicitly, when we exploited target-level caching for our +external dependencies and did not even construct the dependency graph +for that target. A proto library simply provides the dependency +structure of the `.proto` files. -#+BEGIN_SRC sh +``` sh $ just-mr analyse --dump-nodes - address INFO: Requested target is [["@","","","address"],{}] INFO: Result of target [["@","","","address"],{}]: { @@ -146,36 +148,35 @@ INFO: Target nodes of target [["@","","","address"],{}]: } } $ -#+END_SRC +``` -The target has one provider ~"proto"~, which is a node. Nodes are -an abstract representation of a target graph. More precisely, there -are two kind of nodes, and our example contains one of each. +The target has one provider `"proto"`, which is a node. Nodes are an +abstract representation of a target graph. More precisely, there are two +kind of nodes, and our example contains one of each. -The simple kind of nodes are the value nodes; they represent a -target that has a fixed value, and hence are given by artifacts, -runfiles, and provided data. In our case, we have one value node, -the one for the ~.proto~ file. +The simple kind of nodes are the value nodes; they represent a target +that has a fixed value, and hence are given by artifacts, runfiles, and +provided data. In our case, we have one value node, the one for the +`.proto` file. The other kind of nodes are the abstract nodes. They describe the -arguments for a target, but only have an abstract name (i.e., a -string) for the rule. Combining such an abstract target with a -binding for the abstract rule names gives a concrete "anonymous" -target that, in our case, will generate the library with the bindings -for the concrete language. In this example, the abstract name is -~"library"~. The alternative in our proto rules would have been -~"service library"~, for proto libraries that also contain ~rpc~ -definitions (which is used by [[https://grpc.io/][gRPC]]). - -** Using proto libraries - -Using proto libraries requires, as discussed, bindings for the -abstract names. Fortunately, our ~CC~ rules are aware of proto -libraries, so we can simply use them. Our target file hence -continues as follows. - -#+SRCNAME: TARGETS -#+BEGIN_SRC js +arguments for a target, but only have an abstract name (i.e., a string) +for the rule. Combining such an abstract target with a binding for the +abstract rule names gives a concrete "anonymous" target that, in our +case, will generate the library with the bindings for the concrete +language. In this example, the abstract name is `"library"`. The +alternative in our proto rules would have been `"service library"`, for +proto libraries that also contain `rpc` definitions (which is used by +[gRPC](https://grpc.io/)). + +Using proto libraries +--------------------- + +Using proto libraries requires, as discussed, bindings for the abstract +names. Fortunately, our `CC` rules are aware of proto libraries, so we +can simply use them. Our target file hence continues as follows. + +``` {.jsonc srcname="TARGETS"} ... , "add_person": { "type": ["@", "rules", "CC", "binary"] @@ -190,14 +191,14 @@ continues as follows. , "private-proto": ["address"] } ... -#+END_SRC +``` -The first time, we build a target that requires the proto compiler -(in that particular version, built in that particular way), it takes -a bit of time, as the proto compiler has to be built. But in follow-up -builds, also in different projects, the target-level cache is filled already. +The first time, we build a target that requires the proto compiler (in +that particular version, built in that particular way), it takes a bit +of time, as the proto compiler has to be built. But in follow-up builds, +also in different projects, the target-level cache is filled already. -#+BEGIN_SRC sh +``` sh $ just-mr build add_person ... $ just-mr build add_person @@ -210,12 +211,12 @@ INFO: Processed 5 actions, 5 cache hits. INFO: Artifacts built, logical paths are: add_person [bcbb3deabfe0d77e6d3ea35615336a2f59a1b0aa:2285928:x] $ -#+END_SRC +``` If we look at the actions associated with the binary, we find that those are still the two actions we expect: a compile action and a link action. -#+BEGIN_SRC sh +``` sh $ just-mr analyse add_person --dump-actions - INFO: Requested target is [["@","","","add_person"],{}] INFO: Result of target [["@","","","add_person"],{}]: { @@ -251,17 +252,17 @@ INFO: Actions for target [["@","","","add_person"],{}]: } ] $ -#+END_SRC +``` -As discussed, the ~libaddressbook.a~ that is conveniently available -during the linking of the binary (as well as the ~addressbook.pb.h~ -available in the ~include~ tree for the compile action) are generated -by an anonymous target. Using that during the build we already -filled the target-level cache, we can have a look at all targets -still analysed. In the one anonymous target, we find again the -abstract node we discussed earlier. +As discussed, the `libaddressbook.a` that is conveniently available +during the linking of the binary (as well as the `addressbook.pb.h` +available in the `include` tree for the compile action) are generated by +an anonymous target. Using that during the build we already filled the +target-level cache, we can have a look at all targets still analysed. In +the one anonymous target, we find again the abstract node we discussed +earlier. -#+BEGIN_SRC sh +``` sh $ just-mr analyse add_person --dump-targets - INFO: Requested target is [["@","","","add_person"],{}] INFO: Result of target [["@","","","add_person"],{}]: { @@ -302,25 +303,24 @@ INFO: List of analysed targets: } } $ -#+END_SRC - -It should be noted, however, that this tight integration of proto -into our ~C++~ rules is just convenience of our code base. If we had -to cooperate with rules not aware of proto, we could have created -a separate rule delegating the library creation to the anonymous -target and then simply reflecting the values of that target. -In fact, we could simply use an empty library with a public ~proto~ -dependency for this purpose. - -#+SRCNAME: TARGETS -#+BEGIN_SRC js +``` + +It should be noted, however, that this tight integration of proto into +our `C++` rules is just convenience of our code base. If we had to +cooperate with rules not aware of proto, we could have created a +separate rule delegating the library creation to the anonymous target +and then simply reflecting the values of that target. In fact, we could +simply use an empty library with a public `proto` dependency for this +purpose. + +``` {.jsonc srcname="TARGETS"} ... , "address proto library": {"type": ["@", "rules", "CC", "library"], "proto": ["address"]} ... -#+END_SRC +``` -#+BEGIN_SRC sh +``` sh $ just-mr analyse 'address proto library' ... INFO: Requested target is [["@","","","address proto library"],{}] @@ -347,18 +347,18 @@ INFO: Result of target [["@","","","address proto library"],{}]: { } } $ -#+END_SRC +``` -** Adding a test +Adding a test +------------- -Finally, let's add a test. As we use the ~protobuf~ repository as -workspace root, we add the test script ad hoc into a targets file, -using the ~"file_gen"~ rule. For debugging a potentially failing -test, we also keep the intermediate files the test generates. -Create a top-level ~TARGETS~ file with the following content: +Finally, let's add a test. As we use the `protobuf` repository as +workspace root, we add the test script ad hoc into a targets file, using +the `"file_gen"` rule. For debugging a potentially failing test, we also +keep the intermediate files the test generates. Create a top-level +`TARGETS` file with the following content: -#+SRCNAME: TARGETS -#+BEGIN_SRC js +``` {.jsonc srcname="TARGETS"} ... , "test.sh": { "type": "file_gen" @@ -382,17 +382,16 @@ Create a top-level ~TARGETS~ file with the following content: , "keep": ["addressbook.data", "out.txt"] } ... -#+END_SRC +``` -That example also shows why it is important that the generation -of the language bindings is delegated to an anonymous target: we -want to analyse only once how the ~C++~ bindings are generated. -Nevertheless, many targets can depend (directly or indirectly) on -the same proto library. And, indeed, analysing the test, we get -the expected additional targets and the one anonymous target is -reused by both binaries. +That example also shows why it is important that the generation of the +language bindings is delegated to an anonymous target: we want to +analyse only once how the `C++` bindings are generated. Nevertheless, +many targets can depend (directly or indirectly) on the same proto +library. And, indeed, analysing the test, we get the expected additional +targets and the one anonymous target is reused by both binaries. -#+BEGIN_SRC sh +``` sh $ just-mr analyse test --dump-targets - INFO: Requested target is [["@","","","test"],{}] INFO: Result of target [["@","","","test"],{}]: { @@ -444,11 +443,11 @@ INFO: List of analysed targets: } INFO: Target tainted ["test"]. $ -#+END_SRC +``` Finally, the test passes and the output is as expected. -#+BEGIN_SRC sh +``` sh $ just-mr build test -Pwork/out.txt INFO: Requested target is [["@","","","test"],{}] INFO: Analysed target [["@","","","test"],{}] @@ -472,4 +471,4 @@ Person ID: 12345 Updated: 2022-12-14T18:08:36Z INFO: Target tainted ["test"]. $ -#+END_SRC +``` diff --git a/doc/tutorial/rebuild.org b/doc/tutorial/rebuild.md index 80aafb6f..3f1ddd88 100644 --- a/doc/tutorial/rebuild.org +++ b/doc/tutorial/rebuild.md @@ -1,15 +1,17 @@ -* Ensuring reproducibility of the build - -Software builds should be [[https://reproducible-builds.org/][reproducible]]. -The ~just~ tool, supports this goal in local builds by isolating -individual actions, setting permissions and file time stamps to -canonical values, etc; most remote execution systems take even further -measures to ensure the environment always looks the same to every -action. Nevertheless, it is always possible to break reproducibility -by bad actions, both coming from rules not carefully written, as -well as from ad-hoc actions added by the ~generic~ target. - -#+BEGIN_SRC js +Ensuring reproducibility of the build +===================================== + +Software builds should be +[reproducible](https://reproducible-builds.org/). The `just` tool, +supports this goal in local builds by isolating individual actions, +setting permissions and file time stamps to canonical values, etc; most +remote execution systems take even further measures to ensure the +environment always looks the same to every action. Nevertheless, it is +always possible to break reproducibility by bad actions, both coming +from rules not carefully written, as well as from ad-hoc actions added +by the `generic` target. + +``` jsonc ... , "version.h": { "type": "generic" @@ -18,29 +20,29 @@ well as from ad-hoc actions added by the ~generic~ target. , "outs": ["version.h"] } ... -#+END_SRC - -Besides time stamps there are many other sources of nondeterminism, -like properties of the build machine (name, number of CPUs available, -etc), but also subtle ones like ~readdir~ order. Often, those -non-reproducible parts get buried deeply in a final artifact (like -the version string embedded in a binary contained in a compressed -installation archive); and, as long as the non-reproducible action -stays in cache, it does not even result in bad incrementality. -Still, others won't be able to reproduce the exact artifact. - -There are tools like [[https://diffoscope.org/][diffoscope]] to deeply +``` + +Besides time stamps there are many other sources of nondeterminism, like +properties of the build machine (name, number of CPUs available, etc), +but also subtle ones like `readdir` order. Often, those non-reproducible +parts get buried deeply in a final artifact (like the version string +embedded in a binary contained in a compressed installation archive); +and, as long as the non-reproducible action stays in cache, it does not +even result in bad incrementality. Still, others won't be able to +reproduce the exact artifact. + +There are tools like [diffoscope](https://diffoscope.org/) to deeply compare archives and other container formats. Nevertheless, it is desirable to find the root causes, i.e., the first (in topological order) actions that yield a different output. -** Rebuilding +Rebuilding +---------- -For the remainder of this section, we will consider the following example -project with the C++ source file ~hello.cpp~: +For the remainder of this section, we will consider the following +example project with the C++ source file `hello.cpp`: -#+SRCNAME: hello.cpp -#+BEGIN_SRC cpp +``` {.cpp srcname="hello.cpp"} #include <iostream> #include "version.h" @@ -50,12 +52,11 @@ int main(int argc, const char* argv[]) { } return 0; } -#+END_SRC +``` -and the following ~TARGETS~ file: +and the following `TARGETS` file: -#+SRCNAME: TARGETS -#+BEGIN_SRC js +``` {.jsonc srcname="TARGETS"} { "": { "type": "install" , "files": @@ -95,17 +96,17 @@ and the following ~TARGETS~ file: , "deps": ["out.txt"] } } -#+END_SRC +``` -To search for the root cause of non-reproducibility, ~just~ has -a subcommand ~rebuild~. It builds the specified target again, requesting +To search for the root cause of non-reproducibility, `just` has a +subcommand `rebuild`. It builds the specified target again, requesting that every action be executed again (but target-level cache is still active); then the result of every action is compared to the one in the action cache, if present with the same inputs. So, you typically would -first ~build~ and then ~rebuild~. Note that a repeated ~build~ simply +first `build` and then `rebuild`. Note that a repeated `build` simply takes the action result from cache. -#+BEGIN_SRC sh +``` sh $ just-mr build INFO: Requested target is [["@","tutorial","",""],{}] INFO: Analysed target [["@","tutorial","",""],{}] @@ -135,30 +136,31 @@ INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching INFO: Discovered 6 actions, 1 trees, 0 blobs INFO: Rebuilding [["@","tutorial","",""],{}]. WARN: Found flaky action: - - id: c854a382ea26628e1a5b8d4af00d6d0cef433436 - - cmd: ["sh","-c","echo '#define VERSION \"0.0.0.'`date +%Y%m%d%H%M%S`'\"' > version.h\n"] - - output 'version.h' differs: - - [6aac3477e22cd57e8c98ded78562d3c017e5d611:39:f] (rebuilt) - - [789a29f39b6aa966f91776bfe092e247614e6acd:39:f] (cached) + - id: c854a382ea26628e1a5b8d4af00d6d0cef433436 + - cmd: ["sh","-c","echo '#define VERSION \"0.0.0.'`date +%Y%m%d%H%M%S`'\"' > version.h\n"] + - output 'version.h' differs: + - [6aac3477e22cd57e8c98ded78562d3c017e5d611:39:f] (rebuilt) + - [789a29f39b6aa966f91776bfe092e247614e6acd:39:f] (cached) INFO: 2 actions compared with cache, 1 flaky actions found (0 of which tainted), no cache entry found for 4 actions. INFO: Artifacts built, logical paths are: bin/hello [73994ff43ec1161aba96708f277e8c88feab0386:16608:x] share/hello/OUT.txt [428b97b82b6c59cad7488b24e6b618ebbcd819bc:13:f] share/hello/version.txt [8dd65747395c0feab30891eab9e11d4a9dd0c715:39:f] $ -#+END_SRC +``` -In the example, the second action compared to cache is the upper -casing of the output. Even though the generation of ~out.txt~ depends -on the non-reproducible ~hello~, the file itself is reproducible. -Therefore, the follow-up actions are checked as well. +In the example, the second action compared to cache is the upper casing +of the output. Even though the generation of `out.txt` depends on the +non-reproducible `hello`, the file itself is reproducible. Therefore, +the follow-up actions are checked as well. -For this simple example, reading the console output is enough to understand -what's going on. However, checking for reproducibility usually is part -of a larger, quality-assurance process. To support the automation of such -processes, the findings can also be reported in machine-readable form. +For this simple example, reading the console output is enough to +understand what's going on. However, checking for reproducibility +usually is part of a larger, quality-assurance process. To support the +automation of such processes, the findings can also be reported in +machine-readable form. -#+BEGIN_SRC sh +``` sh $ just-mr rebuild --dump-flaky flakes.json --dump-graph actions.json [...] $ cat flakes.json @@ -186,40 +188,40 @@ $ cat flakes.json } } }$ -#+END_SRC +``` The file reports the flaky actions together with the non-reproducible artifacts they generated, reporting both, the cached and the newly -generated output. The files themselves can be obtained via ~just -install-cas~ as usual, allowing deeper comparison of the outputs. -The full definitions of the actions can be found in the action graph, -in the example dumped as well as ~actions.json~; this definition -also includes the origins for each action, i.e., the configured -targets that requested the respective action. +generated output. The files themselves can be obtained via `just +install-cas` as usual, allowing deeper comparison of the outputs. The +full definitions of the actions can be found in the action graph, in the +example dumped as well as `actions.json`; this definition also includes +the origins for each action, i.e., the configured targets that requested +the respective action. - -** Comparing build environments +Comparing build environments +---------------------------- Simply rebuilding on the same machine is good way to detect embedded time stamps of sufficiently small granularity; for other sources of -non-reproducibility, however, more modifications of the environment -are necessary. - -A simple, but effective, way for modifying the build environment -is the option ~-L~ to set the local launcher, a list of -strings the argument vector is prefixed with before the action is -executed. The default ~["env", "--"]~ simply resolves the program -to be executed in the current value of ~PATH~, but a different -value for the launcher can obviously be used to set environment -variables like ~LD_PRELOAD~. Relevant libraries and tools -include [[https://github.com/wolfcw/libfaketime][libfaketime]], -[[https://github.com/dtcooper/fakehostname][fakehostname]], -and [[https://salsa.debian.org/reproducible-builds/disorderfs][disorderfs]]. +non-reproducibility, however, more modifications of the environment are +necessary. + +A simple, but effective, way for modifying the build environment is the +option `-L` to set the local launcher, a list of strings the argument +vector is prefixed with before the action is executed. The default +`["env", "--"]` simply resolves the program to be executed in the +current value of `PATH`, but a different value for the launcher can +obviously be used to set environment variables like `LD_PRELOAD`. +Relevant libraries and tools include +[libfaketime](https://github.com/wolfcw/libfaketime), +[fakehostname](https://github.com/dtcooper/fakehostname), and +[disorderfs](https://salsa.debian.org/reproducible-builds/disorderfs). More variation can be achieved by comparing remote execution builds, -either for two different remote-execution end points or comparing -one remote-execution end point to the local build. The latter is -also a good way to find out where a build that "works on my machine" -differs. The endpoint on which the rebuild is executed can be set, -in the same way as for build with the ~-r~ option; the cache end -point to compare against can be set via the ~--vs~ option. +either for two different remote-execution end points or comparing one +remote-execution end point to the local build. The latter is also a good +way to find out where a build that "works on my machine" differs. The +endpoint on which the rebuild is executed can be set, in the same way as +for build with the `-r` option; the cache end point to compare against +can be set via the `--vs` option. diff --git a/doc/tutorial/target-file-glob-tree.org b/doc/tutorial/target-file-glob-tree.md index 58e9c725..524cf358 100644 --- a/doc/tutorial/target-file-glob-tree.org +++ b/doc/tutorial/target-file-glob-tree.md @@ -1,34 +1,35 @@ -* Target versus ~FILE~, ~GLOB~, and ~TREE~ +Target versus `FILE`, `GLOB`, and `TREE` +======================================== -So far, we referred to defined targets as well as source files -by their name and it just worked. When considering third-party -software we already saw the ~TREE~ reference. In this section, we -will highlight in more detail the ways to refer to sources, as well -as the difference between defined and source targets. The latter -is used, e.g., when third-party software has to be patched. +So far, we referred to defined targets as well as source files by their +name and it just worked. When considering third-party software we +already saw the `TREE` reference. In this section, we will highlight in +more detail the ways to refer to sources, as well as the difference +between defined and source targets. The latter is used, e.g., when +third-party software has to be patched. -As example for this section we use gnu ~units~ where we want to -patch into the standard units definition add two units of area -popular in German news. +As example for this section we use gnu `units` where we want to patch +into the standard units definition add two units of area popular in +German news. -** Repository Config for ~units~ with patches +Repository Config for `units` with patches +------------------------------------------ -Before we begin, we first need to declare where the root of our workspace is -located by creating the empty file ~ROOT~: +Before we begin, we first need to declare where the root of our +workspace is located by creating the empty file `ROOT`: -#+BEGIN_SRC sh +``` sh $ touch ROOT -#+END_SRC +``` The sources are an archive available on the web. As upstream uses a -different build system, we have to provide our own build description; -we take the top-level directory as layer for this. As we also want -to patch the definition file, we add the subdirectory ~files~ as -logical repository for the patches. Hence we create a file ~repos.json~ -with the following content. - -#+SRCNAME: repos.json -#+BEGIN_SRC js +different build system, we have to provide our own build description; we +take the top-level directory as layer for this. As we also want to patch +the definition file, we add the subdirectory `files` as logical +repository for the patches. Hence we create a file `repos.json` with the +following content. + +``` {.jsonc srcname="repos.json"} { "main": "units" , "repositories": { "rules-cc": @@ -55,31 +56,33 @@ with the following content. } } } -#+END_SRC +``` -The repository to set up is ~units~ and, as usual, we can use ~just-mr~ to -fetch the archive and obtain the resulting multi-repository configuration. +The repository to set up is `units` and, as usual, we can use `just-mr` +to fetch the archive and obtain the resulting multi-repository +configuration. -#+BEGIN_SRC sh +``` sh $ just-mr setup units -#+END_SRC +``` -** Patching a file: targets versus ~FILE~ +Patching a file: targets versus `FILE` +-------------------------------------- -Let's start by patching the source file ~definitions.units~. While, -conceptionally, we want to patch a third-party source file, we do /not/ +Let's start by patching the source file `definitions.units`. While, +conceptionally, we want to patch a third-party source file, we do *not* modify the sources. The workspace root is a git tree and stay like this. -Instead, we remember that we specify /targets/ and the definition of a +Instead, we remember that we specify *targets* and the definition of a target is looked up in the targets file; only if not defined there, it is implicitly considered a source target and taken from the target root. -So we will define a /target/ named ~definitions.units~ to replace the +So we will define a *target* named `definitions.units` to replace the original source file. -Let's first generate the patch. As we're already referring to source files -as targets, we have to provide a targets file already; we start with the -empty object and refine it later. +Let's first generate the patch. As we're already referring to source +files as targets, we have to provide a targets file already; we start +with the empty object and refine it later. -#+BEGIN_SRC sh +``` sh $ echo {} > TARGETS.units $ just-mr install -o . definitions.units INFO: Requested target is [["@","units","","definitions.units"],{}] @@ -100,41 +103,39 @@ $ mkdir files $ echo {} > files/TARGETS $ diff -u definitions.units.orig definitions.units > files/definitions.units.diff $ rm definitions.units* -#+END_SRC - -Our rules conveniently contain a rule ~["patch", "file"]~ to patch -a single file, and we already created the patch. The only other -input missing is the source file. So far, we could refer to it as -~"definitions.units"~ because there was no target of that name, but -now we're about to define a target with that very name. Fortunately, -in target files, we can use a special syntax to explicitly refer to -a source file of the current module, even if there is a target with -the same name: ~["FILE", null, "definition.units"]~. The syntax -requires the explicit ~null~ value for the current module, despite -the fact that explicit file references are only allowed for the -current module; in this way, the name is a list of length more than -two and cannot be confused with a top-level module called ~FILE~. -So we add this target and obtain as ~TARGETS.units~ the following. - -#+SRCNAME: TARGETS.units -#+BEGIN_SRC js +``` + +Our rules conveniently contain a rule `["patch", "file"]` to patch a +single file, and we already created the patch. The only other input +missing is the source file. So far, we could refer to it as +`"definitions.units"` because there was no target of that name, but now +we're about to define a target with that very name. Fortunately, in +target files, we can use a special syntax to explicitly refer to a +source file of the current module, even if there is a target with the +same name: `["FILE", null, "definition.units"]`. The syntax requires the +explicit `null` value for the current module, despite the fact that +explicit file references are only allowed for the current module; in +this way, the name is a list of length more than two and cannot be +confused with a top-level module called `FILE`. So we add this target +and obtain as `TARGETS.units` the following. + +``` {.jsonc srcname="TARGETS.units"} { "definitions.units": { "type": ["@", "rules", "patch", "file"] , "src": [["FILE", ".", "definitions.units"]] , "patch": [["@", "patches", "", "definitions.units.diff"]] } } -#+END_SRC +``` -Analysing ~"definitions.units"~ we find our defined target which -contains an action output. Still, it looks like a patched source -file; the new artifact is staged to the original location. Staging -is also used in the action definition, to avoid magic names (like -file names starting with ~-~), in-place operations (all actions -must not modify their inputs) and, in fact, have a -fixed command line. +Analysing `"definitions.units"` we find our defined target which +contains an action output. Still, it looks like a patched source file; +the new artifact is staged to the original location. Staging is also +used in the action definition, to avoid magic names (like file names +starting with `-`), in-place operations (all actions must not modify +their inputs) and, in fact, have a fixed command line. -#+BEGIN_SRC sh +``` sh $ just-mr analyse definitions.units --dump-actions - INFO: Requested target is [["@","units","","definitions.units"],{}] INFO: Result of target [["@","units","","definitions.units"],{}]: { @@ -172,11 +173,11 @@ INFO: Actions for target [["@","units","","definitions.units"],{}]: } ] $ -#+END_SRC +``` -Building ~"definitions.units"~ we find out patch applied correctly. +Building `"definitions.units"` we find out patch applied correctly. -#+BEGIN_SRC sh +``` sh $ just-mr build definitions.units -P definitions.units | grep -A 5 'German units' INFO: Requested target is [["@","units","","definitions.units"],{}] INFO: Analysed target [["@","units","","definitions.units"],{}] @@ -193,24 +194,24 @@ area_soccerfield 105 m * 68 m area_saarland 2570 km^2 zentner 50 kg $ -#+END_SRC +``` -** Globbing source files: ~"GLOB"~ +Globbing source files: `"GLOB"` +------------------------------- -Next, we collect all ~.units~ files. We could simply do this by enumerating -them in a target. +Next, we collect all `.units` files. We could simply do this by +enumerating them in a target. -#+SRCNAME: TARGETS.units -#+BEGIN_SRC js +``` {.jsonc srcname="TARGETS.units"} ... , "data-draft": { "type": "install", "deps": ["definitions.units", "currency.units"]} ... -#+END_SRC +``` -In this way, we get the desired collection of one unmodified source file and -the output of the patch action. +In this way, we get the desired collection of one unmodified source file +and the output of the patch action. -#+BEGIN_SRC sh +``` sh $ just-mr analyse data-draft INFO: Requested target is [["@","units","","data-draft"],{}] INFO: Result of target [["@","units","","data-draft"],{}]: { @@ -226,77 +227,76 @@ INFO: Result of target [["@","units","","data-draft"],{}]: { } } $ -#+END_SRC - -The disadvantage, however, that we might miss newly added ~.units~ -files if we update and upstream added new files. So we want all -source files that have the respective ending. The corresponding -source reference is ~"GLOB"~. A glob expands to the /collection/ -of all /sources/ that are /files/ in the /top-level/ directory of -the current module and that match the given pattern. It is important -to understand this in detail and the rational behind it. -- First of all, the artifact (and runfiles) map has an entry for - each file that matches. In particular, targets have the option to - define individual actions for each file, like ~["CC", "binary"]~ - does for the source files. This is different from ~"TREE"~ where - the artifact map contains a single artifact that happens to be a - directory. The tree behaviour is preferable when the internals - of the directory only matter for the execution of actions and not - for analysis; then there are less entries to carry around during - analysis and action-key computation, and the whole directory - is "reserved" for that tree avoid staging conflicts when latter - adding entries there. -- As a source reference, a glob expands to explicit source files; - targets having the same name as a source file are not taken into - account. In our example, ~["GLOB", null, "*.units"]~ therefore - contains the unpatched source file ~definitions.units~. In this - way, we avoid any surprises in the expansion of a glob when a new - source file is added with a name equal to an already existing target. -- Only files are considered for matching the glob. Directories - are ignored. -- Matches are only considered at the top-level directory. In this - way, only one directory has to be read during analysis; allowing - deeper globs would require traversal of subdirectories requiring - larger cost. While the explicit ~"TREE"~ reference allows recursive - traversal, in the typical use case of the respective workspace root - being a ~git~ root, it is actually cheap; we can look up the - ~git~ tree identifier without traversing the tree. Such a quick - look up would not be possible if matches had to be selected. - -So, ~["GLOB", null, "*.units"]~ expands to all the relevant source -files; but we still want to keep the patching. Most rules, like ~"install"~, -disallow staging conflicts to avoid accidentally ignoring a file due -to conflicting name. In our case, however, the dropping of the source -file in favour of the patched one is deliberate. For this, there is -the rule ~["data", "overlay"]~ taking the union of the artifacts of +``` + +The disadvantage, however, that we might miss newly added `.units` files +if we update and upstream added new files. So we want all source files +that have the respective ending. The corresponding source reference is +`"GLOB"`. A glob expands to the *collection* of all *sources* that are +*files* in the *top-level* directory of the current module and that +match the given pattern. It is important to understand this in detail +and the rational behind it. + + - First of all, the artifact (and runfiles) map has an entry for each + file that matches. In particular, targets have the option to define + individual actions for each file, like `["CC", "binary"]` does for + the source files. This is different from `"TREE"` where the artifact + map contains a single artifact that happens to be a directory. The + tree behaviour is preferable when the internals of the directory + only matter for the execution of actions and not for analysis; then + there are less entries to carry around during analysis and + action-key computation, and the whole directory is "reserved" for + that tree avoid staging conflicts when latter adding entries there. + - As a source reference, a glob expands to explicit source files; + targets having the same name as a source file are not taken into + account. In our example, `["GLOB", null, "*.units"]` therefore + contains the unpatched source file `definitions.units`. In this way, + we avoid any surprises in the expansion of a glob when a new source + file is added with a name equal to an already existing target. + - Only files are considered for matching the glob. Directories are + ignored. + - Matches are only considered at the top-level directory. In this way, + only one directory has to be read during analysis; allowing deeper + globs would require traversal of subdirectories requiring larger + cost. While the explicit `"TREE"` reference allows recursive + traversal, in the typical use case of the respective workspace root + being a `git` root, it is actually cheap; we can look up the `git` + tree identifier without traversing the tree. Such a quick look up + would not be possible if matches had to be selected. + +So, `["GLOB", null, "*.units"]` expands to all the relevant source +files; but we still want to keep the patching. Most rules, like +`"install"`, disallow staging conflicts to avoid accidentally ignoring a +file due to conflicting name. In our case, however, the dropping of the +source file in favour of the patched one is deliberate. For this, there +is the rule `["data", "overlay"]` taking the union of the artifacts of the specified targets, accepting conflicts and resolving them in a -latest-wins fashion. Keep in mind, that our target fields are list, -not sets. Looking at the definition of the rule, one finds that -it is simply a ~"map_union"~. Hence we refine our ~"data"~ target. +latest-wins fashion. Keep in mind, that our target fields are list, not +sets. Looking at the definition of the rule, one finds that it is simply +a `"map_union"`. Hence we refine our `"data"` target. -#+SRCNAME: TARGETS.units -#+BEGIN_SRC js +``` {.jsonc srcname="TARGETS.units"} ... , "data": { "type": ["@", "rules", "data", "overlay"] , "deps": [["GLOB", null, "*.units"], "definitions.units"] } ... -#+END_SRC +``` The result of the analysis, of course, still is the same. -** Finishing the example: binaries from globbed sources +Finishing the example: binaries from globbed sources +---------------------------------------------------- -The source-code organisation of units is pretty simple. All source -and header files are in the top-level directory. As the header files -are not in a directory of their own, we can't use a tree, so we use -a glob, which is fine for the private headers of a binary. For the -source files, we have to have them individually anyway. So our first -attempt of defining the binary is as follows. +The source-code organisation of units is pretty simple. All source and +header files are in the top-level directory. As the header files are not +in a directory of their own, we can't use a tree, so we use a glob, +which is fine for the private headers of a binary. For the source files, +we have to have them individually anyway. So our first attempt of +defining the binary is as follows. -#+SRCNAME: TARGETS.units -#+BEGIN_SRC js +``` {.jsonc srcname="TARGETS.units"} ... , "units-draft": { "type": ["@", "rules", "CC", "binary"] @@ -307,12 +307,12 @@ attempt of defining the binary is as follows. , "private-hdrs": [["GLOB", null, "*.h"]] } ... -#+END_SRC +``` -The result basically work and shows that we have 5 source files in total, -giving 5 compile and one link action. +The result basically work and shows that we have 5 source files in +total, giving 5 compile and one link action. -#+BEGIN_SRC sh +``` sh $ just-mr build units-draft INFO: Requested target is [["@","units","","units-draft"],{}] INFO: Analysed target [["@","units","","units-draft"],{}] @@ -328,13 +328,14 @@ INFO: Processed 6 actions, 0 cache hits. INFO: Artifacts built, logical paths are: units [718cb1489bd006082f966ea73e3fba3dd072d084:124488:x] $ -#+END_SRC +``` -To keep the build clean, we want to get rid of the warning. Of course, we could -simply set an appropriate compiler flag, but let's do things properly and patch -away the underlying reason. To do so, we first create a patch. +To keep the build clean, we want to get rid of the warning. Of course, +we could simply set an appropriate compiler flag, but let's do things +properly and patch away the underlying reason. To do so, we first create +a patch. -#+BEGIN_SRC sh +``` sh $ just-mr install -o . strfunc.c INFO: Requested target is [["@","units","","strfunc.c"],{}] INFO: Analysed target [["@","units","","strfunc.c"],{}] @@ -353,12 +354,11 @@ $ echo -e "109\ns|N|// N\nw\nq" | ed strfunc.c $ diff strfunc.c.orig strfunc.c > files/strfunc.c.diff $ rm strfunc.c* $ -#+END_SRC +``` -Then we amend our ~"units"~ target. +Then we amend our `"units"` target. -#+SRCNAME: TARGETS.units -#+BEGIN_SRC js +``` {.jsonc srcname="TARGETS.units"} ... , "units": { "type": ["@", "rules", "CC", "binary"] @@ -378,14 +378,15 @@ Then we amend our ~"units"~ target. , "patch": [["@", "patches", "", "strfunc.c.diff"]] } ... -#+END_SRC +``` -Building the new target, 2 actions have to be executed: the patching, and -the compiling of the patched source file. As the patched file still generates -the same object file as the unpatched file (after all, we only wanted to get -rid of a warning), the linking step can be taken from cache. +Building the new target, 2 actions have to be executed: the patching, +and the compiling of the patched source file. As the patched file still +generates the same object file as the unpatched file (after all, we only +wanted to get rid of a warning), the linking step can be taken from +cache. -#+BEGIN_SRC sh +``` sh $ just-mr build units INFO: Requested target is [["@","units","","units"],{}] INFO: Analysed target [["@","units","","units"],{}] @@ -396,22 +397,21 @@ INFO: Processed 7 actions, 5 cache hits. INFO: Artifacts built, logical paths are: units [718cb1489bd006082f966ea73e3fba3dd072d084:124488:x] $ -#+END_SRC +``` -To finish the example, we also add a default target (using that, if no target -is specified, ~just~ builds the lexicographically first target), staging -artifacts according to the usual conventions. +To finish the example, we also add a default target (using that, if no +target is specified, `just` builds the lexicographically first target), +staging artifacts according to the usual conventions. -#+SRCNAME: TARGETS.units -#+BEGIN_SRC js +``` {.jsonc srcname="TARGETS.units"} ... , "": {"type": "install", "dirs": [["units", "bin"], ["data", "share/units"]]} ... -#+END_SRC +``` Then things work as expected -#+BEGIN_SRC sh +``` sh $ just-mr install -o /tmp/testinstall INFO: Requested target is [["@","units","",""],{}] INFO: Analysed target [["@","units","",""],{}] @@ -427,4 +427,4 @@ $ /tmp/testinstall/bin/units 'area_saarland' 'area_soccerfield' * 359943.98 / 2.7782101e-06 $ -#+END_SRC +``` diff --git a/doc/tutorial/tests.org b/doc/tutorial/tests.md index d6842ab2..138769b1 100644 --- a/doc/tutorial/tests.org +++ b/doc/tutorial/tests.md @@ -1,38 +1,41 @@ -* Creating Tests +Creating Tests +============== -To run tests with justbuild, we do /not/ have a dedicated ~test~ +To run tests with justbuild, we do *not* have a dedicated `test` subcommand. Instead, we consider tests being a specific action that -generates a test report. Consequently, we use the ~build~ subcommand -to build the test report, and thereby run the test action. Test -actions, however, are slightly different from normal actions in -that we don't want the build of the test report to be aborted if -a test action fails (but still, we want only successfully actions -taken from cache). Rules defining targets containing such special -actions have to identify themselves as /tainted/ by specifying -a string explaining why such special actions are justified; in -our case, the string is ~"test"~. Besides the implicit marking by -using a tainted rule, those tainting strings can also be explicitly -assigned by the user in the definition of a target, e.g., to mark -test data. Any target has to be tainted with (at least) all the -strings any of its dependencies is tainted with. In this way, it -is ensured that no test target will end up in a production build. - -For the remainder of this section, we expect to have the project files available -resulting from successfully completing the tutorial section on /Building C++ -Hello World/. We will demonstrate how to write a test binary for the ~greet~ -library and a shell test for the ~helloworld~ binary. - -** Creating a C++ test binary - -First, we will create a C++ test binary for testing the correct functionality of -the ~greet~ library. Therefore, we need to provide a C++ source file that performs -the actual testing and returns non-~0~ on failure. For simplicity reasons, we do -not use a testing framework for this tutorial. A simple test that captures -standard output and verifies it with the expected output should be provided in -the file ~tests/greet.test.cpp~: - -#+SRCNAME: tests/greet.test.cpp -#+BEGIN_SRC cpp +generates a test report. Consequently, we use the `build` subcommand to +build the test report, and thereby run the test action. Test actions, +however, are slightly different from normal actions in that we don't +want the build of the test report to be aborted if a test action fails +(but still, we want only successfully actions taken from cache). Rules +defining targets containing such special actions have to identify +themselves as *tainted* by specifying a string explaining why such +special actions are justified; in our case, the string is `"test"`. +Besides the implicit marking by using a tainted rule, those tainting +strings can also be explicitly assigned by the user in the definition of +a target, e.g., to mark test data. Any target has to be tainted with (at +least) all the strings any of its dependencies is tainted with. In this +way, it is ensured that no test target will end up in a production +build. + +For the remainder of this section, we expect to have the project files +available resulting from successfully completing the tutorial section on +*Building C++ Hello World*. We will demonstrate how to write a test +binary for the `greet` library and a shell test for the `helloworld` +binary. + +Creating a C++ test binary +-------------------------- + +First, we will create a C++ test binary for testing the correct +functionality of the `greet` library. Therefore, we need to provide a +C++ source file that performs the actual testing and returns non-`0` on +failure. For simplicity reasons, we do not use a testing framework for +this tutorial. A simple test that captures standard output and verifies +it with the expected output should be provided in the file +`tests/greet.test.cpp`: + +``` {.cpp srcname="tests/greet.test.cpp"} #include <functional> #include <iostream> #include <string> @@ -68,15 +71,14 @@ auto test_greet(std::string const& name) -> bool { int main() { return test_greet("World") && test_greet("Universe") ? 0 : 1; } -#+END_SRC +``` -Next, a new test target needs to be created in module ~greet~. This target uses -the rule ~["@", "rules", "CC/test", "test"]~ and needs to depend on the -~["greet", "greet"]~ target. To create the test target, add the following to -~tests/TARGETS~: +Next, a new test target needs to be created in module `greet`. This +target uses the rule `["@", "rules", "CC/test", "test"]` and needs to +depend on the `["greet", "greet"]` target. To create the test target, +add the following to `tests/TARGETS`: -#+SRCNAME: tests/TARGETS -#+BEGIN_SRC js +``` {.jsonc srcname="tests/TARGETS"} { "greet": { "type": ["@", "rules", "CC/test", "test"] , "name": ["test_greet"] @@ -84,35 +86,33 @@ the rule ~["@", "rules", "CC/test", "test"]~ and needs to depend on the , "private-deps": [["greet", "greet"]] } } -#+END_SRC +``` -Before we can run the test, a proper default module for ~CC/test~ must be -provided. By specifying the appropriate target in this module the default test -runner can be overwritten by a different test runner fom the rule's workspace -root. Moreover, all test targets share runner infrastructure from ~shell/test~, -e.g., summarizing multiple runs per test (to detect flakyness) if the configuration -variable ~RUNS_PER_TEST~ is set. +Before we can run the test, a proper default module for `CC/test` must +be provided. By specifying the appropriate target in this module the +default test runner can be overwritten by a different test runner fom +the rule's workspace root. Moreover, all test targets share runner +infrastructure from `shell/test`, e.g., summarizing multiple runs per +test (to detect flakyness) if the configuration variable `RUNS_PER_TEST` +is set. However, in our case, we want to use the default runner and therefore it is sufficient to create an empty module. To do so, create the file -~tutorial-defaults/CC/test/TARGETS~ with content +`tutorial-defaults/CC/test/TARGETS` with content -#+SRCNAME: tutorial-defaults/CC/test/TARGETS -#+BEGIN_SRC js +``` {.jsonc srcname="tutorial-defaults/CC/test/TARGETS"} {} -#+END_SRC +``` -as well as the file ~tutorial-defaults/shell/test/TARGETS~ with content +as well as the file `tutorial-defaults/shell/test/TARGETS` with content -#+SRCNAME: tutorial-defaults/shell/test/TARGETS -#+BEGIN_SRC js +``` {.jsonc srcname="tutorial-defaults/shell/test/TARGETS"} {} -#+END_SRC - +``` Now we can run the test (i.e., build the test result): -#+BEGIN_SRC sh +``` sh $ just-mr build tests greet INFO: Requested target is [["@","tutorial","tests","greet"],{}] INFO: Analysed target [["@","tutorial","tests","greet"],{}] @@ -130,41 +130,45 @@ INFO: Artifacts built, logical paths are: (1 runfiles omitted.) INFO: Target tainted ["test"]. $ -#+END_SRC - -Note that the target is correctly reported as tainted with ~"test"~. It will -produce 3 additional actions for compiling, linking and running the test binary. - -The result of the test target are 5 artifacts: ~result~ (containing ~UNKNOWN~, -~PASS~, or ~FAIL~), ~stderr~, ~stdout~, ~time-start~, and ~time-stop~, and a -single runfile (omitted in the output above), which is a tree artifact with the -name ~test_greet~ that contains all of the above artifacts. The test was run -successfully as otherwise all reported artifacts would have been reported as -~FAILED~ in the output, and justbuild would have returned the exit code ~2~. - -To immediately print the standard output produced by the test binary on the -command line, the ~-P~ option can be used. Argument to this option is the name -of the artifact that should be printed on the command line, in our case -~stdout~: - -#+BEGIN_SRC sh +``` + +Note that the target is correctly reported as tainted with `"test"`. It +will produce 3 additional actions for compiling, linking and running the +test binary. + +The result of the test target are 5 artifacts: `result` (containing +`UNKNOWN`, `PASS`, or `FAIL`), `stderr`, `stdout`, `time-start`, and +`time-stop`, and a single runfile (omitted in the output above), which +is a tree artifact with the name `test_greet` that contains all of the +above artifacts. The test was run successfully as otherwise all reported +artifacts would have been reported as `FAILED` in the output, and +justbuild would have returned the exit code `2`. + +To immediately print the standard output produced by the test binary on +the command line, the `-P` option can be used. Argument to this option +is the name of the artifact that should be printed on the command line, +in our case `stdout`: + +``` sh $ just-mr build tests greet --log-limit 1 -P stdout greet output: Hello World! greet output: Hello Universe! $ -#+END_SRC +``` -Note that ~--log-limit 1~ was just added to omit justbuild's ~INFO:~ prints. +Note that `--log-limit 1` was just added to omit justbuild's `INFO:` +prints. -Our test binary does not have any useful options for directly interacting -with it. When working with test frameworks, it sometimes can be desirable to -get hold of the test binary itself for manual interaction. The running of -the test binary is the last action associated with the test and the test -binary is, of course, one of its inputs. +Our test binary does not have any useful options for directly +interacting with it. When working with test frameworks, it sometimes can +be desirable to get hold of the test binary itself for manual +interaction. The running of the test binary is the last action +associated with the test and the test binary is, of course, one of its +inputs. -#+BEGIN_SRC sh +``` sh $ just-mr analyse --request-action-input -1 tests greet INFO: Requested target is [["@","tutorial","tests","greet"],{}] INFO: Request is input of action #-1 @@ -197,15 +201,15 @@ INFO: Result of input of action #-1 of target [["@","tutorial","tests","greet"], } INFO: Target tainted ["test"]. $ -#+END_SRC +``` The provided data also shows us the precise description of the action -for which we request the input. This allows us to manually rerun -the action. Or we can simply interact with the test binary manually -after installing the inputs to this action. Requesting the inputs -of an action can also be useful when debugging a build failure. +for which we request the input. This allows us to manually rerun the +action. Or we can simply interact with the test binary manually after +installing the inputs to this action. Requesting the inputs of an action +can also be useful when debugging a build failure. -#+BEGIN_SRC sh +``` sh $ just-mr install -o work --request-action-input -1 tests greet INFO: Requested target is [["@","tutorial","tests","greet"],{}] INFO: Request is input of action #-1 @@ -231,26 +235,25 @@ $ echo $? 0 $ cd .. $ rm -rf work -#+END_SRC +``` -** Creating a shell test +Creating a shell test +--------------------- -Similarly, to create a shell test for testing the ~helloworld~ binary, a test -script ~tests/test_helloworld.sh~ must be provided: +Similarly, to create a shell test for testing the `helloworld` binary, a +test script `tests/test_helloworld.sh` must be provided: -#+SRCNAME: tests/test_helloworld.sh -#+BEGIN_SRC sh +``` {.sh srcname="tests/test_helloworld.sh"} set -e [ "$(./helloworld)" = "Hello Universe!" ] -#+END_SRC +``` The test target for this shell tests uses the rule -~["@", "rules", "shell/test", "script"]~ and must depend on the ~"helloworld"~ -target. To create the test target, add the following to the ~tests/TARGETS~ -file: +`["@", "rules", "shell/test", "script"]` and must depend on the +`"helloworld"` target. To create the test target, add the following to +the `tests/TARGETS` file: -#+SRCNAME: tests/TARGETS -#+BEGIN_SRC js +``` {.jsonc srcname="tests/TARGETS"} ... , "helloworld": { "type": ["@", "rules", "shell/test", "script"] @@ -259,11 +262,11 @@ file: , "deps": [["", "helloworld"]] } ... -#+END_SRC +``` Now we can run the shell test (i.e., build the test result): -#+BEGIN_SRC sh +``` sh $ just-mr build tests helloworld INFO: Requested target is [["@","tutorial","tests","helloworld"],{}] INFO: Analysed target [["@","tutorial","tests","helloworld"],{}] @@ -281,29 +284,28 @@ INFO: Artifacts built, logical paths are: (1 runfiles omitted.) INFO: Target tainted ["test"]. $ -#+END_SRC - -The result is also similar, containing also the 5 artifacts and a single runfile -(omitted in the output above), which is a tree artifact with the name -~test_helloworld~ that contains all of the above artifacts. - -** Creating a compound test target - -As most people probably do not want to call every test target by hand, it is -desirable to compound test target that triggers the build of multiple test -reports. To do so, an ~"install"~ target can be used. The field ~"deps"~ of -an install target is a list of targets for which the runfiles are collected. -As for the tests the runfiles happen to be -tree artifacts named the same way as the test and containing all test results, -this is precisely what we need. -Furthermore, as the dependent test targets are tainted by ~"test"~, also the -compound test target must be tainted by the same string. To create the compound -test target combining the two tests above (the tests ~"greet"~ and -~"helloworld"~ from module ~"tests"~), add the following to the ~tests/TARGETS~ -file: - -#+SRCNAME: tests/TARGETS -#+BEGIN_SRC js +``` + +The result is also similar, containing also the 5 artifacts and a single +runfile (omitted in the output above), which is a tree artifact with the +name `test_helloworld` that contains all of the above artifacts. + +Creating a compound test target +------------------------------- + +As most people probably do not want to call every test target by hand, +it is desirable to compound test target that triggers the build of +multiple test reports. To do so, an `"install"` target can be used. The +field `"deps"` of an install target is a list of targets for which the +runfiles are collected. As for the tests the runfiles happen to be tree +artifacts named the same way as the test and containing all test +results, this is precisely what we need. Furthermore, as the dependent +test targets are tainted by `"test"`, also the compound test target must +be tainted by the same string. To create the compound test target +combining the two tests above (the tests `"greet"` and `"helloworld"` +from module `"tests"`), add the following to the `tests/TARGETS` file: + +``` {.jsonc srcname="tests/TARGETS"} ... , "ALL": { "type": "install" @@ -311,12 +313,12 @@ file: , "deps": ["greet", "helloworld"] } ... -#+END_SRC +``` -Now we can run all tests at once by just building the compound test target -~"ALL"~: +Now we can run all tests at once by just building the compound test +target `"ALL"`: -#+BEGIN_SRC sh +``` sh $ just-mr build tests ALL INFO: Requested target is [["@","tutorial","tests","ALL"],{}] INFO: Analysed target [["@","tutorial","tests","ALL"],{}] @@ -330,8 +332,8 @@ INFO: Artifacts built, logical paths are: test_helloworld [63fa5954161b52b275b05c270e1626feaa8e178b:177:t] INFO: Target tainted ["test"]. $ -#+END_SRC +``` -As a result it reports the runfiles (result directories) of both tests as -artifacts. Both tests ran successfully as none of those artifacts in this output -above are tagged as ~FAILED~. +As a result it reports the runfiles (result directories) of both tests +as artifacts. Both tests ran successfully as none of those artifacts in +this output above are tagged as `FAILED`. diff --git a/doc/tutorial/third-party-software.md b/doc/tutorial/third-party-software.md new file mode 100644 index 00000000..daaf5b2d --- /dev/null +++ b/doc/tutorial/third-party-software.md @@ -0,0 +1,473 @@ +Building Third-party Software +============================= + +Third-party projects usually ship with their own build description, +which often happens to be not compatible with justbuild. Nevertheless, +it is highly desireable to include external projects via their source +code base, instead of relying on the integration of out-of-band binary +distributions. justbuild offers a flexible approach to provide the +required build description via an overlay layer without the need to +touch the original code base. + +For the remainder of this section, we expect to have the project files +available resulting from successfully completing the tutorial section on +*Building C++ Hello World*. We will demonstrate how to use the +open-source project [fmtlib](https://github.com/fmtlib/fmt) as an +example for integrating third-party software to a justbuild project. + +Creating the target overlay layer for fmtlib +-------------------------------------------- + +Before we construct the overlay layer for fmtlib, we need to determine +its file structure ([tag +8.1.1](https://github.com/fmtlib/fmt/tree/8.1.1)). The relevant header +and source files are structured as follows: + + fmt + | + +--include + | +--fmt + | +--*.h + | + +--src + +--format.cc + +--os.cc + +The public headers can be found in `include/fmt`, while the library's +source files are located in `src`. For the overlay layer, the `TARGETS` +files should be placed in a tree structure that resembles the original +code base's structure. It is also good practice to provide a top-level +`TARGETS` file, leading to the following structure for the overlay: + + fmt-layer + | + +--TARGETS + +--include + | +--fmt + | +--TARGETS + | + +--src + +--TARGETS + +Let's create the overlay structure: + +``` sh +$ mkdir -p fmt-layer/include/fmt +$ mkdir -p fmt-layer/src +``` + +The directory `include/fmt` contains only header files. As we want all +files in this directory to be included in the `"hdrs"` target, we can +safely use the explicit `TREE` reference[^1], which collects, in a +single artifact (describing a directory) *all* directory contents from +`"."` of the workspace root. Note that the `TARGETS` file is only part +of the overlay, and therefore will not be part of this tree. +Furthermore, this tree should be staged to `"fmt"`, so that any consumer +can include those headers via `<fmt/...>`. The resulting header +directory target `"hdrs"` in `include/fmt/TARGETS` should be described +as: + +``` {.jsonc srcname="fmt-layer/include/fmt/TARGETS"} +{ "hdrs": + { "type": ["@", "rules", "data", "staged"] + , "srcs": [["TREE", null, "."]] + , "stage": ["fmt"] + } +} +``` + +The actual library target is defined in the directory `src`. For the +public headers, it refers to the previously created `"hdrs"` target via +its fully-qualified target name (`["include/fmt", "hdrs"]`). Source +files are the two local files `format.cc`, and `os.cc`. The final target +description in `src/TARGETS` will look like this: + +``` {.jsonc srcname="fmt-layer/src/TARGETS"} +{ "fmt": + { "type": ["@", "rules", "CC", "library"] + , "name": ["fmt"] + , "hdrs": [["include/fmt", "hdrs"]] + , "srcs": ["format.cc", "os.cc"] + } +} +``` + +Finally, the top-level `TARGETS` file can be created. While it is +technically not strictly required, it is considered good practice to +*export* every target that may be used by another project. Exported +targets are subject to high-level target caching, which allows to skip +the analysis and traversal of entire subgraphs in the action graph. +Therefore, we create an export target that exports the target +`["src", "fmt"]`, with only the variables in the field +`"flexible_config"` being propagated. The top-level `TARGETS` file +contains the following content: + +``` {.jsonc srcname="fmt-layer/TARGETS"} +{ "fmt": + { "type": "export" + , "target": ["src", "fmt"] + , "flexible_config": ["CXX", "CXXFLAGS", "ADD_CXXFLAGS", "AR", "ENV"] + } +} +``` + +After adding the library to the multi-repository configuration (next +step), the list of configuration variables a target, like `["src", +"fmt"]`, actually depends on can be obtained using the `--dump-vars` +option of the `analyse` subcommand. In this way, an informed decision +can be taken when deciding which variables of the export target to make +tunable for the consumer. + +Adding fmtlib to the Multi-Repository Configuration +--------------------------------------------------- + +Based on the *hello world* tutorial, we can extend the existing +`repos.json` by the layer definition `"fmt-targets-layer"` and the +repository `"fmtlib"`, which is based on the Git repository with its +target root being overlayed. Furthermore, we want to use `"fmtlib"` in +the repository `"tutorial"`, and therefore need to introduce an +additional binding `"format"` for it: + +``` {.jsonc srcname="repos.json"} +{ "main": "tutorial" +, "repositories": + { "rules-cc": + { "repository": + { "type": "git" + , "branch": "master" + , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" + , "repository": "https://github.com/just-buildsystem/rules-cc.git" + , "subdir": "rules" + } + , "target_root": "tutorial-defaults" + , "rule_root": "rules-cc" + } + , "tutorial": + { "repository": {"type": "file", "path": "."} + , "bindings": {"rules": "rules-cc", "format": "fmtlib"} + } + , "tutorial-defaults": + { "repository": {"type": "file", "path": "./tutorial-defaults"} + } + , "fmt-targets-layer": + { "repository": {"type": "file", "path": "./fmt-layer"} + } + , "fmtlib": + { "repository": + { "type": "git" + , "branch": "8.1.1" + , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9" + , "repository": "https://github.com/fmtlib/fmt.git" + } + , "target_root": "fmt-targets-layer" + , "bindings": {"rules": "rules-cc"} + } + } +} +``` + +This `"format"` binding can you be used to add a new private dependency +in `greet/TARGETS`: + +``` {.jsonc srcname="greet/TARGETS"} +{ "greet": + { "type": ["@", "rules", "CC", "library"] + , "name": ["greet"] + , "hdrs": ["greet.hpp"] + , "srcs": ["greet.cpp"] + , "stage": ["greet"] + , "private-deps": [["@", "format", "", "fmt"]] + } +} +``` + +Consequently, the `fmtlib` library can now be used by `greet/greet.cpp`: + +``` {.cpp srcname="greet/greet.cpp"} +#include "greet.hpp" +#include <fmt/format.h> + +void greet(std::string const& s) { + fmt::print("Hello {}!\n", s); +} +``` + +Due to changes made to `repos.json`, building this tutorial requires to +rerun `just-mr`, which will fetch the necessary sources for the external +repositories: + +``` sh +$ just-mr build helloworld +INFO: Requested target is [["@","tutorial","","helloworld"],{}] +INFO: Analysed target [["@","tutorial","","helloworld"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 1 not eligible for caching +INFO: Discovered 7 actions, 3 trees, 0 blobs +INFO: Building [["@","tutorial","","helloworld"],{}]. +INFO: Processed 7 actions, 1 cache hits. +INFO: Artifacts built, logical paths are: + helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x] +$ +``` + +Note to build the `fmt` target alone, its containing repository `fmtlib` +must be specified via the `--main` option: + +``` sh +$ just-mr --main fmtlib build fmt +INFO: Requested target is [["@","fmtlib","","fmt"],{}] +INFO: Analysed target [["@","fmtlib","","fmt"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 1 not eligible for caching +INFO: Discovered 3 actions, 1 trees, 0 blobs +INFO: Building [["@","fmtlib","","fmt"],{}]. +INFO: Processed 3 actions, 3 cache hits. +INFO: Artifacts built, logical paths are: + libfmt.a [513b2ac17c557675fc841f3ebf279003ff5a73ae:240914:f] + (1 runfiles omitted.) +$ +``` + +Employing high-level target caching +----------------------------------- + +The make use of high-level target caching for exported targets, we need +to ensure that all inputs to an export target are transitively +content-fixed. This is automatically the case for `"type":"git"` +repositories. However, the `libfmt` repository also depends on +`"rules-cc"`, `"tutorial-defaults"`, and `"fmt-target-layer"`. As the +latter two are `"type":"file"` repositories, they must be put under Git +versioning first: + +``` sh +$ git init . +$ git add tutorial-defaults fmt-layer +$ git commit -m"fix compile flags and fmt targets layer" +``` + +Note that `rules-cc` already is under Git versioning. + +Now, to instruct `just-mr` to use the content-fixed, committed source +trees of those `"type":"file"` repositories the pragma `"to_git"` must +be set for them in `repos.json`: + +``` {.jsonc srcname="repos.json"} +{ "main": "tutorial" +, "repositories": + { "rules-cc": + { "repository": + { "type": "git" + , "branch": "master" + , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" + , "repository": "https://github.com/just-buildsystem/rules-cc.git" + , "subdir": "rules" + } + , "target_root": "tutorial-defaults" + , "rule_root": "rules-cc" + } + , "tutorial": + { "repository": {"type": "file", "path": "."} + , "bindings": {"rules": "rules-cc", "format": "fmtlib"} + } + , "tutorial-defaults": + { "repository": + { "type": "file" + , "path": "./tutorial-defaults" + , "pragma": {"to_git": true} + } + } + , "fmt-targets-layer": + { "repository": + { "type": "file" + , "path": "./fmt-layer" + , "pragma": {"to_git": true} + } + } + , "fmtlib": + { "repository": + { "type": "git" + , "branch": "master" + , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9" + , "repository": "https://github.com/fmtlib/fmt.git" + } + , "target_root": "fmt-targets-layer" + , "bindings": {"rules": "rules-cc"} + } + } +} +``` + +Due to changes in the repository configuration, we need to rebuild and +the benefits of the target cache should be visible on the second build: + +``` sh +$ just-mr build helloworld +INFO: Requested target is [["@","tutorial","","helloworld"],{}] +INFO: Analysed target [["@","tutorial","","helloworld"],{}] +INFO: Export targets found: 0 cached, 1 uncached, 0 not eligible for caching +INFO: Discovered 7 actions, 3 trees, 0 blobs +INFO: Building [["@","tutorial","","helloworld"],{}]. +INFO: Processed 7 actions, 7 cache hits. +INFO: Artifacts built, logical paths are: + helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x] +$ +$ just-mr build helloworld +INFO: Requested target is [["@","tutorial","","helloworld"],{}] +INFO: Analysed target [["@","tutorial","","helloworld"],{}] +INFO: Export targets found: 1 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 4 actions, 2 trees, 0 blobs +INFO: Building [["@","tutorial","","helloworld"],{}]. +INFO: Processed 4 actions, 4 cache hits. +INFO: Artifacts built, logical paths are: + helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x] +$ +``` + +Note that in the second run the export target `"fmt"` was taken from +cache and its 3 actions were eliminated, as their result has been +recorded to the high-level target cache during the first run. + +Combining overlay layers for multiple projects +---------------------------------------------- + +Projects typically depend on multiple external repositories. Creating an +overlay layer for each external repository might unnecessarily clutter +up the repository configuration and the file structure of your +repository. One solution to mitigate this issue is to combine the +`TARGETS` files of multiple external repositories in a single overlay +layer. To avoid conflicts, the `TARGETS` files can be assigned different +file names per repository. As an example, imagine a common overlay layer +with the files `TARGETS.fmt` and `TARGETS.gsl` for the repositories +`"fmtlib"` and `"gsl-lite"`, respectively: + + common-layer + | + +--TARGETS.fmt + +--TARGETS.gsl + +--include + | +--fmt + | | +--TARGETS.fmt + | +--gsl + | +--TARGETS.gsl + | + +--src + +--TARGETS.fmt + +Such a common overlay layer can be used as the target root for both +repositories with only one difference: the `"target_file_name"` field. +By specifying this field, the dispatch where to find the respective +target description for each repository is implemented. For the given +example, the following `repos.json` defines the overlay +`"common-targets-layer"`, which is used by `"fmtlib"` and `"gsl-lite"`: + +``` {.jsonc srcname="repos.json"} +{ "main": "tutorial" +, "repositories": + { "rules-cc": + { "repository": + { "type": "git" + , "branch": "master" + , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" + , "repository": "https://github.com/just-buildsystem/rules-cc.git" + , "subdir": "rules" + } + , "target_root": "tutorial-defaults" + , "rule_root": "rules-cc" + } + , "tutorial": + { "repository": {"type": "file", "path": "."} + , "bindings": {"rules": "rules-cc", "format": "fmtlib"} + } + , "tutorial-defaults": + { "repository": + { "type": "file" + , "path": "./tutorial-defaults" + , "pragma": {"to_git": true} + } + } + , "common-targets-layer": + { "repository": + { "type": "file" + , "path": "./common-layer" + , "pragma": {"to_git": true} + } + } + , "fmtlib": + { "repository": + { "type": "git" + , "branch": "8.1.1" + , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9" + , "repository": "https://github.com/fmtlib/fmt.git" + } + , "target_root": "common-targets-layer" + , "target_file_name": "TARGETS.fmt" + , "bindings": {"rules": "rules-cc"} + } + , "gsl-lite": + { "repository": + { "type": "git" + , "branch": "v0.40.0" + , "commit": "d6c8af99a1d95b3db36f26b4f22dc3bad89952de" + , "repository": "https://github.com/gsl-lite/gsl-lite.git" + } + , "target_root": "common-targets-layer" + , "target_file_name": "TARGETS.gsl" + , "bindings": {"rules": "rules-cc"} + } + } +} +``` + +Using pre-built dependencies +---------------------------- + +While building external dependencies from source brings advantages, most +prominently the flexibility to quickly and seamlessly switch to a +different build configuration (production, debug, instrumented for +performance analysis; cross-compiling for a different target +architecture), there are also legitimate reasons to use pre-built +dependencies. The most prominent one is if your project is packaged as +part of a larger distribution. For that reason, just also has (in +`etc/import.prebuilt`) target files for all its dependencies assuming +they are pre-installed. The reason why target files are used at all for +this situation is twofold. + + - On the one hand, having a target allows the remaining targets to not + care about where their dependencies come from, or if it is a build + against pre-installed dependencies or not. Also, the top-level + binary does not have to know the linking requirements of its + transitive dependencies. In other words, information stays where it + belongs to and if one target acquires a new dependency, the + information is automatically propagated to all targets using it. + - Still some information is needed to use a pre-installed library and, + as explained, a target describing the pre-installed library is the + right place to collect this information. + - The public header files of the library. By having this explicit, + we do not accumulate directories in the include search path and + hence also properly detect include conflicts. + - The information on how to link the library itself (i.e., + basically its base name). + - Any dependencies on other libraries that the library might have. + This information is used to obtain the correct linking order and + complete transitive linking arguments while keeping the + description maintainable, as each target still only declares its + direct dependencies. + +The target description for a pre-built version of the format library +that was used as an example in this section is shown next; with our +staging mechanism the logical repository it belongs to is rooted in the +`fmt` subdirectory of the `include` directory of the ambient system. + +``` {.jsonc srcname="etc/import.prebuilt/TARGETS.fmt"} +{ "fmt": + { "type": ["@", "rules", "CC", "library"] + , "name": ["fmt"] + , "stage": ["fmt"] + , "hdrs": [["TREE", null, "."]] + , "private-ldflags": ["-lfmt"] + } +} +``` + +[^1]: Explicit `TREE` references are always a list of length 3, to + distinguish them from target references of length 2 (module and + target name). Furthermore, the second list element is always `null` + as we only want to allow tree references from the current module. diff --git a/doc/tutorial/third-party-software.org b/doc/tutorial/third-party-software.org deleted file mode 100644 index d1712cc8..00000000 --- a/doc/tutorial/third-party-software.org +++ /dev/null @@ -1,475 +0,0 @@ -* Building Third-party Software - -Third-party projects usually ship with their own build description, which often -happens to be not compatible with justbuild. Nevertheless, it is highly -desireable to include external projects via their source code base, instead of -relying on the integration of out-of-band binary distributions. justbuild offers -a flexible approach to provide the required build description via an overlay -layer without the need to touch the original code base. - -For the remainder of this section, we expect to have the project files available -resulting from successfully completing the tutorial section on /Building C++ -Hello World/. We will demonstrate how to use the open-source project -[[https://github.com/fmtlib/fmt][fmtlib]] as an example for integrating -third-party software to a justbuild project. - -** Creating the target overlay layer for fmtlib - -Before we construct the overlay layer for fmtlib, we need to determine its file -structure ([[https://github.com/fmtlib/fmt/tree/8.1.1][tag 8.1.1]]). The -relevant header and source files are structured as follows: - -#+BEGIN_SRC - fmt - | - +--include - | +--fmt - | +--*.h - | - +--src - +--format.cc - +--os.cc -#+END_SRC - -The public headers can be found in ~include/fmt~, while the library's source -files are located in ~src~. For the overlay layer, the ~TARGETS~ files should be -placed in a tree structure that resembles the original code base's structure. -It is also good practice to provide a top-level ~TARGETS~ file, leading to the -following structure for the overlay: - -#+BEGIN_SRC - fmt-layer - | - +--TARGETS - +--include - | +--fmt - | +--TARGETS - | - +--src - +--TARGETS -#+END_SRC - -Let's create the overlay structure: - -#+BEGIN_SRC sh -$ mkdir -p fmt-layer/include/fmt -$ mkdir -p fmt-layer/src -#+END_SRC - -The directory ~include/fmt~ contains only header files. As we want all files in -this directory to be included in the ~"hdrs"~ target, we can safely -use the explicit ~TREE~ reference[fn:1], which collects, in a single -artifact (describing a directory) /all/ directory contents -from ~"."~ of the workspace root. Note that the ~TARGETS~ file is only part of -the overlay, and -therefore will not be part of this tree. Furthermore, this tree should be staged -to ~"fmt"~, so that any consumer can include those headers via ~<fmt/...>~. The -resulting header directory target ~"hdrs"~ in ~include/fmt/TARGETS~ should be -described as: - -[fn:1] Explicit ~TREE~ references are always a list of length 3, to distinguish -them from target references of length 2 (module and target name). Furthermore, -the second list element is always ~null~ as we only want to allow tree -references from the current module. - - -#+SRCNAME: fmt-layer/include/fmt/TARGETS -#+BEGIN_SRC js -{ "hdrs": - { "type": ["@", "rules", "data", "staged"] - , "srcs": [["TREE", null, "."]] - , "stage": ["fmt"] - } -} -#+END_SRC - -The actual library target is defined in the directory ~src~. For the public -headers, it refers to the previously created ~"hdrs"~ target via its -fully-qualified target name (~["include/fmt", "hdrs"]~). Source files are the -two local files ~format.cc~, and ~os.cc~. The final target description in -~src/TARGETS~ will look like this: - -#+SRCNAME: fmt-layer/src/TARGETS -#+BEGIN_SRC js -{ "fmt": - { "type": ["@", "rules", "CC", "library"] - , "name": ["fmt"] - , "hdrs": [["include/fmt", "hdrs"]] - , "srcs": ["format.cc", "os.cc"] - } -} -#+END_SRC - -Finally, the top-level ~TARGETS~ file can be created. While it is technically -not strictly required, it is considered good practice to /export/ every target -that may be used by another project. Exported targets are subject to high-level -target caching, which allows to skip the analysis and traversal of entire -subgraphs in the action graph. Therefore, we create an export target that -exports the target ~["src", "fmt"]~, with only the variables in the field -~"flexible_config"~ being propagated. The top-level ~TARGETS~ file contains the -following content: - -#+SRCNAME: fmt-layer/TARGETS -#+BEGIN_SRC js -{ "fmt": - { "type": "export" - , "target": ["src", "fmt"] - , "flexible_config": ["CXX", "CXXFLAGS", "ADD_CXXFLAGS", "AR", "ENV"] - } -} -#+END_SRC - -After adding the library to the multi-repository configuration (next -step), the list of configuration variables a target, like ~["src", -"fmt"]~, actually depends on can be obtained using the ~--dump-vars~ -option of the ~analyse~ subcommand. In this way, an informed decision -can be taken when deciding which variables of the export target to -make tunable for the consumer. - -** Adding fmtlib to the Multi-Repository Configuration - -Based on the /hello world/ tutorial, we can extend the existing ~repos.json~ by -the layer definition ~"fmt-targets-layer"~ and the repository ~"fmtlib"~, which -is based on the Git repository with its target root being overlayed. -Furthermore, we want to use ~"fmtlib"~ in the repository ~"tutorial"~, and -therefore need to introduce an additional binding ~"format"~ for it: - -#+SRCNAME: repos.json -#+BEGIN_SRC js -{ "main": "tutorial" -, "repositories": - { "rules-cc": - { "repository": - { "type": "git" - , "branch": "master" - , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" - , "repository": "https://github.com/just-buildsystem/rules-cc.git" - , "subdir": "rules" - } - , "target_root": "tutorial-defaults" - , "rule_root": "rules-cc" - } - , "tutorial": - { "repository": {"type": "file", "path": "."} - , "bindings": {"rules": "rules-cc", "format": "fmtlib"} - } - , "tutorial-defaults": - { "repository": {"type": "file", "path": "./tutorial-defaults"} - } - , "fmt-targets-layer": - { "repository": {"type": "file", "path": "./fmt-layer"} - } - , "fmtlib": - { "repository": - { "type": "git" - , "branch": "8.1.1" - , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9" - , "repository": "https://github.com/fmtlib/fmt.git" - } - , "target_root": "fmt-targets-layer" - , "bindings": {"rules": "rules-cc"} - } - } -} -#+END_SRC - -This ~"format"~ binding can you be used to add a new private dependency in -~greet/TARGETS~: - -#+SRCNAME: greet/TARGETS -#+BEGIN_SRC js -{ "greet": - { "type": ["@", "rules", "CC", "library"] - , "name": ["greet"] - , "hdrs": ["greet.hpp"] - , "srcs": ["greet.cpp"] - , "stage": ["greet"] - , "private-deps": [["@", "format", "", "fmt"]] - } -} -#+END_SRC - -Consequently, the ~fmtlib~ library can now be used by ~greet/greet.cpp~: - -#+SRCNAME: greet/greet.cpp -#+BEGIN_SRC cpp -#include "greet.hpp" -#include <fmt/format.h> - -void greet(std::string const& s) { - fmt::print("Hello {}!\n", s); -} -#+END_SRC - -Due to changes made to ~repos.json~, building this tutorial requires to rerun -~just-mr~, which will fetch the necessary sources for the external repositories: - -#+BEGIN_SRC sh -$ just-mr build helloworld -INFO: Requested target is [["@","tutorial","","helloworld"],{}] -INFO: Analysed target [["@","tutorial","","helloworld"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 1 not eligible for caching -INFO: Discovered 7 actions, 3 trees, 0 blobs -INFO: Building [["@","tutorial","","helloworld"],{}]. -INFO: Processed 7 actions, 1 cache hits. -INFO: Artifacts built, logical paths are: - helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x] -$ -#+END_SRC - -Note to build the ~fmt~ target alone, its containing repository ~fmtlib~ must be -specified via the ~--main~ option: -#+BEGIN_SRC sh -$ just-mr --main fmtlib build fmt -INFO: Requested target is [["@","fmtlib","","fmt"],{}] -INFO: Analysed target [["@","fmtlib","","fmt"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 1 not eligible for caching -INFO: Discovered 3 actions, 1 trees, 0 blobs -INFO: Building [["@","fmtlib","","fmt"],{}]. -INFO: Processed 3 actions, 3 cache hits. -INFO: Artifacts built, logical paths are: - libfmt.a [513b2ac17c557675fc841f3ebf279003ff5a73ae:240914:f] - (1 runfiles omitted.) -$ -#+END_SRC - -** Employing high-level target caching - -The make use of high-level target caching for exported targets, we need to -ensure that all inputs to an export target are transitively content-fixed. This -is automatically the case for ~"type":"git"~ repositories. However, the ~libfmt~ -repository also depends on ~"rules-cc"~, ~"tutorial-defaults"~, and -~"fmt-target-layer"~. As the latter two are ~"type":"file"~ repositories, they -must be put under Git versioning first: - -#+BEGIN_SRC sh -$ git init . -$ git add tutorial-defaults fmt-layer -$ git commit -m"fix compile flags and fmt targets layer" -#+END_SRC - -Note that ~rules-cc~ already is under Git versioning. - -Now, to instruct ~just-mr~ to use the content-fixed, committed source trees of -those ~"type":"file"~ repositories the pragma ~"to_git"~ must be set for them in -~repos.json~: - -#+SRCNAME: repos.json -#+BEGIN_SRC js -{ "main": "tutorial" -, "repositories": - { "rules-cc": - { "repository": - { "type": "git" - , "branch": "master" - , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" - , "repository": "https://github.com/just-buildsystem/rules-cc.git" - , "subdir": "rules" - } - , "target_root": "tutorial-defaults" - , "rule_root": "rules-cc" - } - , "tutorial": - { "repository": {"type": "file", "path": "."} - , "bindings": {"rules": "rules-cc", "format": "fmtlib"} - } - , "tutorial-defaults": - { "repository": - { "type": "file" - , "path": "./tutorial-defaults" - , "pragma": {"to_git": true} - } - } - , "fmt-targets-layer": - { "repository": - { "type": "file" - , "path": "./fmt-layer" - , "pragma": {"to_git": true} - } - } - , "fmtlib": - { "repository": - { "type": "git" - , "branch": "master" - , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9" - , "repository": "https://github.com/fmtlib/fmt.git" - } - , "target_root": "fmt-targets-layer" - , "bindings": {"rules": "rules-cc"} - } - } -} -#+END_SRC - -Due to changes in the repository configuration, we need to rebuild and the -benefits of the target cache should be visible on the second build: - -#+BEGIN_SRC sh -$ just-mr build helloworld -INFO: Requested target is [["@","tutorial","","helloworld"],{}] -INFO: Analysed target [["@","tutorial","","helloworld"],{}] -INFO: Export targets found: 0 cached, 1 uncached, 0 not eligible for caching -INFO: Discovered 7 actions, 3 trees, 0 blobs -INFO: Building [["@","tutorial","","helloworld"],{}]. -INFO: Processed 7 actions, 7 cache hits. -INFO: Artifacts built, logical paths are: - helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x] -$ -$ just-mr build helloworld -INFO: Requested target is [["@","tutorial","","helloworld"],{}] -INFO: Analysed target [["@","tutorial","","helloworld"],{}] -INFO: Export targets found: 1 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 4 actions, 2 trees, 0 blobs -INFO: Building [["@","tutorial","","helloworld"],{}]. -INFO: Processed 4 actions, 4 cache hits. -INFO: Artifacts built, logical paths are: - helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x] -$ -#+END_SRC - -Note that in the second run the export target ~"fmt"~ was taken from cache and -its 3 actions were eliminated, as their result has been recorded to the -high-level target cache during the first run. - -** Combining overlay layers for multiple projects - -Projects typically depend on multiple external repositories. Creating an overlay -layer for each external repository might unnecessarily clutter up the repository -configuration and the file structure of your repository. One solution to -mitigate this issue is to combine the ~TARGETS~ files of multiple external -repositories in a single overlay layer. To avoid conflicts, the ~TARGETS~ files -can be assigned different file names per repository. As an example, imagine a -common overlay layer with the files ~TARGETS.fmt~ and ~TARGETS.gsl~ for the -repositories ~"fmtlib"~ and ~"gsl-lite"~, respectively: - -#+BEGIN_SRC - common-layer - | - +--TARGETS.fmt - +--TARGETS.gsl - +--include - | +--fmt - | | +--TARGETS.fmt - | +--gsl - | +--TARGETS.gsl - | - +--src - +--TARGETS.fmt -#+END_SRC - -Such a common overlay layer can be used as the target root for both repositories -with only one difference: the ~"target_file_name"~ field. By specifying this -field, the dispatch where to find the respective target description for each -repository is implemented. For the given example, the following ~repos.json~ -defines the overlay ~"common-targets-layer"~, which is used by ~"fmtlib"~ and -~"gsl-lite"~: - -#+SRCNAME: repos.json -#+BEGIN_SRC js -{ "main": "tutorial" -, "repositories": - { "rules-cc": - { "repository": - { "type": "git" - , "branch": "master" - , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" - , "repository": "https://github.com/just-buildsystem/rules-cc.git" - , "subdir": "rules" - } - , "target_root": "tutorial-defaults" - , "rule_root": "rules-cc" - } - , "tutorial": - { "repository": {"type": "file", "path": "."} - , "bindings": {"rules": "rules-cc", "format": "fmtlib"} - } - , "tutorial-defaults": - { "repository": - { "type": "file" - , "path": "./tutorial-defaults" - , "pragma": {"to_git": true} - } - } - , "common-targets-layer": - { "repository": - { "type": "file" - , "path": "./common-layer" - , "pragma": {"to_git": true} - } - } - , "fmtlib": - { "repository": - { "type": "git" - , "branch": "8.1.1" - , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9" - , "repository": "https://github.com/fmtlib/fmt.git" - } - , "target_root": "common-targets-layer" - , "target_file_name": "TARGETS.fmt" - , "bindings": {"rules": "rules-cc"} - } - , "gsl-lite": - { "repository": - { "type": "git" - , "branch": "v0.40.0" - , "commit": "d6c8af99a1d95b3db36f26b4f22dc3bad89952de" - , "repository": "https://github.com/gsl-lite/gsl-lite.git" - } - , "target_root": "common-targets-layer" - , "target_file_name": "TARGETS.gsl" - , "bindings": {"rules": "rules-cc"} - } - } -} -#+END_SRC - -** Using pre-built dependencies - -While building external dependencies from source brings advantages, -most prominently the flexibility to quickly and seamlessly switch -to a different build configuration (production, debug, instrumented -for performance analysis; cross-compiling for a different target -architecture), there are also legitimate reasons to use pre-built -dependencies. The most prominent one is if your project is packaged -as part of a larger distribution. For that reason, just also has (in -~etc/import.prebuilt~) target files for all its dependencies assuming -they are pre-installed. The reason why target files are used at -all for this situation is twofold. -- On the one hand, having a target allows the remaining targets - to not care about where their dependencies come from, or if it - is a build against pre-installed dependencies or not. Also, the - top-level binary does not have to know the linking requirements - of its transitive dependencies. In other words, information stays - where it belongs to and if one target acquires a new dependency, - the information is automatically propagated to all targets using it. -- Still some information is needed to use a pre-installed library - and, as explained, a target describing the pre-installed library - is the right place to collect this information. - - The public header files of the library. By having this explicit, - we do not accumulate directories in the include search path - and hence also properly detect include conflicts. - - The information on how to link the library itself (i.e., - basically its base name). - - Any dependencies on other libraries that the library might have. - This information is used to obtain the correct linking order - and complete transitive linking arguments while keeping the - description maintainable, as each target still only declares - its direct dependencies. - -The target description for a pre-built version of the format -library that was used as an example in this section is shown next; -with our staging mechanism the logical repository it belongs to is -rooted in the ~fmt~ subdirectory of the ~include~ directory of the -ambient system. - -#+SRCNAME: etc/import.prebuilt/TARGETS.fmt -#+BEGIN_SRC js -{ "fmt": - { "type": ["@", "rules", "CC", "library"] - , "name": ["fmt"] - , "stage": ["fmt"] - , "hdrs": [["TREE", null, "."]] - , "private-ldflags": ["-lfmt"] - } -} -#+END_SRC |