doc: Convert orgmode files to markdown

1 files changed, 339 insertions, 0 deletions

diff --git a/doc/tutorial/tests.md b/doc/tutorial/tests.md
new file mode 100644
index 00000000..138769b1
--- /dev/null
+++ b/doc/tutorial/tests.md
@@ -0,0 +1,339 @@
+Creating Tests
+==============
+
+To run tests with justbuild, we do *not* have a dedicated `test`
+subcommand. Instead, we consider tests being a specific action that
+generates a test report. Consequently, we use the `build` subcommand to
+build the test report, and thereby run the test action. Test actions,
+however, are slightly different from normal actions in that we don't
+want the build of the test report to be aborted if a test action fails
+(but still, we want only successfully actions taken from cache). Rules
+defining targets containing such special actions have to identify
+themselves as *tainted* by specifying a string explaining why such
+special actions are justified; in our case, the string is `"test"`.
+Besides the implicit marking by using a tainted rule, those tainting
+strings can also be explicitly assigned by the user in the definition of
+a target, e.g., to mark test data. Any target has to be tainted with (at
+least) all the strings any of its dependencies is tainted with. In this
+way, it is ensured that no test target will end up in a production
+build.
+
+For the remainder of this section, we expect to have the project files
+available resulting from successfully completing the tutorial section on
+*Building C++ Hello World*. We will demonstrate how to write a test
+binary for the `greet` library and a shell test for the `helloworld`
+binary.
+
+Creating a C++ test binary
+--------------------------
+
+First, we will create a C++ test binary for testing the correct
+functionality of the `greet` library. Therefore, we need to provide a
+C++ source file that performs the actual testing and returns non-`0` on
+failure. For simplicity reasons, we do not use a testing framework for
+this tutorial. A simple test that captures standard output and verifies
+it with the expected output should be provided in the file
+`tests/greet.test.cpp`:
+
+``` {.cpp srcname="tests/greet.test.cpp"}
+#include <functional>
+#include <iostream>
+#include <string>
+#include <unistd.h>
+#include "greet/greet.hpp"
+
+template <std::size_t kMaxBufSize = 1024>
+auto capture_stdout(std::function<void()> const& func) -> std::string {
+  int fd[2];
+  if (pipe(fd) < 0) return {};
+  int fd_stdout = dup(fileno(stdout));
+  fflush(stdout);
+  dup2(fd[1], fileno(stdout));
+
+  func();
+  fflush(stdout);
+
+  std::string buf(kMaxBufSize, '\0');
+  auto n = read(fd[0], &(*buf.begin()), kMaxBufSize);
+  close(fd[0]);
+  close(fd[1]);
+  dup2(fd_stdout, fileno(stdout));
+  return buf.substr(0, n);
+}
+
+auto test_greet(std::string const& name) -> bool {
+  auto expect = std::string{"Hello "} + name + "!\n";
+  auto result = capture_stdout([&name] { greet(name); });
+  std::cout << "greet output: " << result << std::endl;
+  return result == expect;
+}
+
+int main() {
+  return test_greet("World") && test_greet("Universe") ? 0 : 1;
+}
+```
+
+Next, a new test target needs to be created in module `greet`. This
+target uses the rule `["@", "rules", "CC/test", "test"]` and needs to
+depend on the `["greet", "greet"]` target. To create the test target,
+add the following to `tests/TARGETS`:
+
+``` {.jsonc srcname="tests/TARGETS"}
+{ "greet":
+  { "type": ["@", "rules", "CC/test", "test"]
+  , "name": ["test_greet"]
+  , "srcs": ["greet.test.cpp"]
+  , "private-deps": [["greet", "greet"]]
+  }
+}
+```
+
+Before we can run the test, a proper default module for `CC/test` must
+be provided. By specifying the appropriate target in this module the
+default test runner can be overwritten by a different test runner fom
+the rule's workspace root. Moreover, all test targets share runner
+infrastructure from `shell/test`, e.g., summarizing multiple runs per
+test (to detect flakyness) if the configuration variable `RUNS_PER_TEST`
+is set.
+
+However, in our case, we want to use the default runner and therefore it
+is sufficient to create an empty module. To do so, create the file
+`tutorial-defaults/CC/test/TARGETS` with content
+
+``` {.jsonc srcname="tutorial-defaults/CC/test/TARGETS"}
+{}
+```
+
+as well as the file `tutorial-defaults/shell/test/TARGETS` with content
+
+``` {.jsonc srcname="tutorial-defaults/shell/test/TARGETS"}
+{}
+```
+
+Now we can run the test (i.e., build the test result):
+
+``` sh
+$ just-mr build tests greet
+INFO: Requested target is [["@","tutorial","tests","greet"],{}]
+INFO: Analysed target [["@","tutorial","tests","greet"],{}]
+INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
+INFO: Target tainted ["test"].
+INFO: Discovered 5 actions, 3 trees, 1 blobs
+INFO: Building [["@","tutorial","tests","greet"],{}].
+INFO: Processed 5 actions, 2 cache hits.
+INFO: Artifacts built, logical paths are:
+        result [7ef22e9a431ad0272713b71fdc8794016c8ef12f:5:f]
+        stderr [8b137891791fe96927ad78e64b0aad7bded08bdc:1:f]
+        stdout [ae6c6813755da67954a4a562f6d2ef01578c3e89:60:f]
+        time-start [8e7a68af8d5d7a6d0036c1126ff5c16a5045ae95:11:f]
+        time-stop [8e7a68af8d5d7a6d0036c1126ff5c16a5045ae95:11:f]
+      (1 runfiles omitted.)
+INFO: Target tainted ["test"].
+$
+```
+
+Note that the target is correctly reported as tainted with `"test"`. It
+will produce 3 additional actions for compiling, linking and running the
+test binary.
+
+The result of the test target are 5 artifacts: `result` (containing
+`UNKNOWN`, `PASS`, or `FAIL`), `stderr`, `stdout`, `time-start`, and
+`time-stop`, and a single runfile (omitted in the output above), which
+is a tree artifact with the name `test_greet` that contains all of the
+above artifacts. The test was run successfully as otherwise all reported
+artifacts would have been reported as `FAILED` in the output, and
+justbuild would have returned the exit code `2`.
+
+To immediately print the standard output produced by the test binary on
+the command line, the `-P` option can be used. Argument to this option
+is the name of the artifact that should be printed on the command line,
+in our case `stdout`:
+
+``` sh
+$ just-mr build tests greet --log-limit 1 -P stdout
+greet output: Hello World!
+
+greet output: Hello Universe!
+
+$
+```
+
+Note that `--log-limit 1` was just added to omit justbuild's `INFO:`
+prints.
+
+Our test binary does not have any useful options for directly
+interacting with it. When working with test frameworks, it sometimes can
+be desirable to get hold of the test binary itself for manual
+interaction. The running of the test binary is the last action
+associated with the test and the test binary is, of course, one of its
+inputs.
+
+``` sh
+$ just-mr analyse --request-action-input -1 tests greet
+INFO: Requested target is [["@","tutorial","tests","greet"],{}]
+INFO: Request is input of action #-1
+INFO: Result of input of action #-1 of target [["@","tutorial","tests","greet"],{}]: {
+        "artifacts": {
+          "runner": {"data":{"file_type":"x","id":"0647621fba9b22f0727fbef98104f3e398496e2f","size":1876},"type":"KNOWN"},
+          "test": {"data":{"id":"465b690f0b006553c15fb059e2293011c20f74d4","path":"test_greet"},"type":"ACTION"},
+          "test-args.json": {"data":{"file_type":"f","id":"0637a088a01e8ddab3bf3fa98dbe804cbde1a0dc","size":2},"type":"KNOWN"},
+          "test-launcher.json": {"data":{"file_type":"f","id":"0637a088a01e8ddab3bf3fa98dbe804cbde1a0dc","size":2},"type":"KNOWN"}
+        },
+        "provides": {
+          "cmd": [
+            "./runner"
+          ],
+          "env": {
+          },
+          "may_fail": "CC test /test_greet failed",
+          "output": [
+            "result",
+            "stderr",
+            "stdout",
+            "time-start",
+            "time-stop"
+          ],
+          "output_dirs": [
+          ]
+        },
+        "runfiles": {
+        }
+      }
+INFO: Target tainted ["test"].
+$
+```
+
+The provided data also shows us the precise description of the action
+for which we request the input. This allows us to manually rerun the
+action. Or we can simply interact with the test binary manually after
+installing the inputs to this action. Requesting the inputs of an action
+can also be useful when debugging a build failure.
+
+``` sh
+$ just-mr install -o work --request-action-input -1 tests greet
+INFO: Requested target is [["@","tutorial","tests","greet"],{}]
+INFO: Request is input of action #-1
+INFO: Analysed target [["@","tutorial","tests","greet"],{}]
+INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
+INFO: Target tainted ["test"].
+INFO: Discovered 5 actions, 3 trees, 1 blobs
+INFO: Building input of action #-1 of [["@","tutorial","tests","greet"],{}].
+INFO: Processed 4 actions, 4 cache hits.
+INFO: Artifacts can be found in:
+        /tmp/tutorial/work/runner [0647621fba9b22f0727fbef98104f3e398496e2f:1876:x]
+        /tmp/tutorial/work/test [00458536b165abdee1802e5fb7b0922e04c81491:20352:x]
+        /tmp/tutorial/work/test-args.json [0637a088a01e8ddab3bf3fa98dbe804cbde1a0dc:2:f]
+        /tmp/tutorial/work/test-launcher.json [0637a088a01e8ddab3bf3fa98dbe804cbde1a0dc:2:f]
+INFO: Target tainted ["test"].
+$ cd work/
+$ ./test
+greet output: Hello World!
+
+greet output: Hello Universe!
+
+$ echo $?
+0
+$ cd ..
+$ rm -rf work
+```
+
+Creating a shell test
+---------------------
+
+Similarly, to create a shell test for testing the `helloworld` binary, a
+test script `tests/test_helloworld.sh` must be provided:
+
+``` {.sh srcname="tests/test_helloworld.sh"}
+set -e
+[ "$(./helloworld)" = "Hello Universe!" ]
+```
+
+The test target for this shell tests uses the rule
+`["@", "rules", "shell/test", "script"]` and must depend on the
+`"helloworld"` target. To create the test target, add the following to
+the `tests/TARGETS` file:
+
+``` {.jsonc srcname="tests/TARGETS"}
+...
+, "helloworld":
+  { "type": ["@", "rules", "shell/test", "script"]
+  , "name": ["test_helloworld"]
+  , "test": ["test_helloworld.sh"]
+  , "deps": [["", "helloworld"]]
+  }
+...
+```
+
+Now we can run the shell test (i.e., build the test result):
+
+``` sh
+$ just-mr build tests helloworld
+INFO: Requested target is [["@","tutorial","tests","helloworld"],{}]
+INFO: Analysed target [["@","tutorial","tests","helloworld"],{}]
+INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
+INFO: Target tainted ["test"].
+INFO: Discovered 5 actions, 4 trees, 0 blobs
+INFO: Building [["@","tutorial","tests","helloworld"],{}].
+INFO: Processed 5 actions, 4 cache hits.
+INFO: Artifacts built, logical paths are:
+        result [7ef22e9a431ad0272713b71fdc8794016c8ef12f:5:f]
+        stderr [e69de29bb2d1d6434b8b29ae775ad8c2e48c5391:0:f]
+        stdout [e69de29bb2d1d6434b8b29ae775ad8c2e48c5391:0:f]
+        time-start [c31e92e6b16dacec4ee95beefcc6a688dbffee2d:11:f]
+        time-stop [c31e92e6b16dacec4ee95beefcc6a688dbffee2d:11:f]
+      (1 runfiles omitted.)
+INFO: Target tainted ["test"].
+$
+```
+
+The result is also similar, containing also the 5 artifacts and a single
+runfile (omitted in the output above), which is a tree artifact with the
+name `test_helloworld` that contains all of the above artifacts.
+
+Creating a compound test target
+-------------------------------
+
+As most people probably do not want to call every test target by hand,
+it is desirable to compound test target that triggers the build of
+multiple test reports. To do so, an `"install"` target can be used. The
+field `"deps"` of an install target is a list of targets for which the
+runfiles are collected. As for the tests the runfiles happen to be tree
+artifacts named the same way as the test and containing all test
+results, this is precisely what we need. Furthermore, as the dependent
+test targets are tainted by `"test"`, also the compound test target must
+be tainted by the same string. To create the compound test target
+combining the two tests above (the tests `"greet"` and `"helloworld"`
+from module `"tests"`), add the following to the `tests/TARGETS` file:
+
+``` {.jsonc srcname="tests/TARGETS"}
+...
+, "ALL":
+  { "type": "install"
+  , "tainted": ["test"]
+  , "deps": ["greet", "helloworld"]
+  }
+...
+```
+
+Now we can run all tests at once by just building the compound test
+target `"ALL"`:
+
+``` sh
+$ just-mr build tests ALL
+INFO: Requested target is [["@","tutorial","tests","ALL"],{}]
+INFO: Analysed target [["@","tutorial","tests","ALL"],{}]
+INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
+INFO: Target tainted ["test"].
+INFO: Discovered 8 actions, 5 trees, 1 blobs
+INFO: Building [["@","tutorial","tests","ALL"],{}].
+INFO: Processed 8 actions, 8 cache hits.
+INFO: Artifacts built, logical paths are:
+        test_greet [251ba2038ccdb8ba1ae2f4e963751b9230b36646:177:t]
+        test_helloworld [63fa5954161b52b275b05c270e1626feaa8e178b:177:t]
+INFO: Target tainted ["test"].
+$
+```
+
+As a result it reports the runfiles (result directories) of both tests
+as artifacts. Both tests ran successfully as none of those artifacts in
+this output above are tagged as `FAILED`.