summaryrefslogtreecommitdiff
path: root/doc/tutorial/tests.org
diff options
context:
space:
mode:
authorOliver Reiche <oliver.reiche@huawei.com>2023-06-01 13:36:32 +0200
committerOliver Reiche <oliver.reiche@huawei.com>2023-06-12 16:29:05 +0200
commitb66a7359fbbff35af630c88c56598bbc06b393e1 (patch)
treed866802c4b44c13cbd90f9919cc7fc472091be0c /doc/tutorial/tests.org
parent144b2c619f28c91663936cd445251ca28af45f88 (diff)
downloadjustbuild-b66a7359fbbff35af630c88c56598bbc06b393e1.tar.gz
doc: Convert orgmode files to markdown
Diffstat (limited to 'doc/tutorial/tests.org')
-rw-r--r--doc/tutorial/tests.org337
1 files changed, 0 insertions, 337 deletions
diff --git a/doc/tutorial/tests.org b/doc/tutorial/tests.org
deleted file mode 100644
index d6842ab2..00000000
--- a/doc/tutorial/tests.org
+++ /dev/null
@@ -1,337 +0,0 @@
-* Creating Tests
-
-To run tests with justbuild, we do /not/ have a dedicated ~test~
-subcommand. Instead, we consider tests being a specific action that
-generates a test report. Consequently, we use the ~build~ subcommand
-to build the test report, and thereby run the test action. Test
-actions, however, are slightly different from normal actions in
-that we don't want the build of the test report to be aborted if
-a test action fails (but still, we want only successfully actions
-taken from cache). Rules defining targets containing such special
-actions have to identify themselves as /tainted/ by specifying
-a string explaining why such special actions are justified; in
-our case, the string is ~"test"~. Besides the implicit marking by
-using a tainted rule, those tainting strings can also be explicitly
-assigned by the user in the definition of a target, e.g., to mark
-test data. Any target has to be tainted with (at least) all the
-strings any of its dependencies is tainted with. In this way, it
-is ensured that no test target will end up in a production build.
-
-For the remainder of this section, we expect to have the project files available
-resulting from successfully completing the tutorial section on /Building C++
-Hello World/. We will demonstrate how to write a test binary for the ~greet~
-library and a shell test for the ~helloworld~ binary.
-
-** Creating a C++ test binary
-
-First, we will create a C++ test binary for testing the correct functionality of
-the ~greet~ library. Therefore, we need to provide a C++ source file that performs
-the actual testing and returns non-~0~ on failure. For simplicity reasons, we do
-not use a testing framework for this tutorial. A simple test that captures
-standard output and verifies it with the expected output should be provided in
-the file ~tests/greet.test.cpp~:
-
-#+SRCNAME: tests/greet.test.cpp
-#+BEGIN_SRC cpp
-#include <functional>
-#include <iostream>
-#include <string>
-#include <unistd.h>
-#include "greet/greet.hpp"
-
-template <std::size_t kMaxBufSize = 1024>
-auto capture_stdout(std::function<void()> const& func) -> std::string {
- int fd[2];
- if (pipe(fd) < 0) return {};
- int fd_stdout = dup(fileno(stdout));
- fflush(stdout);
- dup2(fd[1], fileno(stdout));
-
- func();
- fflush(stdout);
-
- std::string buf(kMaxBufSize, '\0');
- auto n = read(fd[0], &(*buf.begin()), kMaxBufSize);
- close(fd[0]);
- close(fd[1]);
- dup2(fd_stdout, fileno(stdout));
- return buf.substr(0, n);
-}
-
-auto test_greet(std::string const& name) -> bool {
- auto expect = std::string{"Hello "} + name + "!\n";
- auto result = capture_stdout([&name] { greet(name); });
- std::cout << "greet output: " << result << std::endl;
- return result == expect;
-}
-
-int main() {
- return test_greet("World") && test_greet("Universe") ? 0 : 1;
-}
-#+END_SRC
-
-Next, a new test target needs to be created in module ~greet~. This target uses
-the rule ~["@", "rules", "CC/test", "test"]~ and needs to depend on the
-~["greet", "greet"]~ target. To create the test target, add the following to
-~tests/TARGETS~:
-
-#+SRCNAME: tests/TARGETS
-#+BEGIN_SRC js
-{ "greet":
- { "type": ["@", "rules", "CC/test", "test"]
- , "name": ["test_greet"]
- , "srcs": ["greet.test.cpp"]
- , "private-deps": [["greet", "greet"]]
- }
-}
-#+END_SRC
-
-Before we can run the test, a proper default module for ~CC/test~ must be
-provided. By specifying the appropriate target in this module the default test
-runner can be overwritten by a different test runner fom the rule's workspace
-root. Moreover, all test targets share runner infrastructure from ~shell/test~,
-e.g., summarizing multiple runs per test (to detect flakyness) if the configuration
-variable ~RUNS_PER_TEST~ is set.
-
-However, in our case, we want to use the default runner and therefore it
-is sufficient to create an empty module. To do so, create the file
-~tutorial-defaults/CC/test/TARGETS~ with content
-
-#+SRCNAME: tutorial-defaults/CC/test/TARGETS
-#+BEGIN_SRC js
-{}
-#+END_SRC
-
-as well as the file ~tutorial-defaults/shell/test/TARGETS~ with content
-
-#+SRCNAME: tutorial-defaults/shell/test/TARGETS
-#+BEGIN_SRC js
-{}
-#+END_SRC
-
-
-Now we can run the test (i.e., build the test result):
-
-#+BEGIN_SRC sh
-$ just-mr build tests greet
-INFO: Requested target is [["@","tutorial","tests","greet"],{}]
-INFO: Analysed target [["@","tutorial","tests","greet"],{}]
-INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
-INFO: Target tainted ["test"].
-INFO: Discovered 5 actions, 3 trees, 1 blobs
-INFO: Building [["@","tutorial","tests","greet"],{}].
-INFO: Processed 5 actions, 2 cache hits.
-INFO: Artifacts built, logical paths are:
- result [7ef22e9a431ad0272713b71fdc8794016c8ef12f:5:f]
- stderr [8b137891791fe96927ad78e64b0aad7bded08bdc:1:f]
- stdout [ae6c6813755da67954a4a562f6d2ef01578c3e89:60:f]
- time-start [8e7a68af8d5d7a6d0036c1126ff5c16a5045ae95:11:f]
- time-stop [8e7a68af8d5d7a6d0036c1126ff5c16a5045ae95:11:f]
- (1 runfiles omitted.)
-INFO: Target tainted ["test"].
-$
-#+END_SRC
-
-Note that the target is correctly reported as tainted with ~"test"~. It will
-produce 3 additional actions for compiling, linking and running the test binary.
-
-The result of the test target are 5 artifacts: ~result~ (containing ~UNKNOWN~,
-~PASS~, or ~FAIL~), ~stderr~, ~stdout~, ~time-start~, and ~time-stop~, and a
-single runfile (omitted in the output above), which is a tree artifact with the
-name ~test_greet~ that contains all of the above artifacts. The test was run
-successfully as otherwise all reported artifacts would have been reported as
-~FAILED~ in the output, and justbuild would have returned the exit code ~2~.
-
-To immediately print the standard output produced by the test binary on the
-command line, the ~-P~ option can be used. Argument to this option is the name
-of the artifact that should be printed on the command line, in our case
-~stdout~:
-
-#+BEGIN_SRC sh
-$ just-mr build tests greet --log-limit 1 -P stdout
-greet output: Hello World!
-
-greet output: Hello Universe!
-
-$
-#+END_SRC
-
-Note that ~--log-limit 1~ was just added to omit justbuild's ~INFO:~ prints.
-
-Our test binary does not have any useful options for directly interacting
-with it. When working with test frameworks, it sometimes can be desirable to
-get hold of the test binary itself for manual interaction. The running of
-the test binary is the last action associated with the test and the test
-binary is, of course, one of its inputs.
-
-#+BEGIN_SRC sh
-$ just-mr analyse --request-action-input -1 tests greet
-INFO: Requested target is [["@","tutorial","tests","greet"],{}]
-INFO: Request is input of action #-1
-INFO: Result of input of action #-1 of target [["@","tutorial","tests","greet"],{}]: {
- "artifacts": {
- "runner": {"data":{"file_type":"x","id":"0647621fba9b22f0727fbef98104f3e398496e2f","size":1876},"type":"KNOWN"},
- "test": {"data":{"id":"465b690f0b006553c15fb059e2293011c20f74d4","path":"test_greet"},"type":"ACTION"},
- "test-args.json": {"data":{"file_type":"f","id":"0637a088a01e8ddab3bf3fa98dbe804cbde1a0dc","size":2},"type":"KNOWN"},
- "test-launcher.json": {"data":{"file_type":"f","id":"0637a088a01e8ddab3bf3fa98dbe804cbde1a0dc","size":2},"type":"KNOWN"}
- },
- "provides": {
- "cmd": [
- "./runner"
- ],
- "env": {
- },
- "may_fail": "CC test /test_greet failed",
- "output": [
- "result",
- "stderr",
- "stdout",
- "time-start",
- "time-stop"
- ],
- "output_dirs": [
- ]
- },
- "runfiles": {
- }
- }
-INFO: Target tainted ["test"].
-$
-#+END_SRC
-
-The provided data also shows us the precise description of the action
-for which we request the input. This allows us to manually rerun
-the action. Or we can simply interact with the test binary manually
-after installing the inputs to this action. Requesting the inputs
-of an action can also be useful when debugging a build failure.
-
-#+BEGIN_SRC sh
-$ just-mr install -o work --request-action-input -1 tests greet
-INFO: Requested target is [["@","tutorial","tests","greet"],{}]
-INFO: Request is input of action #-1
-INFO: Analysed target [["@","tutorial","tests","greet"],{}]
-INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
-INFO: Target tainted ["test"].
-INFO: Discovered 5 actions, 3 trees, 1 blobs
-INFO: Building input of action #-1 of [["@","tutorial","tests","greet"],{}].
-INFO: Processed 4 actions, 4 cache hits.
-INFO: Artifacts can be found in:
- /tmp/tutorial/work/runner [0647621fba9b22f0727fbef98104f3e398496e2f:1876:x]
- /tmp/tutorial/work/test [00458536b165abdee1802e5fb7b0922e04c81491:20352:x]
- /tmp/tutorial/work/test-args.json [0637a088a01e8ddab3bf3fa98dbe804cbde1a0dc:2:f]
- /tmp/tutorial/work/test-launcher.json [0637a088a01e8ddab3bf3fa98dbe804cbde1a0dc:2:f]
-INFO: Target tainted ["test"].
-$ cd work/
-$ ./test
-greet output: Hello World!
-
-greet output: Hello Universe!
-
-$ echo $?
-0
-$ cd ..
-$ rm -rf work
-#+END_SRC
-
-** Creating a shell test
-
-Similarly, to create a shell test for testing the ~helloworld~ binary, a test
-script ~tests/test_helloworld.sh~ must be provided:
-
-#+SRCNAME: tests/test_helloworld.sh
-#+BEGIN_SRC sh
-set -e
-[ "$(./helloworld)" = "Hello Universe!" ]
-#+END_SRC
-
-The test target for this shell tests uses the rule
-~["@", "rules", "shell/test", "script"]~ and must depend on the ~"helloworld"~
-target. To create the test target, add the following to the ~tests/TARGETS~
-file:
-
-#+SRCNAME: tests/TARGETS
-#+BEGIN_SRC js
-...
-, "helloworld":
- { "type": ["@", "rules", "shell/test", "script"]
- , "name": ["test_helloworld"]
- , "test": ["test_helloworld.sh"]
- , "deps": [["", "helloworld"]]
- }
-...
-#+END_SRC
-
-Now we can run the shell test (i.e., build the test result):
-
-#+BEGIN_SRC sh
-$ just-mr build tests helloworld
-INFO: Requested target is [["@","tutorial","tests","helloworld"],{}]
-INFO: Analysed target [["@","tutorial","tests","helloworld"],{}]
-INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
-INFO: Target tainted ["test"].
-INFO: Discovered 5 actions, 4 trees, 0 blobs
-INFO: Building [["@","tutorial","tests","helloworld"],{}].
-INFO: Processed 5 actions, 4 cache hits.
-INFO: Artifacts built, logical paths are:
- result [7ef22e9a431ad0272713b71fdc8794016c8ef12f:5:f]
- stderr [e69de29bb2d1d6434b8b29ae775ad8c2e48c5391:0:f]
- stdout [e69de29bb2d1d6434b8b29ae775ad8c2e48c5391:0:f]
- time-start [c31e92e6b16dacec4ee95beefcc6a688dbffee2d:11:f]
- time-stop [c31e92e6b16dacec4ee95beefcc6a688dbffee2d:11:f]
- (1 runfiles omitted.)
-INFO: Target tainted ["test"].
-$
-#+END_SRC
-
-The result is also similar, containing also the 5 artifacts and a single runfile
-(omitted in the output above), which is a tree artifact with the name
-~test_helloworld~ that contains all of the above artifacts.
-
-** Creating a compound test target
-
-As most people probably do not want to call every test target by hand, it is
-desirable to compound test target that triggers the build of multiple test
-reports. To do so, an ~"install"~ target can be used. The field ~"deps"~ of
-an install target is a list of targets for which the runfiles are collected.
-As for the tests the runfiles happen to be
-tree artifacts named the same way as the test and containing all test results,
-this is precisely what we need.
-Furthermore, as the dependent test targets are tainted by ~"test"~, also the
-compound test target must be tainted by the same string. To create the compound
-test target combining the two tests above (the tests ~"greet"~ and
-~"helloworld"~ from module ~"tests"~), add the following to the ~tests/TARGETS~
-file:
-
-#+SRCNAME: tests/TARGETS
-#+BEGIN_SRC js
-...
-, "ALL":
- { "type": "install"
- , "tainted": ["test"]
- , "deps": ["greet", "helloworld"]
- }
-...
-#+END_SRC
-
-Now we can run all tests at once by just building the compound test target
-~"ALL"~:
-
-#+BEGIN_SRC sh
-$ just-mr build tests ALL
-INFO: Requested target is [["@","tutorial","tests","ALL"],{}]
-INFO: Analysed target [["@","tutorial","tests","ALL"],{}]
-INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
-INFO: Target tainted ["test"].
-INFO: Discovered 8 actions, 5 trees, 1 blobs
-INFO: Building [["@","tutorial","tests","ALL"],{}].
-INFO: Processed 8 actions, 8 cache hits.
-INFO: Artifacts built, logical paths are:
- test_greet [251ba2038ccdb8ba1ae2f4e963751b9230b36646:177:t]
- test_helloworld [63fa5954161b52b275b05c270e1626feaa8e178b:177:t]
-INFO: Target tainted ["test"].
-$
-#+END_SRC
-
-As a result it reports the runfiles (result directories) of both tests as
-artifacts. Both tests ran successfully as none of those artifacts in this output
-above are tagged as ~FAILED~.