diff options
author | Oliver Reiche <oliver.reiche@huawei.com> | 2023-06-01 13:36:32 +0200 |
---|---|---|
committer | Oliver Reiche <oliver.reiche@huawei.com> | 2023-06-12 16:29:05 +0200 |
commit | b66a7359fbbff35af630c88c56598bbc06b393e1 (patch) | |
tree | d866802c4b44c13cbd90f9919cc7fc472091be0c /doc/tutorial/tests.org | |
parent | 144b2c619f28c91663936cd445251ca28af45f88 (diff) | |
download | justbuild-b66a7359fbbff35af630c88c56598bbc06b393e1.tar.gz |
doc: Convert orgmode files to markdown
Diffstat (limited to 'doc/tutorial/tests.org')
-rw-r--r-- | doc/tutorial/tests.org | 337 |
1 files changed, 0 insertions, 337 deletions
diff --git a/doc/tutorial/tests.org b/doc/tutorial/tests.org deleted file mode 100644 index d6842ab2..00000000 --- a/doc/tutorial/tests.org +++ /dev/null @@ -1,337 +0,0 @@ -* Creating Tests - -To run tests with justbuild, we do /not/ have a dedicated ~test~ -subcommand. Instead, we consider tests being a specific action that -generates a test report. Consequently, we use the ~build~ subcommand -to build the test report, and thereby run the test action. Test -actions, however, are slightly different from normal actions in -that we don't want the build of the test report to be aborted if -a test action fails (but still, we want only successfully actions -taken from cache). Rules defining targets containing such special -actions have to identify themselves as /tainted/ by specifying -a string explaining why such special actions are justified; in -our case, the string is ~"test"~. Besides the implicit marking by -using a tainted rule, those tainting strings can also be explicitly -assigned by the user in the definition of a target, e.g., to mark -test data. Any target has to be tainted with (at least) all the -strings any of its dependencies is tainted with. In this way, it -is ensured that no test target will end up in a production build. - -For the remainder of this section, we expect to have the project files available -resulting from successfully completing the tutorial section on /Building C++ -Hello World/. We will demonstrate how to write a test binary for the ~greet~ -library and a shell test for the ~helloworld~ binary. - -** Creating a C++ test binary - -First, we will create a C++ test binary for testing the correct functionality of -the ~greet~ library. Therefore, we need to provide a C++ source file that performs -the actual testing and returns non-~0~ on failure. For simplicity reasons, we do -not use a testing framework for this tutorial. A simple test that captures -standard output and verifies it with the expected output should be provided in -the file ~tests/greet.test.cpp~: - -#+SRCNAME: tests/greet.test.cpp -#+BEGIN_SRC cpp -#include <functional> -#include <iostream> -#include <string> -#include <unistd.h> -#include "greet/greet.hpp" - -template <std::size_t kMaxBufSize = 1024> -auto capture_stdout(std::function<void()> const& func) -> std::string { - int fd[2]; - if (pipe(fd) < 0) return {}; - int fd_stdout = dup(fileno(stdout)); - fflush(stdout); - dup2(fd[1], fileno(stdout)); - - func(); - fflush(stdout); - - std::string buf(kMaxBufSize, '\0'); - auto n = read(fd[0], &(*buf.begin()), kMaxBufSize); - close(fd[0]); - close(fd[1]); - dup2(fd_stdout, fileno(stdout)); - return buf.substr(0, n); -} - -auto test_greet(std::string const& name) -> bool { - auto expect = std::string{"Hello "} + name + "!\n"; - auto result = capture_stdout([&name] { greet(name); }); - std::cout << "greet output: " << result << std::endl; - return result == expect; -} - -int main() { - return test_greet("World") && test_greet("Universe") ? 0 : 1; -} -#+END_SRC - -Next, a new test target needs to be created in module ~greet~. This target uses -the rule ~["@", "rules", "CC/test", "test"]~ and needs to depend on the -~["greet", "greet"]~ target. To create the test target, add the following to -~tests/TARGETS~: - -#+SRCNAME: tests/TARGETS -#+BEGIN_SRC js -{ "greet": - { "type": ["@", "rules", "CC/test", "test"] - , "name": ["test_greet"] - , "srcs": ["greet.test.cpp"] - , "private-deps": [["greet", "greet"]] - } -} -#+END_SRC - -Before we can run the test, a proper default module for ~CC/test~ must be -provided. By specifying the appropriate target in this module the default test -runner can be overwritten by a different test runner fom the rule's workspace -root. Moreover, all test targets share runner infrastructure from ~shell/test~, -e.g., summarizing multiple runs per test (to detect flakyness) if the configuration -variable ~RUNS_PER_TEST~ is set. - -However, in our case, we want to use the default runner and therefore it -is sufficient to create an empty module. To do so, create the file -~tutorial-defaults/CC/test/TARGETS~ with content - -#+SRCNAME: tutorial-defaults/CC/test/TARGETS -#+BEGIN_SRC js -{} -#+END_SRC - -as well as the file ~tutorial-defaults/shell/test/TARGETS~ with content - -#+SRCNAME: tutorial-defaults/shell/test/TARGETS -#+BEGIN_SRC js -{} -#+END_SRC - - -Now we can run the test (i.e., build the test result): - -#+BEGIN_SRC sh -$ just-mr build tests greet -INFO: Requested target is [["@","tutorial","tests","greet"],{}] -INFO: Analysed target [["@","tutorial","tests","greet"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Target tainted ["test"]. -INFO: Discovered 5 actions, 3 trees, 1 blobs -INFO: Building [["@","tutorial","tests","greet"],{}]. -INFO: Processed 5 actions, 2 cache hits. -INFO: Artifacts built, logical paths are: - result [7ef22e9a431ad0272713b71fdc8794016c8ef12f:5:f] - stderr [8b137891791fe96927ad78e64b0aad7bded08bdc:1:f] - stdout [ae6c6813755da67954a4a562f6d2ef01578c3e89:60:f] - time-start [8e7a68af8d5d7a6d0036c1126ff5c16a5045ae95:11:f] - time-stop [8e7a68af8d5d7a6d0036c1126ff5c16a5045ae95:11:f] - (1 runfiles omitted.) -INFO: Target tainted ["test"]. -$ -#+END_SRC - -Note that the target is correctly reported as tainted with ~"test"~. It will -produce 3 additional actions for compiling, linking and running the test binary. - -The result of the test target are 5 artifacts: ~result~ (containing ~UNKNOWN~, -~PASS~, or ~FAIL~), ~stderr~, ~stdout~, ~time-start~, and ~time-stop~, and a -single runfile (omitted in the output above), which is a tree artifact with the -name ~test_greet~ that contains all of the above artifacts. The test was run -successfully as otherwise all reported artifacts would have been reported as -~FAILED~ in the output, and justbuild would have returned the exit code ~2~. - -To immediately print the standard output produced by the test binary on the -command line, the ~-P~ option can be used. Argument to this option is the name -of the artifact that should be printed on the command line, in our case -~stdout~: - -#+BEGIN_SRC sh -$ just-mr build tests greet --log-limit 1 -P stdout -greet output: Hello World! - -greet output: Hello Universe! - -$ -#+END_SRC - -Note that ~--log-limit 1~ was just added to omit justbuild's ~INFO:~ prints. - -Our test binary does not have any useful options for directly interacting -with it. When working with test frameworks, it sometimes can be desirable to -get hold of the test binary itself for manual interaction. The running of -the test binary is the last action associated with the test and the test -binary is, of course, one of its inputs. - -#+BEGIN_SRC sh -$ just-mr analyse --request-action-input -1 tests greet -INFO: Requested target is [["@","tutorial","tests","greet"],{}] -INFO: Request is input of action #-1 -INFO: Result of input of action #-1 of target [["@","tutorial","tests","greet"],{}]: { - "artifacts": { - "runner": {"data":{"file_type":"x","id":"0647621fba9b22f0727fbef98104f3e398496e2f","size":1876},"type":"KNOWN"}, - "test": {"data":{"id":"465b690f0b006553c15fb059e2293011c20f74d4","path":"test_greet"},"type":"ACTION"}, - "test-args.json": {"data":{"file_type":"f","id":"0637a088a01e8ddab3bf3fa98dbe804cbde1a0dc","size":2},"type":"KNOWN"}, - "test-launcher.json": {"data":{"file_type":"f","id":"0637a088a01e8ddab3bf3fa98dbe804cbde1a0dc","size":2},"type":"KNOWN"} - }, - "provides": { - "cmd": [ - "./runner" - ], - "env": { - }, - "may_fail": "CC test /test_greet failed", - "output": [ - "result", - "stderr", - "stdout", - "time-start", - "time-stop" - ], - "output_dirs": [ - ] - }, - "runfiles": { - } - } -INFO: Target tainted ["test"]. -$ -#+END_SRC - -The provided data also shows us the precise description of the action -for which we request the input. This allows us to manually rerun -the action. Or we can simply interact with the test binary manually -after installing the inputs to this action. Requesting the inputs -of an action can also be useful when debugging a build failure. - -#+BEGIN_SRC sh -$ just-mr install -o work --request-action-input -1 tests greet -INFO: Requested target is [["@","tutorial","tests","greet"],{}] -INFO: Request is input of action #-1 -INFO: Analysed target [["@","tutorial","tests","greet"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Target tainted ["test"]. -INFO: Discovered 5 actions, 3 trees, 1 blobs -INFO: Building input of action #-1 of [["@","tutorial","tests","greet"],{}]. -INFO: Processed 4 actions, 4 cache hits. -INFO: Artifacts can be found in: - /tmp/tutorial/work/runner [0647621fba9b22f0727fbef98104f3e398496e2f:1876:x] - /tmp/tutorial/work/test [00458536b165abdee1802e5fb7b0922e04c81491:20352:x] - /tmp/tutorial/work/test-args.json [0637a088a01e8ddab3bf3fa98dbe804cbde1a0dc:2:f] - /tmp/tutorial/work/test-launcher.json [0637a088a01e8ddab3bf3fa98dbe804cbde1a0dc:2:f] -INFO: Target tainted ["test"]. -$ cd work/ -$ ./test -greet output: Hello World! - -greet output: Hello Universe! - -$ echo $? -0 -$ cd .. -$ rm -rf work -#+END_SRC - -** Creating a shell test - -Similarly, to create a shell test for testing the ~helloworld~ binary, a test -script ~tests/test_helloworld.sh~ must be provided: - -#+SRCNAME: tests/test_helloworld.sh -#+BEGIN_SRC sh -set -e -[ "$(./helloworld)" = "Hello Universe!" ] -#+END_SRC - -The test target for this shell tests uses the rule -~["@", "rules", "shell/test", "script"]~ and must depend on the ~"helloworld"~ -target. To create the test target, add the following to the ~tests/TARGETS~ -file: - -#+SRCNAME: tests/TARGETS -#+BEGIN_SRC js -... -, "helloworld": - { "type": ["@", "rules", "shell/test", "script"] - , "name": ["test_helloworld"] - , "test": ["test_helloworld.sh"] - , "deps": [["", "helloworld"]] - } -... -#+END_SRC - -Now we can run the shell test (i.e., build the test result): - -#+BEGIN_SRC sh -$ just-mr build tests helloworld -INFO: Requested target is [["@","tutorial","tests","helloworld"],{}] -INFO: Analysed target [["@","tutorial","tests","helloworld"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Target tainted ["test"]. -INFO: Discovered 5 actions, 4 trees, 0 blobs -INFO: Building [["@","tutorial","tests","helloworld"],{}]. -INFO: Processed 5 actions, 4 cache hits. -INFO: Artifacts built, logical paths are: - result [7ef22e9a431ad0272713b71fdc8794016c8ef12f:5:f] - stderr [e69de29bb2d1d6434b8b29ae775ad8c2e48c5391:0:f] - stdout [e69de29bb2d1d6434b8b29ae775ad8c2e48c5391:0:f] - time-start [c31e92e6b16dacec4ee95beefcc6a688dbffee2d:11:f] - time-stop [c31e92e6b16dacec4ee95beefcc6a688dbffee2d:11:f] - (1 runfiles omitted.) -INFO: Target tainted ["test"]. -$ -#+END_SRC - -The result is also similar, containing also the 5 artifacts and a single runfile -(omitted in the output above), which is a tree artifact with the name -~test_helloworld~ that contains all of the above artifacts. - -** Creating a compound test target - -As most people probably do not want to call every test target by hand, it is -desirable to compound test target that triggers the build of multiple test -reports. To do so, an ~"install"~ target can be used. The field ~"deps"~ of -an install target is a list of targets for which the runfiles are collected. -As for the tests the runfiles happen to be -tree artifacts named the same way as the test and containing all test results, -this is precisely what we need. -Furthermore, as the dependent test targets are tainted by ~"test"~, also the -compound test target must be tainted by the same string. To create the compound -test target combining the two tests above (the tests ~"greet"~ and -~"helloworld"~ from module ~"tests"~), add the following to the ~tests/TARGETS~ -file: - -#+SRCNAME: tests/TARGETS -#+BEGIN_SRC js -... -, "ALL": - { "type": "install" - , "tainted": ["test"] - , "deps": ["greet", "helloworld"] - } -... -#+END_SRC - -Now we can run all tests at once by just building the compound test target -~"ALL"~: - -#+BEGIN_SRC sh -$ just-mr build tests ALL -INFO: Requested target is [["@","tutorial","tests","ALL"],{}] -INFO: Analysed target [["@","tutorial","tests","ALL"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching -INFO: Target tainted ["test"]. -INFO: Discovered 8 actions, 5 trees, 1 blobs -INFO: Building [["@","tutorial","tests","ALL"],{}]. -INFO: Processed 8 actions, 8 cache hits. -INFO: Artifacts built, logical paths are: - test_greet [251ba2038ccdb8ba1ae2f4e963751b9230b36646:177:t] - test_helloworld [63fa5954161b52b275b05c270e1626feaa8e178b:177:t] -INFO: Target tainted ["test"]. -$ -#+END_SRC - -As a result it reports the runfiles (result directories) of both tests as -artifacts. Both tests ran successfully as none of those artifacts in this output -above are tagged as ~FAILED~. |