diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/being-consumed-by-cmake.md | 167 | ||||
| -rw-r--r-- | doc/consume-cmake-libraries.md | 138 | ||||
| -rw-r--r-- | doc/debug-fission.md | 139 |
3 files changed, 444 insertions, 0 deletions
diff --git a/doc/being-consumed-by-cmake.md b/doc/being-consumed-by-cmake.md new file mode 100644 index 0000000..b4e0bc5 --- /dev/null +++ b/doc/being-consumed-by-cmake.md @@ -0,0 +1,167 @@ +Being consumed by CMake +======================= + +To support libraries built with JustBuild to be consumed by CMake, the +rule `["CC", "install-with-deps"]` can be used. This rule will install +the library's artifacts, its public headers, and a pkg-config file, +which can be used by CMake to properly consume the library. + +In this example, we show how to build and install the `ssl` library with +JustBuild, so it can be picked up and used by CMake afterwards. + +Example: Build and install the `ssl` Library with JustBuild +----------------------------------------------------------- + +First make sure that `just`, `just-mr`, and `just-import-git` are +available in your `PATH`. Then, define a new workspace by creating the +`ROOT` marker. + +``` sh +$ touch ROOT +``` + +The `ssl` library that we want to use is a defined dependency of the +JustBuild repository `just/ssl`. To define a repository for that +library, create the following `repos.template.json` file. + +``` {.jsonc srcname="repos.template.json"} +{ "main": "ssl" +, "repositories": + { "ssl": + { "repository": {"type": "file", "path": "."} + , "bindings": {"rules": "rules-cc", "just-ssl": "just/ssl"} + } + } +} +``` + +and import the `rules-cc` and JustBuild repository, including its +dependency `just/ssl`, via: + +``` sh +$ cat repos.template.json \ + | just-import-git -C - --as just -b master https://github.com/just-buildsystem/justbuild \ + | just-import-git -C - --as rules-cc -b master https://github.com/just-buildsystem/rules-cc \ + > repos.json +$ +``` + +Next, the following `TARGETS` file needs to be provided. + +``` {.jsonc srcname="TARGETS"} +{ "ssl-lib": + { "type": ["@", "rules", "CC", "library"] + , "name": ["ssl"] + , "deps": [["@", "just-ssl", "", "ssl"]] + } +, "ssl-pkg": + {"type": ["@", "rules", "CC", "install-with-deps"], "targets": ["ssl-lib"]} +} +``` + +The library target `ssl-lib` specifies the name `"ssl"` (also used as +package name) and a dependency on the `ssl` library from `just-ssl`. +Note that although this library target is merely a "wrapper" +(header-only library without headers), it could equally well be a +full-blown library target. Furthermore, the package target `ssl-pkg` +installs the `ssl-lib` target including its dependencies and generates a +pkg-config file that can be later used by CMake. + +Finally, the `ssl-pkg` target can be built and installed to a specific +`PREFIX` (note that `OS` and `ARCH` must be set as well). + +``` sh +$ export PREFIX=/tmp/just_ssl +$ just-mr install -D'{"OS":"linux", "ARCH":"x64_64", "PREFIX":"'$PREFIX'"}' -o $PREFIX ssl-pkg +INFO: Requested target is [["@","ssl","","ssl-pkg"],{"ARCH":"x64_64","OS":"linux","PREFIX":"/tmp/just_ssl"}] +INFO: Analysed target [["@","ssl","","ssl-pkg"],{"ARCH":"x64_64","OS":"linux","PREFIX":"/tmp/just_ssl"}] +INFO: Export targets found: 0 cached, 2 uncached, 0 not eligible for caching +INFO: Discovered 269 actions, 3 trees, 1 blobs +INFO: Building [["@","ssl","","ssl-pkg"],{"ARCH":"x64_64","OS":"linux","PREFIX":"/tmp/just_ssl"}]. +PROG: [ 79%] 0 cached, 215 run, 8 processing (["@","just/ssl","","crypto-lib"]#105, ...). +INFO: Processed 269 actions, 0 cache hits. +INFO: Artifacts can be found in: + /tmp/just_ssl/include/openssl/aead.h [6d78db27d24746b39eacc20374697476e080c9f6:21054:f] + ... + /tmp/just_ssl/lib/libcrypto.a [df64da43f0168a717d09d609bf0c631d29c86b61:2336092:f] + /tmp/just_ssl/lib/libssl.a [77d2c2bfbe3ef3608895c854f1d1f6e1c200efd0:852620:f] + /tmp/just_ssl/lib/pkgconfig/ssl.pc [9b69c758430f5b5fb6ff7a9b1f1ffc89471509af:406:f] +$ +``` + +Example: Consume the installed `ssl` Library with CMake +------------------------------------------------------- + +As an example, a minimal `main.cpp` file is created that depends on the +`ssl` library. + +``` {.cpp srcname="main.cpp"} +#include <openssl/evp.h> + +int main(int argc, char** argv) { + OpenSSL_add_all_algorithms(); + return 0; +} +``` + +In the `CMakeLists.txt` file, the `pkg_check_modules()` macro can be +used to determine the compile and linker flags needed to consume our +`ssl` library with CMake. + +``` {.cmake srcname="CMakeLists.txt"} +cmake_minimum_required(VERSION 3.1) + +project(test_ssl) + +find_package(PkgConfig REQUIRED) +pkg_check_modules(SSL REQUIRED ssl) + +add_executable(main main.cpp) +target_include_directories(main PRIVATE ${SSL_INCLUDE_DIRS}) +target_compile_options(main PRIVATE ${SSL_CFLAGS_OTHER}) +target_link_directories(main PRIVATE ${SSL_LIBRARY_DIRS}) +target_link_libraries(main PRIVATE ${SSL_LDFLAGS_OTHER} ${SSL_LIBRARIES}) +``` + +Note that `${SSL_LDFLAGS_OTHER}` can be omitted if the `ssl-pkg` target +was defined with a non-empty value for the field `"flat-libs"` (see rule +documentation of `["CC", "install-with-deps"]`). + +Finally, when running CMake, make sure to set the `CMAKE_PREFIX_PATH` to +the previously used install prefix in order to successfully find the +installed `ssl` library and its pkg-config file. + +``` sh +$ cmake -DCMAKE_PREFIX_PATH=/tmp/just_ssl -S . -B /tmp/test_ssl +-- The C compiler identification is GNU 10.2.1 +-- The CXX compiler identification is GNU 10.2.1 +-- Detecting C compiler ABI info +-- Detecting C compiler ABI info - done +-- Check for working C compiler: /usr/bin/cc - skipped +-- Detecting C compile features +-- Detecting C compile features - done +-- Detecting CXX compiler ABI info +-- Detecting CXX compiler ABI info - done +-- Check for working CXX compiler: /usr/bin/c++ - skipped +-- Detecting CXX compile features +-- Detecting CXX compile features - done +-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.2") +-- Checking for module 'ssl' +-- Found ssl, version unknown +-- Configuring done +-- Generating done +-- Build files have been written to: /tmp/test_ssl +$ cmake --build /tmp/test_ssl +[ 50%] Building CXX object CMakeFiles/main.dir/main.cpp.o +[100%] Linking CXX executable main +[100%] Built target main +$ +``` + +Note that if the package is moved to a different location, the `prefix` +variable within the pkg-config file `lib/pkgconfig/ssl.pc` must be +updated as well. Alternatively, CMake can be instructed to automatically +guess the correct prefix by setting the variable `PKG_CONFIG_ARGN` to +`"--define-prefix"` (either on the command line or in the +`CMakeLists.txt`). However, this is a fairly recent CMake feature and +requires at least CMake version 3.22. diff --git a/doc/consume-cmake-libraries.md b/doc/consume-cmake-libraries.md new file mode 100644 index 0000000..5ba9b41 --- /dev/null +++ b/doc/consume-cmake-libraries.md @@ -0,0 +1,138 @@ +Consume CMake Libraries +======================= + +To support libraries built with CMake to be consumed by JustBuild, the +rule `["CC/foreign/cmake", "library"]` can be used. This rule will run +CMake and collect the library's artifacts, public headers, and +pkg-config files (if available), to produce a proper JustBuild library +target. + +In this example, we show how to consume the `gtest` library built with +CMake using JustBuild. + +Example: Build the `gtest` Library with CMake +--------------------------------------------- + +First make sure that `just`, `just-mr`, and `just-import-git` are +available in your `PATH`. Then, define a new workspace by creating a +`ROOT` marker. + +``` sh +$ touch ROOT +``` + +To define a repository for the +[gtest](https://github.com/google/googletest/tree/v1.13.0) library, +create the following `repos.template.json` file. + +``` {.jsonc srcname="repos.template.json"} +{ "main": "tests" +, "repositories": + { "imports": {"repository": {"type": "file", "path": "imports"}} + , "gtest": + { "repository": + { "type": "git" + , "branch": "v1.13.0" + , "commit": "b796f7d44681514f58a683a3a71ff17c94edb0c1" + , "repository": "https://github.com/google/googletest" + } + , "target_root": "imports" + , "target_file_name": "gtest.TARGETS" + , "bindings": {"rules": "rules-cc"} + } + , "tests": + { "repository": {"type": "file", "path": "."} + , "bindings": {"rules": "rules-cc", "gtest": "gtest"} + } + } +} +``` + +Now the missing `rules-cc` repository can be imported via: + +``` sh +$ just-import-git -C repos.template.json --as rules-cc -b master https://github.com/just-buildsystem/rules-cc > repos.json +$ +``` + +Create the file `imports/gtest.TARGETS` with the following content. + +``` {.jsonc srcname="imports/gtest.TARGETS"} +{ "gtest_main": + { "type": ["@", "rules", "CC/foreign/cmake", "library"] + , "name": ["gtest_main"] + , "version": ["1", "13", "0"] + , "project": [["TREE", null, "."]] + , "defines": ["BUILD_SHARED_LIBS=ON"] + , "out_hdr_dirs": ["gtest"] + , "out_libs": ["libgtest_main.so.1.13.0", "libgtest.so.1.13.0"] + , "pkg-config": ["gtest_main.pc", "gtest.pc"] + } +} +``` + +The library `gtest_main` uses the rule +`["CC/foreign/cmake", "library"]`. It sets `defines` to build shared +libraries and collects the public header directory `gtest`, as well as +the two libraries `libgtest_main.so.1.13.0` and `libgtest.so.1.13.0` (in +this particular link order). Furthermore, to automatically infer public +compile and link flags (e.g., a link dependency to the system's +`pthread` library), the pkg-config files `gtest_main.pc` and `gtest.pc` +are read, with the former (entry point) depending on the latter. + +Now, create the actual `test` target, which consumes the `gtest_main` +library, by creating the following `TARGETS` file. + +``` {.jsonc srcname="TARGETS"} +{ "test": + { "type": ["@", "rules", "CC/test", "test"] + , "name": ["test"] + , "srcs": ["test.cpp"] + , "private-deps": [["@", "gtest", "", "gtest_main"]] + } +} +``` + +The last file missing yet is the actual test source file `test.cpp`. + +``` {.cpp srcname="test.cpp"} +#include <gtest/gtest.h> + +TEST(CastTest, double) { + EXPECT_EQ (42.0, double(42)); +} +``` + +Finally, build the `test` target to run the test. + +``` sh +$ just-mr build test -Pstdout +INFO: Requested target is [["@","tests","","test"],{}] +INFO: Analysed target [["@","tests","","test"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Target tainted ["test"]. +INFO: Discovered 7 actions, 2 trees, 3 blobs +INFO: Building [["@","tests","","test"],{}]. +INFO: Processed 7 actions, 0 cache hits. +INFO: Artifacts built, logical paths are: + result [7ef22e9a431ad0272713b71fdc8794016c8ef12f:5:f] + stderr [8b137891791fe96927ad78e64b0aad7bded08bdc:1:f] + stdout [fc46e9e95a8a393e3c94875d66b0de956305f6a6:728:f] + time-start [93887fee067665917f67e2cb757dd30bf22b23a0:11:f] + time-stop [93887fee067665917f67e2cb757dd30bf22b23a0:11:f] + (1 runfiles omitted.) +Running main() from src/gtest_main.cc +[==========] Running 1 test from 1 test suite. +[----------] Global test environment set-up. +[----------] 1 test from CastTest +[ RUN ] CastTest.double +[ OK ] CastTest.double (0 ms) +[----------] 1 test from CastTest (0 ms total) + +[----------] Global test environment tear-down +[==========] 1 test from 1 test suite ran. (0 ms total) +[ PASSED ] 1 test. + +INFO: Target tainted ["test"]. +$ +``` diff --git a/doc/debug-fission.md b/doc/debug-fission.md new file mode 100644 index 0000000..573bbb4 --- /dev/null +++ b/doc/debug-fission.md @@ -0,0 +1,139 @@ +Debug fission support in C/C++ rules +==================================== + +Motivation +---------- + +Building with debug symbols is needed for debugging purposes, however it +results in artifacts that are many times larger than their release versions. +In general, debug builds are also slower than release builds due to various +reasons, the main ones being the disabling of certain code optimizations (in +order to allow debuggers to properly work) and debug-only checks and +diagnostics. Furthermore, sections of debug symbols from common dependencies can +be replicated many times between different artifacts, but also inside single +artifacts. + +In the cases where one does not produce separate release and debug versions, +it is usual to just generate the debug artifacts, then strip the debug symbols +from them to obtain a pseudo-release version. Even in this case, being able to +reduce the time and space requirements for producing the debug artifacts in the +first place is of value. + +Moreover, distributions usually provide debug information in different packages +and for that purpose apply themselves debug fission or stripping techniques. +Projects that provide separated debug information have thus an advantage in +getting accepted by reducing the work involved in packaging them for distros. + +This method can be applied when one already splits builds into their +constituting compile and link steps, which is the case with most modern build +tools, including *justbuild*. Debug fission proposes that the compilation step +of a debug build produce, instead of one object file, two artifacts: a `.dwo` +DWARF file containing all the debug symbols of the compilation unit, and the +(now smaller) `.o` object file containing now only references to the debug +symbols from the `.dwo` file. These object files can be linked as usual to +produce the final build artifact, and the `.dwo` files can be either retained +as-is, or packed into a corresponding `.dwp` file. + +By splitting the debug symbols of each compilation unit into separate +artifacts, these can be cached and reused as needed, removing any previous +debug symbols duplication across build artifacts. This in general also has a +beneficial impact on the build times. Moreover, the stripped artifacts remain +quasi-identical (e.g., executables differ only in their internal Build ID). + +Usage +----- + +Our approach targets specifically Linux ELF files and `gdb(1)`. This is, +however, more than enough to cover the most used UNIX-like platforms and most +general purpose debugging tools, which rely on `gdb(1)`. + +To enable a debug fission build, simply provide as `"DEBUG"` configuration +variable a map providing the necessary mandatory and optional fields, as +described in the documentation of the variable in the rules. + +For example, a configuration of +``` json +{ "DEBUG": + { "USE_DEBUG_FISSION": true + , "FISSION_CONFIG": + { "USE_SPLIT_DWARF": true + , "DWARF_VERSION": 4 + } + } +} +``` +instructs that the debug fission rules should be used and that the compile flags +`-gsplit-dwarf` and `-gdwarf-4` should be added additionally to the `-g` flag +(which is set for any debug build with no other flags otherwise set). + +**IMPORTANT:** Currently only the `-gslit-dwarf` compile flag can trigger the +generation of `.dwo` files and therefore the entry `"USE_SPLIT_DWARF": true` +should always be provided if debug fission rules are used (i.e., +`"USE_DEBUG_FISSION": true`). + +Additional notes +---------------- + +Below we provide more context, possible caveats, and known limitations to the +currently supported compile and/or link flags, which can be enabled via various +subfields of the `"FISSION_CONFIG"` debug configuration entry. + +### DWARF format versions + +Each toolchain supporting generation of debug symbols using the DWARF format +comes with a default in terms of which version of this format is used (in our +rules providable explicitly via the `"DWARF_VERSION"` subfield). Basically all +reasonably modern toolchains (GCC >=4.8.1, Clang >=7.0.0 at least) and debugging +tools (GDB >= 7.0) use DWARFv4 by default, with the more recent versions having +already switched to using the newer, upward compatible +[DWARFv5](https://dwarfstd.org/dwarf5std.html) format. However, the degree of +implementation and default support of the various compilers and tools differs, +so we recommend to always use version 4. + +### Support for `--gdb-index` link flag + +This flag (in our rules enabled via the `"USE_GDB_INDEX"` flag) enables, in +linkers that support it, an optimization which bundles certain debug symbols +sections together into a single `.gdb_index` section, reducing the size of the +final artifact (quite significantly for large artifacts) and drastically +improving the debugger start time, but at the cost of a slower linking step. + +As mentioned, this flag is not available to all linkers, so below is a +non-exhaustive listing to guide you: + +- Known supported linkers: `lld` (LLVM >=7), `gold`* (binutils >=2.24), `mold` (>=2.3) + + *As per the [release notes](https://lwn.net/Articles/1007541/), the main + `binutils` 2.44 (2025-02-02) release tarball does **NOT** contain the `gold` + linker anymore, but can still be acquired moving forward (for an unknown, but + still presumably reasonable long time) in a separate amended release tarball + for all even-numbered releases of `binutils` >=2.44. Importantly, this also + affects the distribution of the DWARF packing tool (`dwp`) with `binutils`, + which will also be available moving forwards only together with the `gold` + linker. + +- Known unsupported linkers: `ld` (GNU) + +### Separate types debug sections and artifact size reduction + +The `-fdebug-types-section` compile flag (in our rules enabled via the +`"USE_DEBUG_TYPES_SECTION"` flag) enables, for toolchains supporting at least +DWARFv4 (i.e., any reasonably modern toolchain), an optimization that produces +separate debug symbols sections for certain large data types, thus providing the +linker the opportunity to deduplicate more debug information, resulting in +smaller artifacts. + +More performant approaches to reduce the size of the debug information exist, +but are not as straight-forward to implement as enabling a flag. For example, +Fedora opted instead to use the `dwz` compression tool, another known approach, +and make it its default for handling debug RPMs already since version 18 +(2013-01-15). This can be a possible future avenue for further expansion of our +rules' debug fission feature set. + +References +---------- + +- The introduction of [debug fission](https://gcc.gnu.org/wiki/DebugFission) + support in GCC 4.7. +- A comprehensive and modern look into + [separating debug symbols from executables](https://www.tweag.io/blog/2023-11-23-debug-fission). |