diff options
Diffstat (limited to 'doc/tutorial/third-party-software.org')
| -rw-r--r-- | doc/tutorial/third-party-software.org | 475 |
1 files changed, 0 insertions, 475 deletions
diff --git a/doc/tutorial/third-party-software.org b/doc/tutorial/third-party-software.org deleted file mode 100644 index d1712cc8..00000000 --- a/doc/tutorial/third-party-software.org +++ /dev/null @@ -1,475 +0,0 @@ -* Building Third-party Software - -Third-party projects usually ship with their own build description, which often -happens to be not compatible with justbuild. Nevertheless, it is highly -desireable to include external projects via their source code base, instead of -relying on the integration of out-of-band binary distributions. justbuild offers -a flexible approach to provide the required build description via an overlay -layer without the need to touch the original code base. - -For the remainder of this section, we expect to have the project files available -resulting from successfully completing the tutorial section on /Building C++ -Hello World/. We will demonstrate how to use the open-source project -[[https://github.com/fmtlib/fmt][fmtlib]] as an example for integrating -third-party software to a justbuild project. - -** Creating the target overlay layer for fmtlib - -Before we construct the overlay layer for fmtlib, we need to determine its file -structure ([[https://github.com/fmtlib/fmt/tree/8.1.1][tag 8.1.1]]). The -relevant header and source files are structured as follows: - -#+BEGIN_SRC - fmt - | - +--include - | +--fmt - | +--*.h - | - +--src - +--format.cc - +--os.cc -#+END_SRC - -The public headers can be found in ~include/fmt~, while the library's source -files are located in ~src~. For the overlay layer, the ~TARGETS~ files should be -placed in a tree structure that resembles the original code base's structure. -It is also good practice to provide a top-level ~TARGETS~ file, leading to the -following structure for the overlay: - -#+BEGIN_SRC - fmt-layer - | - +--TARGETS - +--include - | +--fmt - | +--TARGETS - | - +--src - +--TARGETS -#+END_SRC - -Let's create the overlay structure: - -#+BEGIN_SRC sh -$ mkdir -p fmt-layer/include/fmt -$ mkdir -p fmt-layer/src -#+END_SRC - -The directory ~include/fmt~ contains only header files. As we want all files in -this directory to be included in the ~"hdrs"~ target, we can safely -use the explicit ~TREE~ reference[fn:1], which collects, in a single -artifact (describing a directory) /all/ directory contents -from ~"."~ of the workspace root. Note that the ~TARGETS~ file is only part of -the overlay, and -therefore will not be part of this tree. Furthermore, this tree should be staged -to ~"fmt"~, so that any consumer can include those headers via ~<fmt/...>~. The -resulting header directory target ~"hdrs"~ in ~include/fmt/TARGETS~ should be -described as: - -[fn:1] Explicit ~TREE~ references are always a list of length 3, to distinguish -them from target references of length 2 (module and target name). Furthermore, -the second list element is always ~null~ as we only want to allow tree -references from the current module. - - -#+SRCNAME: fmt-layer/include/fmt/TARGETS -#+BEGIN_SRC js -{ "hdrs": - { "type": ["@", "rules", "data", "staged"] - , "srcs": [["TREE", null, "."]] - , "stage": ["fmt"] - } -} -#+END_SRC - -The actual library target is defined in the directory ~src~. For the public -headers, it refers to the previously created ~"hdrs"~ target via its -fully-qualified target name (~["include/fmt", "hdrs"]~). Source files are the -two local files ~format.cc~, and ~os.cc~. The final target description in -~src/TARGETS~ will look like this: - -#+SRCNAME: fmt-layer/src/TARGETS -#+BEGIN_SRC js -{ "fmt": - { "type": ["@", "rules", "CC", "library"] - , "name": ["fmt"] - , "hdrs": [["include/fmt", "hdrs"]] - , "srcs": ["format.cc", "os.cc"] - } -} -#+END_SRC - -Finally, the top-level ~TARGETS~ file can be created. While it is technically -not strictly required, it is considered good practice to /export/ every target -that may be used by another project. Exported targets are subject to high-level -target caching, which allows to skip the analysis and traversal of entire -subgraphs in the action graph. Therefore, we create an export target that -exports the target ~["src", "fmt"]~, with only the variables in the field -~"flexible_config"~ being propagated. The top-level ~TARGETS~ file contains the -following content: - -#+SRCNAME: fmt-layer/TARGETS -#+BEGIN_SRC js -{ "fmt": - { "type": "export" - , "target": ["src", "fmt"] - , "flexible_config": ["CXX", "CXXFLAGS", "ADD_CXXFLAGS", "AR", "ENV"] - } -} -#+END_SRC - -After adding the library to the multi-repository configuration (next -step), the list of configuration variables a target, like ~["src", -"fmt"]~, actually depends on can be obtained using the ~--dump-vars~ -option of the ~analyse~ subcommand. In this way, an informed decision -can be taken when deciding which variables of the export target to -make tunable for the consumer. - -** Adding fmtlib to the Multi-Repository Configuration - -Based on the /hello world/ tutorial, we can extend the existing ~repos.json~ by -the layer definition ~"fmt-targets-layer"~ and the repository ~"fmtlib"~, which -is based on the Git repository with its target root being overlayed. -Furthermore, we want to use ~"fmtlib"~ in the repository ~"tutorial"~, and -therefore need to introduce an additional binding ~"format"~ for it: - -#+SRCNAME: repos.json -#+BEGIN_SRC js -{ "main": "tutorial" -, "repositories": - { "rules-cc": - { "repository": - { "type": "git" - , "branch": "master" - , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" - , "repository": "https://github.com/just-buildsystem/rules-cc.git" - , "subdir": "rules" - } - , "target_root": "tutorial-defaults" - , "rule_root": "rules-cc" - } - , "tutorial": - { "repository": {"type": "file", "path": "."} - , "bindings": {"rules": "rules-cc", "format": "fmtlib"} - } - , "tutorial-defaults": - { "repository": {"type": "file", "path": "./tutorial-defaults"} - } - , "fmt-targets-layer": - { "repository": {"type": "file", "path": "./fmt-layer"} - } - , "fmtlib": - { "repository": - { "type": "git" - , "branch": "8.1.1" - , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9" - , "repository": "https://github.com/fmtlib/fmt.git" - } - , "target_root": "fmt-targets-layer" - , "bindings": {"rules": "rules-cc"} - } - } -} -#+END_SRC - -This ~"format"~ binding can you be used to add a new private dependency in -~greet/TARGETS~: - -#+SRCNAME: greet/TARGETS -#+BEGIN_SRC js -{ "greet": - { "type": ["@", "rules", "CC", "library"] - , "name": ["greet"] - , "hdrs": ["greet.hpp"] - , "srcs": ["greet.cpp"] - , "stage": ["greet"] - , "private-deps": [["@", "format", "", "fmt"]] - } -} -#+END_SRC - -Consequently, the ~fmtlib~ library can now be used by ~greet/greet.cpp~: - -#+SRCNAME: greet/greet.cpp -#+BEGIN_SRC cpp -#include "greet.hpp" -#include <fmt/format.h> - -void greet(std::string const& s) { - fmt::print("Hello {}!\n", s); -} -#+END_SRC - -Due to changes made to ~repos.json~, building this tutorial requires to rerun -~just-mr~, which will fetch the necessary sources for the external repositories: - -#+BEGIN_SRC sh -$ just-mr build helloworld -INFO: Requested target is [["@","tutorial","","helloworld"],{}] -INFO: Analysed target [["@","tutorial","","helloworld"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 1 not eligible for caching -INFO: Discovered 7 actions, 3 trees, 0 blobs -INFO: Building [["@","tutorial","","helloworld"],{}]. -INFO: Processed 7 actions, 1 cache hits. -INFO: Artifacts built, logical paths are: - helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x] -$ -#+END_SRC - -Note to build the ~fmt~ target alone, its containing repository ~fmtlib~ must be -specified via the ~--main~ option: -#+BEGIN_SRC sh -$ just-mr --main fmtlib build fmt -INFO: Requested target is [["@","fmtlib","","fmt"],{}] -INFO: Analysed target [["@","fmtlib","","fmt"],{}] -INFO: Export targets found: 0 cached, 0 uncached, 1 not eligible for caching -INFO: Discovered 3 actions, 1 trees, 0 blobs -INFO: Building [["@","fmtlib","","fmt"],{}]. -INFO: Processed 3 actions, 3 cache hits. -INFO: Artifacts built, logical paths are: - libfmt.a [513b2ac17c557675fc841f3ebf279003ff5a73ae:240914:f] - (1 runfiles omitted.) -$ -#+END_SRC - -** Employing high-level target caching - -The make use of high-level target caching for exported targets, we need to -ensure that all inputs to an export target are transitively content-fixed. This -is automatically the case for ~"type":"git"~ repositories. However, the ~libfmt~ -repository also depends on ~"rules-cc"~, ~"tutorial-defaults"~, and -~"fmt-target-layer"~. As the latter two are ~"type":"file"~ repositories, they -must be put under Git versioning first: - -#+BEGIN_SRC sh -$ git init . -$ git add tutorial-defaults fmt-layer -$ git commit -m"fix compile flags and fmt targets layer" -#+END_SRC - -Note that ~rules-cc~ already is under Git versioning. - -Now, to instruct ~just-mr~ to use the content-fixed, committed source trees of -those ~"type":"file"~ repositories the pragma ~"to_git"~ must be set for them in -~repos.json~: - -#+SRCNAME: repos.json -#+BEGIN_SRC js -{ "main": "tutorial" -, "repositories": - { "rules-cc": - { "repository": - { "type": "git" - , "branch": "master" - , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" - , "repository": "https://github.com/just-buildsystem/rules-cc.git" - , "subdir": "rules" - } - , "target_root": "tutorial-defaults" - , "rule_root": "rules-cc" - } - , "tutorial": - { "repository": {"type": "file", "path": "."} - , "bindings": {"rules": "rules-cc", "format": "fmtlib"} - } - , "tutorial-defaults": - { "repository": - { "type": "file" - , "path": "./tutorial-defaults" - , "pragma": {"to_git": true} - } - } - , "fmt-targets-layer": - { "repository": - { "type": "file" - , "path": "./fmt-layer" - , "pragma": {"to_git": true} - } - } - , "fmtlib": - { "repository": - { "type": "git" - , "branch": "master" - , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9" - , "repository": "https://github.com/fmtlib/fmt.git" - } - , "target_root": "fmt-targets-layer" - , "bindings": {"rules": "rules-cc"} - } - } -} -#+END_SRC - -Due to changes in the repository configuration, we need to rebuild and the -benefits of the target cache should be visible on the second build: - -#+BEGIN_SRC sh -$ just-mr build helloworld -INFO: Requested target is [["@","tutorial","","helloworld"],{}] -INFO: Analysed target [["@","tutorial","","helloworld"],{}] -INFO: Export targets found: 0 cached, 1 uncached, 0 not eligible for caching -INFO: Discovered 7 actions, 3 trees, 0 blobs -INFO: Building [["@","tutorial","","helloworld"],{}]. -INFO: Processed 7 actions, 7 cache hits. -INFO: Artifacts built, logical paths are: - helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x] -$ -$ just-mr build helloworld -INFO: Requested target is [["@","tutorial","","helloworld"],{}] -INFO: Analysed target [["@","tutorial","","helloworld"],{}] -INFO: Export targets found: 1 cached, 0 uncached, 0 not eligible for caching -INFO: Discovered 4 actions, 2 trees, 0 blobs -INFO: Building [["@","tutorial","","helloworld"],{}]. -INFO: Processed 4 actions, 4 cache hits. -INFO: Artifacts built, logical paths are: - helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x] -$ -#+END_SRC - -Note that in the second run the export target ~"fmt"~ was taken from cache and -its 3 actions were eliminated, as their result has been recorded to the -high-level target cache during the first run. - -** Combining overlay layers for multiple projects - -Projects typically depend on multiple external repositories. Creating an overlay -layer for each external repository might unnecessarily clutter up the repository -configuration and the file structure of your repository. One solution to -mitigate this issue is to combine the ~TARGETS~ files of multiple external -repositories in a single overlay layer. To avoid conflicts, the ~TARGETS~ files -can be assigned different file names per repository. As an example, imagine a -common overlay layer with the files ~TARGETS.fmt~ and ~TARGETS.gsl~ for the -repositories ~"fmtlib"~ and ~"gsl-lite"~, respectively: - -#+BEGIN_SRC - common-layer - | - +--TARGETS.fmt - +--TARGETS.gsl - +--include - | +--fmt - | | +--TARGETS.fmt - | +--gsl - | +--TARGETS.gsl - | - +--src - +--TARGETS.fmt -#+END_SRC - -Such a common overlay layer can be used as the target root for both repositories -with only one difference: the ~"target_file_name"~ field. By specifying this -field, the dispatch where to find the respective target description for each -repository is implemented. For the given example, the following ~repos.json~ -defines the overlay ~"common-targets-layer"~, which is used by ~"fmtlib"~ and -~"gsl-lite"~: - -#+SRCNAME: repos.json -#+BEGIN_SRC js -{ "main": "tutorial" -, "repositories": - { "rules-cc": - { "repository": - { "type": "git" - , "branch": "master" - , "commit": "123d8b03bf2440052626151c14c54abce2726e6f" - , "repository": "https://github.com/just-buildsystem/rules-cc.git" - , "subdir": "rules" - } - , "target_root": "tutorial-defaults" - , "rule_root": "rules-cc" - } - , "tutorial": - { "repository": {"type": "file", "path": "."} - , "bindings": {"rules": "rules-cc", "format": "fmtlib"} - } - , "tutorial-defaults": - { "repository": - { "type": "file" - , "path": "./tutorial-defaults" - , "pragma": {"to_git": true} - } - } - , "common-targets-layer": - { "repository": - { "type": "file" - , "path": "./common-layer" - , "pragma": {"to_git": true} - } - } - , "fmtlib": - { "repository": - { "type": "git" - , "branch": "8.1.1" - , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9" - , "repository": "https://github.com/fmtlib/fmt.git" - } - , "target_root": "common-targets-layer" - , "target_file_name": "TARGETS.fmt" - , "bindings": {"rules": "rules-cc"} - } - , "gsl-lite": - { "repository": - { "type": "git" - , "branch": "v0.40.0" - , "commit": "d6c8af99a1d95b3db36f26b4f22dc3bad89952de" - , "repository": "https://github.com/gsl-lite/gsl-lite.git" - } - , "target_root": "common-targets-layer" - , "target_file_name": "TARGETS.gsl" - , "bindings": {"rules": "rules-cc"} - } - } -} -#+END_SRC - -** Using pre-built dependencies - -While building external dependencies from source brings advantages, -most prominently the flexibility to quickly and seamlessly switch -to a different build configuration (production, debug, instrumented -for performance analysis; cross-compiling for a different target -architecture), there are also legitimate reasons to use pre-built -dependencies. The most prominent one is if your project is packaged -as part of a larger distribution. For that reason, just also has (in -~etc/import.prebuilt~) target files for all its dependencies assuming -they are pre-installed. The reason why target files are used at -all for this situation is twofold. -- On the one hand, having a target allows the remaining targets - to not care about where their dependencies come from, or if it - is a build against pre-installed dependencies or not. Also, the - top-level binary does not have to know the linking requirements - of its transitive dependencies. In other words, information stays - where it belongs to and if one target acquires a new dependency, - the information is automatically propagated to all targets using it. -- Still some information is needed to use a pre-installed library - and, as explained, a target describing the pre-installed library - is the right place to collect this information. - - The public header files of the library. By having this explicit, - we do not accumulate directories in the include search path - and hence also properly detect include conflicts. - - The information on how to link the library itself (i.e., - basically its base name). - - Any dependencies on other libraries that the library might have. - This information is used to obtain the correct linking order - and complete transitive linking arguments while keeping the - description maintainable, as each target still only declares - its direct dependencies. - -The target description for a pre-built version of the format -library that was used as an example in this section is shown next; -with our staging mechanism the logical repository it belongs to is -rooted in the ~fmt~ subdirectory of the ~include~ directory of the -ambient system. - -#+SRCNAME: etc/import.prebuilt/TARGETS.fmt -#+BEGIN_SRC js -{ "fmt": - { "type": ["@", "rules", "CC", "library"] - , "name": ["fmt"] - , "stage": ["fmt"] - , "hdrs": [["TREE", null, "."]] - , "private-ldflags": ["-lfmt"] - } -} -#+END_SRC |