Tutorial: Add section about 3rd party software

1 files changed, 350 insertions, 0 deletions

diff --git a/doc/tutorial/third-party-software.org b/doc/tutorial/third-party-software.org
new file mode 100644
index 00000000..2832ad62
--- /dev/null
+++ b/doc/tutorial/third-party-software.org
@@ -0,0 +1,350 @@
+* Building Third-party Software
+
+Third-party projects usually ship with their own build description, which often
+happens to be not compatible with justbuild. Nevertheless, it is highly
+desireable to include external projects via their source code base, instead of
+relying on the integration of out-of-band binary distributions. justbuild offers
+a flexible approach to provide the required build description via an overlay
+layer without the need to touch the original code base.
+
+In the remainder of this section, we will use the open-source project
+[[https://github.com/fmtlib/fmt][fmtlib]] as an example for integrating
+third-party software to a justbuild project.
+
+** Creating the target overlay layer for fmtlib
+
+Before we construct the overlay layer for fmtlib, we need to determine its file
+structure ([[https://github.com/fmtlib/fmt/tree/8.1.1][tag 8.1.1]]). The
+relevant header and source files are structured as follows:
+
+#+BEGIN_SRC
+  fmt
+  |
+  +--include
+  |  +--fmt
+  |     +--*.h
+  |
+  +--src
+     +--format.cc
+     +--os.cc
+#+END_SRC
+
+The public headers can be found in ~include/fmt~, while the library's source
+files are located in ~src~. For the overlay layer, the ~TARGETS~ files should be
+placed in a tree structure that resembles the original code base's structure.
+It is also good practice to provide a top-level ~TARGETS~ file, leading to the
+following structure for the overlay:
+
+#+BEGIN_SRC
+  fmt-layer
+  |
+  +--TARGETS
+  +--include
+  |  +--fmt
+  |     +--TARGETS
+  |
+  +--src
+     +--TARGETS
+#+END_SRC
+
+Let's create the overlay structure:
+
+#+BEGIN_SRC shell
+$ mkdir -p fmt-layer/include/fmt
+$ mkdir -p fmt-layer/src
+#+END_SRC
+
+The directory ~include/fmt~ contains only header files. As we want all files in
+this directory to be included in the ~"header directory"~ target, we can safely
+use the explicit ~TREE~ reference[fn:1], which collects /all/ directory contents
+from ~"."~. Note that the ~TARGETS~ file is only part of the overlay, and
+therefore will not be part of this tree. Furthermore, this tree should be staged
+to ~"fmt"~, so that any consumer can include those headers via ~<fmt/...>~. The
+resulting header directory target ~"hdrs"~ in ~include/fmt/TARGETS~ should be
+described as:
+
+[fn:1] Explicit ~TREE~ references are always a list of length 3, to distinguish
+them from target references of length 2 (module and target name). Furthermore,
+the second list element is always ~null~ as we only want to allow tree
+references from the current module.
+
+
+#+BEGIN_SRC js
+{ "hdrs":
+  { "type": ["@", "rules", "CC", "header directory"]
+  , "hdrs": [["TREE", null, "."]]
+  , "stage": ["fmt"]
+  }
+}
+#+END_SRC
+
+The actual library target is defined in the directory ~src~. For the public
+headers, it refers to the previously created header directory via its
+fully-qualified target name (~["include/fmt", "hdrs"]~). Source files are the
+two local files ~format.cc~, and ~os.cc~. The final target description in
+~src/TARGETS~ will look like this:
+
+#+BEGIN_SRC js
+{ "fmt":
+  { "type": ["@", "rules", "CC", "library"]
+  , "name": ["fmt"]
+  , "hdrs": [["include/fmt", "hdrs"]]
+  , "srcs": ["format.cc", "os.cc"]
+  }
+}
+#+END_SRC
+
+Finally, the top-level ~TARGETS~ file can be created. While it is technically
+not strictly required, it is considered good practice to /export/ every target
+that may be used by another project. Exported targets are subject to high-level
+target caching, which allows to skip the analysis and traversal of entire
+subgraphs in the action graph. Therefore, we create an export target that
+exports the target ~["src", "fmt"]~, with only the variables in the field
+~"flexible_config"~ being propagated. The top-level ~TARGET~ file contains the
+following content:
+
+#+BEGIN_SRC js
+{ "fmt":
+  { "type": "export"
+  , "target": ["src", "fmt"]
+  , "flexible_config": ["CXX", "CXXFLAGS", "AR", "ENV"]
+  }
+}
+#+END_SRC
+
+** Adding fmtlib to the Multi-Repository Configuration
+
+Based on the /hello world/ tutorial, we can extend the existing ~repos.json~ by
+the layer definition ~"fmt-targets-layer"~ and the repository ~"fmtlib"~, which
+is based on the Git repository with its target root being overlayed.
+Furthermore, we want to use ~"fmtlib"~ in the repository ~"tutorial"~, and
+therefore need to introduce an additional binding ~"format"~ for it:
+
+#+BEGIN_SRC js
+{ "repositories":
+  { "tutorial":
+    { "repository": {"type": "file", "path": "."}
+    , "bindings": {"rules": "just-rules", "format": "fmtlib"}
+    }
+  , "just-rules": { /* ... unchanged ... */ }
+  , "tutorial-defaults": { /* ... unchanged ... */ }
+  , "fmt-targets-layer":
+    { "repository": {"type": "file", "path": "./fmt-layer"}
+    }
+  , "fmtlib":
+    { "repository":
+      { "type": "git"
+      , "branch": "8.1.1"
+      , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9"
+      , "repository": "https://github.com/fmtlib/fmt.git"
+      }
+    , "target_root": "fmt-targets-layer"
+    , "bindings": {"rules": "just-rules"}
+    }
+  }
+}
+#+END_SRC
+
+This ~"format"~ binding can you be used to add a new dependency in
+~greet/TARGETS~:
+
+#+BEGIN_SRC js
+{ "greet":
+  { "type": ["@", "rules", "CC", "library"]
+  , "name": ["greet"]
+  , "hdrs": ["greet.hpp"]
+  , "srcs": ["greet.cpp"]
+  , "stage": ["greet"]
+  , "deps": [["@", "format", "", "fmt"]]
+  }
+}
+#+END_SRC
+
+Consequently, the ~fmtlib~ library can now be used by ~greet/greet.cpp~:
+
+#+BEGIN_SRC cpp
+#include "greet.hpp"
+#include <fmt/format.h>
+
+void greet(std::string const& s) {
+  fmt::print("Hello {}!\n", s);
+}
+#+END_SRC
+
+Due to changes made to ~repos.json~, building this tutorial requires to rerun
+~just-mr~, which will fetch the necessary sources for the external repositories:
+
+#+BEGIN_SRC shell
+$ CONF=$(/usr/src/justbuild/bin/just-mr.py -C repos.json setup tutorial)
+$ just build -C $CONF helloworld
+INFO: Requested target is [["@","tutorial","","helloworld"],{}]
+INFO: Analysed target [["@","tutorial","","helloworld"],{}]
+INFO: Export targets found: 0 cached, 0 uncached, 1 not eligible for caching
+INFO: Discovered 7 actions, 5 trees, 0 blobs
+INFO: Building [["@","tutorial","","helloworld"],{}].
+INFO: Processed 7 actions, 1 cache hits.
+INFO: Artifacts built, logical paths are:
+        helloworld [e489bdd234787c49c4fefdd3b8a03c399a2d46f5:133000:x]
+#+END_SRC
+
+Note that the single cache hit we observe, is the compile action of ~main.cpp~,
+which is unchanged.
+
+** Employing high-level target caching
+
+The make use of high-level target caching for exported targets, we need to
+ensure that all inputs to an export target are transitively content-fixed. This
+is automatically the case for Git repositories. However, the ~"fmtlib"~
+repository also depends on ~"fmt-target-layer"~, ~"just-rules"~, and
+~"tutorial-defaults"~. To enforce those repositories to be content-fixed as
+well, the pragma ~"to_git"~ can be set, which instructs ~just-mr~ to create a
+Git repository from those file roots (or determine the committed ~tree-id~ if
+those roots are already under Git versioning). The ~repos.json~ with support for
+high-level target caching will look like this:
+
+#+BEGIN_SRC js
+{ "repositories":
+  { "tutorial":
+    { "repository": {"type": "file", "path": "."}
+    , "bindings": {"rules": "just-rules", "format": "fmtlib"}
+    }
+  , "just-rules":
+    { "repository":
+      { "type": "file"
+      , "path": "/usr/src/justbuild/rules"
+      , "pragma": {"to_git": true}
+      }
+    , "target_root": "tutorial-defaults"
+    , "rule_root": "just-rules"
+    }
+  , "tutorial-defaults":
+    { "repository":
+      { "type": "file"
+      , "path": "./tutorial-defaults"
+      , "pragma": {"to_git": true}
+      }
+    }
+  , "fmt-targets-layer":
+    { "repository":
+      { "type": "file"
+      , "path": "./fmt-layer"
+      , "pragma": {"to_git": true}
+      }
+    }
+  , "fmtlib":
+    { "repository":
+      { "type": "git"
+      , "branch": "master"
+      , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9"
+      , "repository": "https://github.com/fmtlib/fmt.git"
+      }
+    , "target_root": "fmt-targets-layer"
+    , "bindings": {"rules": "just-rules"}
+    }
+  }
+}
+#+END_SRC
+
+Due to changes in the repository configuration, ~just-mr~ needs to be rerun and
+the benefits of the target cache should be visible on the second build:
+
+#+BEGIN_SRC shell
+$ CONF=$(/usr/src/justbuild/bin/just-mr.py -C repos.json setup tutorial)
+From ~/.cache/just/tmp-workspaces/file/tmp/tutorial/tutorial-defaults
+ * branch              HEAD       -> FETCH_HEAD
+From ~/.cache/just/tmp-workspaces/file/tmp/tutorial/fmt-layer
+ * branch              HEAD       -> FETCH_HEAD
+
+$ just build -C $CONF helloworld
+INFO: Requested target is [["@","tutorial","","helloworld"],{}]
+INFO: Analysed target [["@","tutorial","","helloworld"],{}]
+INFO: Export targets found: 0 cached, 1 uncached, 0 not eligible for caching
+INFO: Discovered 7 actions, 5 trees, 0 blobs
+INFO: Building [["@","tutorial","","helloworld"],{}].
+INFO: Processed 7 actions, 7 cache hits.
+INFO: Artifacts built, logical paths are:
+        helloworld [e489bdd234787c49c4fefdd3b8a03c399a2d46f5:133000:x]
+
+$ just build -C $CONF helloworld
+INFO: Requested target is [["@","tutorial","","helloworld"],{}]
+INFO: Analysed target [["@","tutorial","","helloworld"],{}]
+INFO: Export targets found: 1 cached, 0 uncached, 0 not eligible for caching
+INFO: Discovered 4 actions, 2 trees, 0 blobs
+INFO: Building [["@","tutorial","","helloworld"],{}].
+INFO: Processed 4 actions, 4 cache hits.
+INFO: Artifacts built, logical paths are:
+        helloworld [e489bdd234787c49c4fefdd3b8a03c399a2d46f5:133000:x]
+#+END_SRC
+
+Note that in the second run the export target ~"fmt"~ was taken from cache and
+its 3 actions were eliminated, as their result has been recorded to the
+high-level target cache during the first run.
+
+** Combining overlay layers for multiple projects
+
+Projects typically depend on multiple external repositories. Creating an overlay
+layer for each external repository might unnecessarily clutter up the repository
+configuration and the file structure of your repository. One solution to
+mitigate this issue is to combine the ~TARGETS~ files of multiple external
+repositories in a single overlay layer. To avoid conflicts, the ~TARGETS~ files
+can be assigned different file names per repository. As an example, imagine a
+common overlay layer with the files ~TARGETS.fmt~ and ~TARGETS.gsl~ for the
+repositories ~"fmtlib"~ and ~"gsl-lite"~, respectively:
+
+#+BEGIN_SRC
+  common-layer
+  |
+  +--TARGETS.fmt
+  +--TARGETS.gsl
+  +--include
+  |  +--fmt
+  |     +--TARGETS.fmt
+  |
+  +--src
+     +--TARGETS.fmt
+#+END_SRC
+
+Such a common overlay layer can be used as the target root for both repositories
+with only one difference: the ~"target_file_name"~ field. By specifying this
+field, the dispatch where to find the respective target description for each
+repository is implemented. For the given example, the following ~repos.json~
+defines the overlay ~"common-targets-layer"~, which is used by ~"fmtlib"~ and
+~"gsl-lite"~:
+
+#+BEGIN_SRC js
+{ "repositories":
+  { "tutorial": { /* ... unchanged ... */ }
+  , "just-rules": { /* ... unchanged ... */ }
+  , "tutorial-defaults": { /* ... unchanged ... */ }
+  , "common-targets-layer":
+    { "repository":
+      { "type": "file"
+      , "path": "./common-layer"
+      , "pragma": {"to_git": true}
+      }
+    }
+  , "fmtlib":
+    { "repository":
+      { "type": "git"
+      , "branch": "8.1.1"
+      , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9"
+      , "repository": "https://github.com/fmtlib/fmt.git"
+      }
+    , "target_root": "common-targets-layer"
+    , "target_file_name": "TARGETS.fmt"
+    , "bindings": {"rules": "just-rules"}
+    }
+  , "gsl-lite":
+    { "repository":
+      { "type": "git"
+      , "branch": "v0.40.0"
+      , "commit": "d6c8af99a1d95b3db36f26b4f22dc3bad89952de"
+      , "repository": "https://github.com/gsl-lite/gsl-lite.git"
+      }
+    , "target_root": "common-targets-layer"
+    , "target_file_name": "TARGETS.gsl"
+    , "bindings": {"rules": "just-rules"}
+    }
+  }
+}
+#+END_SRC