path: root/doc/tutorial/third-party-software.org

* Building Third-party Software

Third-party projects usually ship with their own build description, which often
happens to be not compatible with justbuild. Nevertheless, it is highly
desireable to include external projects via their source code base, instead of
relying on the integration of out-of-band binary distributions. justbuild offers
a flexible approach to provide the required build description via an overlay
layer without the need to touch the original code base.

In the remainder of this section, we will use the open-source project
[[https://github.com/fmtlib/fmt][fmtlib]] as an example for integrating
third-party software to a justbuild project.

** Creating the target overlay layer for fmtlib

Before we construct the overlay layer for fmtlib, we need to determine its file
structure ([[https://github.com/fmtlib/fmt/tree/8.1.1][tag 8.1.1]]). The
relevant header and source files are structured as follows:

#+BEGIN_SRC
  fmt
  |
  +--include
  |  +--fmt
  |     +--*.h
  |
  +--src
     +--format.cc
     +--os.cc
#+END_SRC

The public headers can be found in ~include/fmt~, while the library's source
files are located in ~src~. For the overlay layer, the ~TARGETS~ files should be
placed in a tree structure that resembles the original code base's structure.
It is also good practice to provide a top-level ~TARGETS~ file, leading to the
following structure for the overlay:

#+BEGIN_SRC
  fmt-layer
  |
  +--TARGETS
  +--include
  |  +--fmt
  |     +--TARGETS
  |
  +--src
     +--TARGETS
#+END_SRC

Let's create the overlay structure:

#+BEGIN_SRC sh
$ mkdir -p fmt-layer/include/fmt
$ mkdir -p fmt-layer/src
#+END_SRC

The directory ~include/fmt~ contains only header files. As we want all files in
this directory to be included in the ~"header directory"~ target, we can safely
use the explicit ~TREE~ reference[fn:1], which collects /all/ directory contents
from ~"."~. Note that the ~TARGETS~ file is only part of the overlay, and
therefore will not be part of this tree. Furthermore, this tree should be staged
to ~"fmt"~, so that any consumer can include those headers via ~<fmt/...>~. The
resulting header directory target ~"hdrs"~ in ~include/fmt/TARGETS~ should be
described as:

[fn:1] Explicit ~TREE~ references are always a list of length 3, to distinguish
them from target references of length 2 (module and target name). Furthermore,
the second list element is always ~null~ as we only want to allow tree
references from the current module.


#+BEGIN_SRC js
{ "hdrs":
  { "type": ["@", "rules", "CC", "header directory"]
  , "hdrs": [["TREE", null, "."]]
  , "stage": ["fmt"]
  }
}
#+END_SRC

The actual library target is defined in the directory ~src~. For the public
headers, it refers to the previously created header directory via its
fully-qualified target name (~["include/fmt", "hdrs"]~). Source files are the
two local files ~format.cc~, and ~os.cc~. The final target description in
~src/TARGETS~ will look like this:

#+BEGIN_SRC js
{ "fmt":
  { "type": ["@", "rules", "CC", "library"]
  , "name": ["fmt"]
  , "hdrs": [["include/fmt", "hdrs"]]
  , "srcs": ["format.cc", "os.cc"]
  }
}
#+END_SRC

Finally, the top-level ~TARGETS~ file can be created. While it is technically
not strictly required, it is considered good practice to /export/ every target
that may be used by another project. Exported targets are subject to high-level
target caching, which allows to skip the analysis and traversal of entire
subgraphs in the action graph. Therefore, we create an export target that
exports the target ~["src", "fmt"]~, with only the variables in the field
~"flexible_config"~ being propagated. The top-level ~TARGET~ file contains the
following content:

#+BEGIN_SRC js
{ "fmt":
  { "type": "export"
  , "target": ["src", "fmt"]
  , "flexible_config": ["CXX", "CXXFLAGS", "AR", "ENV"]
  }
}
#+END_SRC

** Adding fmtlib to the Multi-Repository Configuration

Based on the /hello world/ tutorial, we can extend the existing ~repos.json~ by
the layer definition ~"fmt-targets-layer"~ and the repository ~"fmtlib"~, which
is based on the Git repository with its target root being overlayed.
Furthermore, we want to use ~"fmtlib"~ in the repository ~"tutorial"~, and
therefore need to introduce an additional binding ~"format"~ for it:

#+BEGIN_SRC js
{ "repositories":
  { "tutorial":
    { "repository": {"type": "file", "path": "."}
    , "bindings": {"rules": "just-rules", "format": "fmtlib"}
    }
  , "just-rules": { /* ... unchanged ... */ }
  , "tutorial-defaults": { /* ... unchanged ... */ }
  , "fmt-targets-layer":
    { "repository": {"type": "file", "path": "./fmt-layer"}
    }
  , "fmtlib":
    { "repository":
      { "type": "git"
      , "branch": "8.1.1"
      , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9"
      , "repository": "https://github.com/fmtlib/fmt.git"
      }
    , "target_root": "fmt-targets-layer"
    , "bindings": {"rules": "just-rules"}
    }
  }
}
#+END_SRC

This ~"format"~ binding can you be used to add a new dependency in
~greet/TARGETS~:

#+BEGIN_SRC js
{ "greet":
  { "type": ["@", "rules", "CC", "library"]
  , "name": ["greet"]
  , "hdrs": ["greet.hpp"]
  , "srcs": ["greet.cpp"]
  , "stage": ["greet"]
  , "deps": [["@", "format", "", "fmt"]]
  }
}
#+END_SRC

Consequently, the ~fmtlib~ library can now be used by ~greet/greet.cpp~:

#+BEGIN_SRC cpp
#include "greet.hpp"
#include <fmt/format.h>

void greet(std::string const& s) {
  fmt::print("Hello {}!\n", s);
}
#+END_SRC

Due to changes made to ~repos.json~, building this tutorial requires to rerun
~just-mr~, which will fetch the necessary sources for the external repositories:

#+BEGIN_SRC sh
$ CONF=$(/usr/src/justbuild/bin/just-mr.py -C repos.json setup tutorial)
$ just build -C $CONF helloworld
INFO: Requested target is [["@","tutorial","","helloworld"],{}]
INFO: Analysed target [["@","tutorial","","helloworld"],{}]
INFO: Export targets found: 0 cached, 0 uncached, 1 not eligible for caching
INFO: Discovered 7 actions, 5 trees, 0 blobs
INFO: Building [["@","tutorial","","helloworld"],{}].
INFO: Processed 7 actions, 1 cache hits.
INFO: Artifacts built, logical paths are:
        helloworld [e489bdd234787c49c4fefdd3b8a03c399a2d46f5:133000:x]
$
#+END_SRC

Note that the single cache hit we observe, is the compile action of ~main.cpp~,
which is unchanged.

** Employing high-level target caching

The make use of high-level target caching for exported targets, we need to
ensure that all inputs to an export target are transitively content-fixed. This
is automatically the case for Git repositories. However, the ~"fmtlib"~
repository also depends on ~"fmt-target-layer"~, ~"just-rules"~, and
~"tutorial-defaults"~. To enforce those repositories to be content-fixed as
well, the pragma ~"to_git"~ can be set, which instructs ~just-mr~ to create a
Git repository from those file roots (or determine the committed ~tree-id~ if
those roots are already under Git versioning). The ~repos.json~ with support for
high-level target caching will look like this:

#+BEGIN_SRC js
{ "repositories":
  { "tutorial":
    { "repository": {"type": "file", "path": "."}
    , "bindings": {"rules": "just-rules", "format": "fmtlib"}
    }
  , "just-rules":
    { "repository":
      { "type": "file"
      , "path": "/usr/src/justbuild/rules"
      , "pragma": {"to_git": true}
      }
    , "target_root": "tutorial-defaults"
    , "rule_root": "just-rules"
    }
  , "tutorial-defaults":
    { "repository":
      { "type": "file"
      , "path": "./tutorial-defaults"
      , "pragma": {"to_git": true}
      }
    }
  , "fmt-targets-layer":
    { "repository":
      { "type": "file"
      , "path": "./fmt-layer"
      , "pragma": {"to_git": true}
      }
    }
  , "fmtlib":
    { "repository":
      { "type": "git"
      , "branch": "master"
      , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9"
      , "repository": "https://github.com/fmtlib/fmt.git"
      }
    , "target_root": "fmt-targets-layer"
    , "bindings": {"rules": "just-rules"}
    }
  }
}
#+END_SRC

Due to changes in the repository configuration, ~just-mr~ needs to be rerun and
the benefits of the target cache should be visible on the second build:

#+BEGIN_SRC sh
$ CONF=$(/usr/src/justbuild/bin/just-mr.py -C repos.json setup tutorial)
From ~/.cache/just/tmp-workspaces/file/tmp/tutorial/tutorial-defaults
 * branch              HEAD       -> FETCH_HEAD
From ~/.cache/just/tmp-workspaces/file/tmp/tutorial/fmt-layer
 * branch              HEAD       -> FETCH_HEAD

$ just build -C $CONF helloworld
INFO: Requested target is [["@","tutorial","","helloworld"],{}]
INFO: Analysed target [["@","tutorial","","helloworld"],{}]
INFO: Export targets found: 0 cached, 1 uncached, 0 not eligible for caching
INFO: Discovered 7 actions, 5 trees, 0 blobs
INFO: Building [["@","tutorial","","helloworld"],{}].
INFO: Processed 7 actions, 7 cache hits.
INFO: Artifacts built, logical paths are:
        helloworld [e489bdd234787c49c4fefdd3b8a03c399a2d46f5:133000:x]

$ just build -C $CONF helloworld
INFO: Requested target is [["@","tutorial","","helloworld"],{}]
INFO: Analysed target [["@","tutorial","","helloworld"],{}]
INFO: Export targets found: 1 cached, 0 uncached, 0 not eligible for caching
INFO: Discovered 4 actions, 2 trees, 0 blobs
INFO: Building [["@","tutorial","","helloworld"],{}].
INFO: Processed 4 actions, 4 cache hits.
INFO: Artifacts built, logical paths are:
        helloworld [e489bdd234787c49c4fefdd3b8a03c399a2d46f5:133000:x]
$
#+END_SRC

Note that in the second run the export target ~"fmt"~ was taken from cache and
its 3 actions were eliminated, as their result has been recorded to the
high-level target cache during the first run.

** Combining overlay layers for multiple projects

Projects typically depend on multiple external repositories. Creating an overlay
layer for each external repository might unnecessarily clutter up the repository
configuration and the file structure of your repository. One solution to
mitigate this issue is to combine the ~TARGETS~ files of multiple external
repositories in a single overlay layer. To avoid conflicts, the ~TARGETS~ files
can be assigned different file names per repository. As an example, imagine a
common overlay layer with the files ~TARGETS.fmt~ and ~TARGETS.gsl~ for the
repositories ~"fmtlib"~ and ~"gsl-lite"~, respectively:

#+BEGIN_SRC
  common-layer
  |
  +--TARGETS.fmt
  +--TARGETS.gsl
  +--include
  |  +--fmt
  |     +--TARGETS.fmt
  |
  +--src
     +--TARGETS.fmt
#+END_SRC

Such a common overlay layer can be used as the target root for both repositories
with only one difference: the ~"target_file_name"~ field. By specifying this
field, the dispatch where to find the respective target description for each
repository is implemented. For the given example, the following ~repos.json~
defines the overlay ~"common-targets-layer"~, which is used by ~"fmtlib"~ and
~"gsl-lite"~:

#+BEGIN_SRC js
{ "repositories":
  { "tutorial": { /* ... unchanged ... */ }
  , "just-rules": { /* ... unchanged ... */ }
  , "tutorial-defaults": { /* ... unchanged ... */ }
  , "common-targets-layer":
    { "repository":
      { "type": "file"
      , "path": "./common-layer"
      , "pragma": {"to_git": true}
      }
    }
  , "fmtlib":
    { "repository":
      { "type": "git"
      , "branch": "8.1.1"
      , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9"
      , "repository": "https://github.com/fmtlib/fmt.git"
      }
    , "target_root": "common-targets-layer"
    , "target_file_name": "TARGETS.fmt"
    , "bindings": {"rules": "just-rules"}
    }
  , "gsl-lite":
    { "repository":
      { "type": "git"
      , "branch": "v0.40.0"
      , "commit": "d6c8af99a1d95b3db36f26b4f22dc3bad89952de"
      , "repository": "https://github.com/gsl-lite/gsl-lite.git"
      }
    , "target_root": "common-targets-layer"
    , "target_file_name": "TARGETS.gsl"
    , "bindings": {"rules": "just-rules"}
    }
  }
}
#+END_SRC