path: root/doc/tutorial/third-party-software.org

* Building Third-party Software

Third-party projects usually ship with their own build description, which often
happens to be not compatible with justbuild. Nevertheless, it is highly
desireable to include external projects via their source code base, instead of
relying on the integration of out-of-band binary distributions. justbuild offers
a flexible approach to provide the required build description via an overlay
layer without the need to touch the original code base.

For the remainder of this section, we expect to have the project files available
resulting from successfully completing the tutorial section on /Building C++
Hello World/. We will demonstrate how to use the open-source project
[[https://github.com/fmtlib/fmt][fmtlib]] as an example for integrating
third-party software to a justbuild project.

** Creating the target overlay layer for fmtlib

Before we construct the overlay layer for fmtlib, we need to determine its file
structure ([[https://github.com/fmtlib/fmt/tree/8.1.1][tag 8.1.1]]). The
relevant header and source files are structured as follows:

#+BEGIN_SRC
  fmt
  |
  +--include
  |  +--fmt
  |     +--*.h
  |
  +--src
     +--format.cc
     +--os.cc
#+END_SRC

The public headers can be found in ~include/fmt~, while the library's source
files are located in ~src~. For the overlay layer, the ~TARGETS~ files should be
placed in a tree structure that resembles the original code base's structure.
It is also good practice to provide a top-level ~TARGETS~ file, leading to the
following structure for the overlay:

#+BEGIN_SRC
  fmt-layer
  |
  +--TARGETS
  +--include
  |  +--fmt
  |     +--TARGETS
  |
  +--src
     +--TARGETS
#+END_SRC

Let's create the overlay structure:

#+BEGIN_SRC sh
$ mkdir -p fmt-layer/include/fmt
$ mkdir -p fmt-layer/src
#+END_SRC

The directory ~include/fmt~ contains only header files. As we want all files in
this directory to be included in the ~"hdrs"~ target, we can safely
use the explicit ~TREE~ reference[fn:1], which collects, in a single
artifact (describing a directory) /all/ directory contents
from ~"."~ of the workspace root. Note that the ~TARGETS~ file is only part of
the overlay, and
therefore will not be part of this tree. Furthermore, this tree should be staged
to ~"fmt"~, so that any consumer can include those headers via ~<fmt/...>~. The
resulting header directory target ~"hdrs"~ in ~include/fmt/TARGETS~ should be
described as:

[fn:1] Explicit ~TREE~ references are always a list of length 3, to distinguish
them from target references of length 2 (module and target name). Furthermore,
the second list element is always ~null~ as we only want to allow tree
references from the current module.


#+SRCNAME: fmt-layer/include/fmt/TARGETS
#+BEGIN_SRC js
{ "hdrs":
  { "type": ["@", "rules", "data", "staged"]
  , "srcs": [["TREE", null, "."]]
  , "stage": ["fmt"]
  }
}
#+END_SRC

The actual library target is defined in the directory ~src~. For the public
headers, it refers to the previously created ~"hdrs"~ target via its
fully-qualified target name (~["include/fmt", "hdrs"]~). Source files are the
two local files ~format.cc~, and ~os.cc~. The final target description in
~src/TARGETS~ will look like this:

#+SRCNAME: fmt-layer/src/TARGETS
#+BEGIN_SRC js
{ "fmt":
  { "type": ["@", "rules", "CC", "library"]
  , "name": ["fmt"]
  , "hdrs": [["include/fmt", "hdrs"]]
  , "srcs": ["format.cc", "os.cc"]
  }
}
#+END_SRC

Finally, the top-level ~TARGETS~ file can be created. While it is technically
not strictly required, it is considered good practice to /export/ every target
that may be used by another project. Exported targets are subject to high-level
target caching, which allows to skip the analysis and traversal of entire
subgraphs in the action graph. Therefore, we create an export target that
exports the target ~["src", "fmt"]~, with only the variables in the field
~"flexible_config"~ being propagated. The top-level ~TARGETS~ file contains the
following content:

#+SRCNAME: fmt-layer/TARGETS
#+BEGIN_SRC js
{ "fmt":
  { "type": "export"
  , "target": ["src", "fmt"]
  , "flexible_config": ["CXX", "CXXFLAGS", "ADD_CXXFLAGS", "AR", "ENV"]
  }
}
#+END_SRC

After adding the library to the multi-repository configuration (next
step), the list of configuration variables a target, like ~["src",
"fmt"]~, actually depends on can be obtained using the ~--dump-vars~
option of the ~analyse~ subcommand. In this way, an informed decision
can be taken when deciding which variables of the export target to
make tunable for the consumer.

** Adding fmtlib to the Multi-Repository Configuration

Based on the /hello world/ tutorial, we can extend the existing ~repos.json~ by
the layer definition ~"fmt-targets-layer"~ and the repository ~"fmtlib"~, which
is based on the Git repository with its target root being overlayed.
Furthermore, we want to use ~"fmtlib"~ in the repository ~"tutorial"~, and
therefore need to introduce an additional binding ~"format"~ for it:

#+SRCNAME: repos.json
#+BEGIN_SRC js
{ "main": "tutorial"
, "repositories":
  { "rules-cc":
    { "repository":
      { "type": "git"
      , "branch": "master"
      , "commit": "123d8b03bf2440052626151c14c54abce2726e6f"
      , "repository": "https://github.com/just-buildsystem/rules-cc.git"
      , "subdir": "rules"
      }
    , "target_root": "tutorial-defaults"
    , "rule_root": "rules-cc"
    }
  , "tutorial":
    { "repository": {"type": "file", "path": "."}
    , "bindings": {"rules": "rules-cc", "format": "fmtlib"}
    }
  , "tutorial-defaults":
    { "repository": {"type": "file", "path": "./tutorial-defaults"}
    }
  , "fmt-targets-layer":
    { "repository": {"type": "file", "path": "./fmt-layer"}
    }
  , "fmtlib":
    { "repository":
      { "type": "git"
      , "branch": "8.1.1"
      , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9"
      , "repository": "https://github.com/fmtlib/fmt.git"
      }
    , "target_root": "fmt-targets-layer"
    , "bindings": {"rules": "rules-cc"}
    }
  }
}
#+END_SRC

This ~"format"~ binding can you be used to add a new private dependency in
~greet/TARGETS~:

#+SRCNAME: greet/TARGETS
#+BEGIN_SRC js
{ "greet":
  { "type": ["@", "rules", "CC", "library"]
  , "name": ["greet"]
  , "hdrs": ["greet.hpp"]
  , "srcs": ["greet.cpp"]
  , "stage": ["greet"]
  , "private-deps": [["@", "format", "", "fmt"]]
  }
}
#+END_SRC

Consequently, the ~fmtlib~ library can now be used by ~greet/greet.cpp~:

#+SRCNAME: greet/greet.cpp
#+BEGIN_SRC cpp
#include "greet.hpp"
#include <fmt/format.h>

void greet(std::string const& s) {
  fmt::print("Hello {}!\n", s);
}
#+END_SRC

Due to changes made to ~repos.json~, building this tutorial requires to rerun
~just-mr~, which will fetch the necessary sources for the external repositories:

#+BEGIN_SRC sh
$ just-mr build helloworld
INFO: Requested target is [["@","tutorial","","helloworld"],{}]
INFO: Analysed target [["@","tutorial","","helloworld"],{}]
INFO: Export targets found: 0 cached, 0 uncached, 1 not eligible for caching
INFO: Discovered 7 actions, 3 trees, 0 blobs
INFO: Building [["@","tutorial","","helloworld"],{}].
INFO: Processed 7 actions, 1 cache hits.
INFO: Artifacts built, logical paths are:
        helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x]
$
#+END_SRC

Note to build the ~fmt~ target alone, its containing repository ~fmtlib~ must be
specified via the ~--main~ option:
#+BEGIN_SRC sh
$ just-mr --main fmtlib build fmt
INFO: Requested target is [["@","fmtlib","","fmt"],{}]
INFO: Analysed target [["@","fmtlib","","fmt"],{}]
INFO: Export targets found: 0 cached, 0 uncached, 1 not eligible for caching
INFO: Discovered 3 actions, 1 trees, 0 blobs
INFO: Building [["@","fmtlib","","fmt"],{}].
INFO: Processed 3 actions, 3 cache hits.
INFO: Artifacts built, logical paths are:
        libfmt.a [513b2ac17c557675fc841f3ebf279003ff5a73ae:240914:f]
      (1 runfiles omitted.)
$
#+END_SRC

** Employing high-level target caching

The make use of high-level target caching for exported targets, we need to
ensure that all inputs to an export target are transitively content-fixed. This
is automatically the case for ~"type":"git"~ repositories. However, the ~libfmt~
repository also depends on ~"rules-cc"~, ~"tutorial-defaults"~, and
~"fmt-target-layer"~. As the latter two are ~"type":"file"~ repositories, they
must be put under Git versioning first:

#+BEGIN_SRC sh
$ git init .
$ git add tutorial-defaults fmt-layer
$ git commit -m"fix compile flags and fmt targets layer"
#+END_SRC

Note that ~rules-cc~ already is under Git versioning.

Now, to instruct ~just-mr~ to use the content-fixed, committed source trees of
those ~"type":"file"~ repositories the pragma ~"to_git"~ must be set for them in
~repos.json~:

#+SRCNAME: repos.json
#+BEGIN_SRC js
{ "main": "tutorial"
, "repositories":
  { "rules-cc":
    { "repository":
      { "type": "git"
      , "branch": "master"
      , "commit": "123d8b03bf2440052626151c14c54abce2726e6f"
      , "repository": "https://github.com/just-buildsystem/rules-cc.git"
      , "subdir": "rules"
      }
    , "target_root": "tutorial-defaults"
    , "rule_root": "rules-cc"
    }
  , "tutorial":
    { "repository": {"type": "file", "path": "."}
    , "bindings": {"rules": "rules-cc", "format": "fmtlib"}
    }
  , "tutorial-defaults":
    { "repository":
      { "type": "file"
      , "path": "./tutorial-defaults"
      , "pragma": {"to_git": true}
      }
    }
  , "fmt-targets-layer":
    { "repository":
      { "type": "file"
      , "path": "./fmt-layer"
      , "pragma": {"to_git": true}
      }
    }
  , "fmtlib":
    { "repository":
      { "type": "git"
      , "branch": "master"
      , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9"
      , "repository": "https://github.com/fmtlib/fmt.git"
      }
    , "target_root": "fmt-targets-layer"
    , "bindings": {"rules": "rules-cc"}
    }
  }
}
#+END_SRC

Due to changes in the repository configuration, we need to rebuild and the
benefits of the target cache should be visible on the second build:

#+BEGIN_SRC sh
$ just-mr build helloworld
INFO: Requested target is [["@","tutorial","","helloworld"],{}]
INFO: Analysed target [["@","tutorial","","helloworld"],{}]
INFO: Export targets found: 0 cached, 1 uncached, 0 not eligible for caching
INFO: Discovered 7 actions, 3 trees, 0 blobs
INFO: Building [["@","tutorial","","helloworld"],{}].
INFO: Processed 7 actions, 7 cache hits.
INFO: Artifacts built, logical paths are:
        helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x]
$
$ just-mr build helloworld
INFO: Requested target is [["@","tutorial","","helloworld"],{}]
INFO: Analysed target [["@","tutorial","","helloworld"],{}]
INFO: Export targets found: 1 cached, 0 uncached, 0 not eligible for caching
INFO: Discovered 4 actions, 2 trees, 0 blobs
INFO: Building [["@","tutorial","","helloworld"],{}].
INFO: Processed 4 actions, 4 cache hits.
INFO: Artifacts built, logical paths are:
        helloworld [0ec4e36cfb5f2c3efa0fff789349a46694a6d303:132736:x]
$
#+END_SRC

Note that in the second run the export target ~"fmt"~ was taken from cache and
its 3 actions were eliminated, as their result has been recorded to the
high-level target cache during the first run.

** Combining overlay layers for multiple projects

Projects typically depend on multiple external repositories. Creating an overlay
layer for each external repository might unnecessarily clutter up the repository
configuration and the file structure of your repository. One solution to
mitigate this issue is to combine the ~TARGETS~ files of multiple external
repositories in a single overlay layer. To avoid conflicts, the ~TARGETS~ files
can be assigned different file names per repository. As an example, imagine a
common overlay layer with the files ~TARGETS.fmt~ and ~TARGETS.gsl~ for the
repositories ~"fmtlib"~ and ~"gsl-lite"~, respectively:

#+BEGIN_SRC
  common-layer
  |
  +--TARGETS.fmt
  +--TARGETS.gsl
  +--include
  |  +--fmt
  |  |  +--TARGETS.fmt
  |  +--gsl
  |     +--TARGETS.gsl
  |
  +--src
     +--TARGETS.fmt
#+END_SRC

Such a common overlay layer can be used as the target root for both repositories
with only one difference: the ~"target_file_name"~ field. By specifying this
field, the dispatch where to find the respective target description for each
repository is implemented. For the given example, the following ~repos.json~
defines the overlay ~"common-targets-layer"~, which is used by ~"fmtlib"~ and
~"gsl-lite"~:

#+SRCNAME: repos.json
#+BEGIN_SRC js
{ "main": "tutorial"
, "repositories":
  { "rules-cc":
    { "repository":
      { "type": "git"
      , "branch": "master"
      , "commit": "123d8b03bf2440052626151c14c54abce2726e6f"
      , "repository": "https://github.com/just-buildsystem/rules-cc.git"
      , "subdir": "rules"
      }
    , "target_root": "tutorial-defaults"
    , "rule_root": "rules-cc"
    }
  , "tutorial":
    { "repository": {"type": "file", "path": "."}
    , "bindings": {"rules": "rules-cc", "format": "fmtlib"}
    }
  , "tutorial-defaults":
    { "repository":
      { "type": "file"
      , "path": "./tutorial-defaults"
      , "pragma": {"to_git": true}
      }
    }
  , "common-targets-layer":
    { "repository":
      { "type": "file"
      , "path": "./common-layer"
      , "pragma": {"to_git": true}
      }
    }
  , "fmtlib":
    { "repository":
      { "type": "git"
      , "branch": "8.1.1"
      , "commit": "b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9"
      , "repository": "https://github.com/fmtlib/fmt.git"
      }
    , "target_root": "common-targets-layer"
    , "target_file_name": "TARGETS.fmt"
    , "bindings": {"rules": "rules-cc"}
    }
  , "gsl-lite":
    { "repository":
      { "type": "git"
      , "branch": "v0.40.0"
      , "commit": "d6c8af99a1d95b3db36f26b4f22dc3bad89952de"
      , "repository": "https://github.com/gsl-lite/gsl-lite.git"
      }
    , "target_root": "common-targets-layer"
    , "target_file_name": "TARGETS.gsl"
    , "bindings": {"rules": "rules-cc"}
    }
  }
}
#+END_SRC

** Using pre-built dependencies

While building external dependencies from source brings advantages,
most prominently the flexibility to quickly and seamlessly switch
to a different build configuration (production, debug, instrumented
for performance analysis; cross-compiling for a different target
architecture), there are also legitimate reasons to use pre-built
dependencies. The most prominent one is if your project is packaged
as part of a larger distribution. For that reason, just also has (in
~etc/import.prebuilt~) target files for all its dependencies assuming
they are pre-installed. The reason why target files are used at
all for this situation is twofold.
- On the one hand, having a target allows the remaining targets
  to not care about where their dependencies come from, or if it
  is a build against pre-installed dependencies or not. Also, the
  top-level binary does not have to know the linking requirements
  of its transitive dependencies. In other words, information stays
  where it belongs to and if one target acquires a new dependency,
  the information is automatically propagated to all targets using it.
- Still some information is needed to use a pre-installed library
  and, as explained, a target describing the pre-installed library
  is the right place to collect this information.
  - The public header files of the library. By having this explicit,
    we do not accumulate directories in the include search path
    and hence also properly detect include conflicts.
  - The information on how to link the library itself (i.e.,
    basically its base name).
  - Any dependencies on other libraries that the library might have.
    This information is used to obtain the correct linking order
    and complete transitive linking arguments while keeping the
    description maintainable, as each target still only declares
    its direct dependencies.

The target description for a pre-built version of the format
library that was used as an example in this section is shown next;
with our staging mechanism the logical repository it belongs to is
rooted in the ~fmt~ subdirectory of the ~include~ directory of the
ambient system.

#+SRCNAME: etc/import.prebuilt/TARGETS.fmt
#+BEGIN_SRC js
{ "fmt":
  { "type": ["@", "rules", "CC", "library"]
  , "name": ["fmt"]
  , "stage": ["fmt"]
  , "hdrs": [["TREE", null, "."]]
  , "private-ldflags": ["-lfmt"]
  }
}
#+END_SRC