Initial commit

1 files changed, 167 insertions, 0 deletions

diff --git a/README.md b/README.md
new file mode 100644
index 0000000..cfb1773
--- /dev/null
+++ b/README.md
@@ -0,0 +1,167 @@
+# Justbuild on Nix
+
+This repository demonstrates a way to build locally with
+[justbuild](https://github.com/just-buildsystem/justbuild)
+using dependencies from [Nix](https://nixos.org/) in a clean and
+versioned way.
+
+## Background: Standard Paths on non-Nix Distributions
+
+On many non-Nix Linux distributions the local tools are installed to
+a small number of "standard" paths (like `/bin`, `/sbin`, `/usr/bin`,
+`/usr/sbin`). Moreover, tools like `sh` (used by default by the
+built-in `"generic"` rule) and `env` (the default action launcher)
+are configured to take those standard paths into account if the
+`PATH` environment variable is not set on their startup. Hence
+the tools installed in those standard paths are always available
+for local builds, without the need of setting `PATH` in action
+definitions.
+
+Such an approach obviously has the disadvantage that the cache has
+to be cleared, whenever the tools installed in those standard
+paths are updated. For users following a stable release with only
+security fixes, this can be acceptable.
+
+While not the idiomatic way of building on Nix, such a
+behavior can be simulated by using as launcher a tool
+like [withExtendedPath](./src/withExtendedPath/extend-path.cc)
+extending the value of the environment variable `PATH` by
+a given string, e.g., your `~/.nix-profile` controlled by
+the [home-manager](https://rycee.gitlab.io/home-manager/"). To
+avoid interfering with the clean Nix-idiomatic builds described
+in the following sections, it is recommended to also set the local
+build root to a different directory; remember to `just gc` twice
+whenever updating your `~/.nix-profile`. Your `~/.just-mrrc` thus
+would look something like
+```
+{ "local launcher":
+  [ "/home/YOURUSERNAME/.nix-profile/bin/withExtendedPath"
+  , "/home/YOURUSERNAME/.nix-profile/bin"
+  ]
+, "local build root": {"root": "home", "path": ".cache/just-nix-home"}
+}
+```
+Such an approach can be useful when dealing with a large number of
+project repositories all requiring basically the same tool chain.
+This discussion also shows how to easily provide a well-defined
+remote-execution environment on Nix: local launcher and local build
+root can also be set on the command line when starting `just execute`.
+
+## Getting Nix Paths into Justbuild Actions
+
+On Nix, the usual "standard" paths are pretty empty. Instead, all
+packages are installed into paths in the nix store containing a recursive
+hash of the full build description. So a path to the nix store brings
+a well-defined dependency and we're only left with the problem of
+getting the right paths into the actions.
+
+Justbuild deliberately ignores any environment variables of the invocation;
+the environment of an actions has to be provided through the build description.
+When using [`rules-cc`](https://github.com/just-buildsystem/rules-cc),
+the `"defaults"` targets can be used to set tools and paths for
+targets defined by a particular rule. We link those targets to
+the [`nixpkgs`](https://github.com/NixOS/nixpkgs) in a maintable
+way as follows.
+ - All the `"defaults"` targets simply take their values from appropriate
+   parts of the `"TOOLCHAIN_CONFIG"` just configuration variable.
+ - The just configuration file is generated by a
+   nix [derivation](./nix-dependencies/dependencies.nix) using that
+   nix derivations have easy access to the needed paths in the nix
+   store. That derivation also generates an rc-file pointing to
+   that configuration.
+ - The precise [sources](./nix/sources.json) of the `nixpkgs` are
+   pinned using [niv](https://github.com/nmattia/niv).
+ - A [nix shell](./shell.nix) uses the derivation at the pinned
+   snapshot of the `nixpkgs` and sets an alias for `just-mr` to
+   use the derived rc-file.
+
+So to build with the correct dependencies for the checked out version,
+simply start a `nix-shell` at the top level of this repository and
+use `just-mr build` as usual. It should be noted that the `nix-shell`
+does not pull in justbuild itself and instead inherits it from the
+invoking shell; in that way, when going back to an old snapshot
+it is still built using the currect justbuild. The reason is that,
+while newer versions of justbuild can work with a local build root
+generated by older versions, this is not necessarily the case the
+other way round (e.g., versions before `1.3.0` are not aware of
+the large-object CAS introduced in that release, and hence will
+not find certain artifacts referenced in the action cache). So,
+by using the current justbuild when checking out older snapshots,
+we can reconstruct the old actions without the need of cleaning up
+the local build root.
+
+## Shell commands
+
+The built-in rule `"generic"` allows to define a target by
+executing a shell command (via `sh -c ...`). This is convenient
+on systems where the shell is configured to have "the standard
+tools" available by default. On Nix, one would have to set
+the `"env"` appropriately on every invocation. As would be
+quite cumbersome, preference is given to the
+rule [`["shell", "cmds"]`](https://github.com/just-buildsystem/rules-cc?tab=readme-ov-file#rule-shell-cmds)
+that is a replacement for `"generic"` honoring the shell toolchain (as
+defined in the appropriate `"defaults"` target).
+
+
+## Repository Overview
+
+### Version pinning and updating
+
+There are two files pinning dependencies
+ - The `nixpkgs` are pinned in [nix/sources.json](./nix/sources.json). They can
+   be updated using [update-nix-dependencies.sh](./update-nix-dependencies.sh)
+   which simply calls `niv update` in our `nix-shell`.
+ - The dependencies on other other justbuild projects are
+   pinned in [etc/repos.json](etc/repos.json). As, at the
+   moment, there is only one external justbuild dependency,
+   the [rules-cc](https://github.com/just-buildsystem/rules-cc),
+   this is updated by hand. For larger
+   projects [just-import-git](https://github.com/just-buildsystem/justbuild/blob/master/share/man/just-import-git.1.md)
+   would be used to generate this file out of a description of the
+   local repositories and the dependencies to import.
+
+### Logical repositories
+
+This project uses several logical repositories.
+
+ - The directories `src` and `test` provide standard main and test
+   repositories. The logical structure is, as usual, that the test
+   repository has access to the main repository, but not the other
+   way round. In this way, the main repository can be imported
+   without pulling in the test dependencies. Note that in those
+   directories (where the main project-development work will happen)
+   there is nothing Nix-specific, except maybe for the choice to
+   refrain from using the built-in rule `"generic"`.
+
+ - The rules `rules/nix` and `rules/nix-test` are derived from
+   the [rules-cc](https://github.com/just-buildsystem/rules-cc) by
+   setting the target-file layer to the directory `etc/defaults`.
+   Here the toolchains are defined in a fine-granular way. The toolchains
+   of `rules/nix-test` inherit from the ones in `rules/nix`. In this
+   way, we can bring in additional tools only for tests. Those extra
+   paths are set in `TOOLCHAIN_CONFIG["test"]["PATH"]`.
+
+This fine-granular setting of the toolchains has the effect that actions only
+have the paths necessary set and in that way are not unnecessarily affected
+by changes of the depedencies. A good example for this granularity can be
+seen by building verbosely
+```
+[nix-shell]$  just-mr --main test build hello '' --log-limit 5
+```
+and looking at the differnt values of the environment: `PKG_CONFIG_PATH`
+is only set in the actions calling `pkg-config`, only the test
+action has the additional test tools in `PATH`.
+
+### Nix dependencies
+
+As explained, the nix-dependencies are declared in
+a [derivation](./nix-dependencies/dependencies.nix). For
+libraries, which probably form the majority of dependencies
+added, they way they become available is via `PKG_CONFIG_PATH`
+which makes them discoverable
+as [`["CC/pkgconfig", "system_library"]`](https://github.com/just-buildsystem/rules-cc?tab=readme-ov-file#rule-ccpkgconfig-system_library).
+As in the `buildPhase` of a Nix derivation, the environment variable
+`PKC_CONFIG_PATH` is already set appropriately, it is sufficient
+to simply add new libraries to `buildInputs`. As an example, note
+that `fmt` is available without being explicitly mentioned in
+`buildPhase`.