From 3626877554b6c567d43336ec49414cedfe487260 Mon Sep 17 00:00:00 2001 From: Klaus Aehlig Date: Fri, 31 May 2024 18:30:10 +0200 Subject: Initial commit --- README.md | 167 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 167 insertions(+) create mode 100644 README.md (limited to 'README.md') diff --git a/README.md b/README.md new file mode 100644 index 0000000..cfb1773 --- /dev/null +++ b/README.md @@ -0,0 +1,167 @@ +# Justbuild on Nix + +This repository demonstrates a way to build locally with +[justbuild](https://github.com/just-buildsystem/justbuild) +using dependencies from [Nix](https://nixos.org/) in a clean and +versioned way. + +## Background: Standard Paths on non-Nix Distributions + +On many non-Nix Linux distributions the local tools are installed to +a small number of "standard" paths (like `/bin`, `/sbin`, `/usr/bin`, +`/usr/sbin`). Moreover, tools like `sh` (used by default by the +built-in `"generic"` rule) and `env` (the default action launcher) +are configured to take those standard paths into account if the +`PATH` environment variable is not set on their startup. Hence +the tools installed in those standard paths are always available +for local builds, without the need of setting `PATH` in action +definitions. + +Such an approach obviously has the disadvantage that the cache has +to be cleared, whenever the tools installed in those standard +paths are updated. For users following a stable release with only +security fixes, this can be acceptable. + +While not the idiomatic way of building on Nix, such a +behavior can be simulated by using as launcher a tool +like [withExtendedPath](./src/withExtendedPath/extend-path.cc) +extending the value of the environment variable `PATH` by +a given string, e.g., your `~/.nix-profile` controlled by +the [home-manager](https://rycee.gitlab.io/home-manager/"). To +avoid interfering with the clean Nix-idiomatic builds described +in the following sections, it is recommended to also set the local +build root to a different directory; remember to `just gc` twice +whenever updating your `~/.nix-profile`. Your `~/.just-mrrc` thus +would look something like +``` +{ "local launcher": + [ "/home/YOURUSERNAME/.nix-profile/bin/withExtendedPath" + , "/home/YOURUSERNAME/.nix-profile/bin" + ] +, "local build root": {"root": "home", "path": ".cache/just-nix-home"} +} +``` +Such an approach can be useful when dealing with a large number of +project repositories all requiring basically the same tool chain. +This discussion also shows how to easily provide a well-defined +remote-execution environment on Nix: local launcher and local build +root can also be set on the command line when starting `just execute`. + +## Getting Nix Paths into Justbuild Actions + +On Nix, the usual "standard" paths are pretty empty. Instead, all +packages are installed into paths in the nix store containing a recursive +hash of the full build description. So a path to the nix store brings +a well-defined dependency and we're only left with the problem of +getting the right paths into the actions. + +Justbuild deliberately ignores any environment variables of the invocation; +the environment of an actions has to be provided through the build description. +When using [`rules-cc`](https://github.com/just-buildsystem/rules-cc), +the `"defaults"` targets can be used to set tools and paths for +targets defined by a particular rule. We link those targets to +the [`nixpkgs`](https://github.com/NixOS/nixpkgs) in a maintable +way as follows. + - All the `"defaults"` targets simply take their values from appropriate + parts of the `"TOOLCHAIN_CONFIG"` just configuration variable. + - The just configuration file is generated by a + nix [derivation](./nix-dependencies/dependencies.nix) using that + nix derivations have easy access to the needed paths in the nix + store. That derivation also generates an rc-file pointing to + that configuration. + - The precise [sources](./nix/sources.json) of the `nixpkgs` are + pinned using [niv](https://github.com/nmattia/niv). + - A [nix shell](./shell.nix) uses the derivation at the pinned + snapshot of the `nixpkgs` and sets an alias for `just-mr` to + use the derived rc-file. + +So to build with the correct dependencies for the checked out version, +simply start a `nix-shell` at the top level of this repository and +use `just-mr build` as usual. It should be noted that the `nix-shell` +does not pull in justbuild itself and instead inherits it from the +invoking shell; in that way, when going back to an old snapshot +it is still built using the currect justbuild. The reason is that, +while newer versions of justbuild can work with a local build root +generated by older versions, this is not necessarily the case the +other way round (e.g., versions before `1.3.0` are not aware of +the large-object CAS introduced in that release, and hence will +not find certain artifacts referenced in the action cache). So, +by using the current justbuild when checking out older snapshots, +we can reconstruct the old actions without the need of cleaning up +the local build root. + +## Shell commands + +The built-in rule `"generic"` allows to define a target by +executing a shell command (via `sh -c ...`). This is convenient +on systems where the shell is configured to have "the standard +tools" available by default. On Nix, one would have to set +the `"env"` appropriately on every invocation. As would be +quite cumbersome, preference is given to the +rule [`["shell", "cmds"]`](https://github.com/just-buildsystem/rules-cc?tab=readme-ov-file#rule-shell-cmds) +that is a replacement for `"generic"` honoring the shell toolchain (as +defined in the appropriate `"defaults"` target). + + +## Repository Overview + +### Version pinning and updating + +There are two files pinning dependencies + - The `nixpkgs` are pinned in [nix/sources.json](./nix/sources.json). They can + be updated using [update-nix-dependencies.sh](./update-nix-dependencies.sh) + which simply calls `niv update` in our `nix-shell`. + - The dependencies on other other justbuild projects are + pinned in [etc/repos.json](etc/repos.json). As, at the + moment, there is only one external justbuild dependency, + the [rules-cc](https://github.com/just-buildsystem/rules-cc), + this is updated by hand. For larger + projects [just-import-git](https://github.com/just-buildsystem/justbuild/blob/master/share/man/just-import-git.1.md) + would be used to generate this file out of a description of the + local repositories and the dependencies to import. + +### Logical repositories + +This project uses several logical repositories. + + - The directories `src` and `test` provide standard main and test + repositories. The logical structure is, as usual, that the test + repository has access to the main repository, but not the other + way round. In this way, the main repository can be imported + without pulling in the test dependencies. Note that in those + directories (where the main project-development work will happen) + there is nothing Nix-specific, except maybe for the choice to + refrain from using the built-in rule `"generic"`. + + - The rules `rules/nix` and `rules/nix-test` are derived from + the [rules-cc](https://github.com/just-buildsystem/rules-cc) by + setting the target-file layer to the directory `etc/defaults`. + Here the toolchains are defined in a fine-granular way. The toolchains + of `rules/nix-test` inherit from the ones in `rules/nix`. In this + way, we can bring in additional tools only for tests. Those extra + paths are set in `TOOLCHAIN_CONFIG["test"]["PATH"]`. + +This fine-granular setting of the toolchains has the effect that actions only +have the paths necessary set and in that way are not unnecessarily affected +by changes of the depedencies. A good example for this granularity can be +seen by building verbosely +``` +[nix-shell]$ just-mr --main test build hello '' --log-limit 5 +``` +and looking at the differnt values of the environment: `PKG_CONFIG_PATH` +is only set in the actions calling `pkg-config`, only the test +action has the additional test tools in `PATH`. + +### Nix dependencies + +As explained, the nix-dependencies are declared in +a [derivation](./nix-dependencies/dependencies.nix). For +libraries, which probably form the majority of dependencies +added, they way they become available is via `PKG_CONFIG_PATH` +which makes them discoverable +as [`["CC/pkgconfig", "system_library"]`](https://github.com/just-buildsystem/rules-cc?tab=readme-ov-file#rule-ccpkgconfig-system_library). +As in the `buildPhase` of a Nix derivation, the environment variable +`PKC_CONFIG_PATH` is already set appropriately, it is sufficient +to simply add new libraries to `buildInputs`. As an example, note +that `fmt` is available without being explicitly mentioned in +`buildPhase`. -- cgit v1.2.3