diff options
author | Klaus Aehlig <klaus.aehlig@huawei.com> | 2024-05-31 18:30:10 +0200 |
---|---|---|
committer | Klaus Aehlig <klaus.aehlig@huawei.com> | 2024-06-03 10:09:24 +0200 |
commit | 3626877554b6c567d43336ec49414cedfe487260 (patch) | |
tree | f80c51ec1867fea029b955d85e89e39017f920e3 | |
download | hello-nix-3626877554b6c567d43336ec49414cedfe487260.tar.gz |
Initial commit
34 files changed, 1140 insertions, 0 deletions
@@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/README.md b/README.md new file mode 100644 index 0000000..cfb1773 --- /dev/null +++ b/README.md @@ -0,0 +1,167 @@ +# Justbuild on Nix + +This repository demonstrates a way to build locally with +[justbuild](https://github.com/just-buildsystem/justbuild) +using dependencies from [Nix](https://nixos.org/) in a clean and +versioned way. + +## Background: Standard Paths on non-Nix Distributions + +On many non-Nix Linux distributions the local tools are installed to +a small number of "standard" paths (like `/bin`, `/sbin`, `/usr/bin`, +`/usr/sbin`). Moreover, tools like `sh` (used by default by the +built-in `"generic"` rule) and `env` (the default action launcher) +are configured to take those standard paths into account if the +`PATH` environment variable is not set on their startup. Hence +the tools installed in those standard paths are always available +for local builds, without the need of setting `PATH` in action +definitions. + +Such an approach obviously has the disadvantage that the cache has +to be cleared, whenever the tools installed in those standard +paths are updated. For users following a stable release with only +security fixes, this can be acceptable. + +While not the idiomatic way of building on Nix, such a +behavior can be simulated by using as launcher a tool +like [withExtendedPath](./src/withExtendedPath/extend-path.cc) +extending the value of the environment variable `PATH` by +a given string, e.g., your `~/.nix-profile` controlled by +the [home-manager](https://rycee.gitlab.io/home-manager/"). To +avoid interfering with the clean Nix-idiomatic builds described +in the following sections, it is recommended to also set the local +build root to a different directory; remember to `just gc` twice +whenever updating your `~/.nix-profile`. Your `~/.just-mrrc` thus +would look something like +``` +{ "local launcher": + [ "/home/YOURUSERNAME/.nix-profile/bin/withExtendedPath" + , "/home/YOURUSERNAME/.nix-profile/bin" + ] +, "local build root": {"root": "home", "path": ".cache/just-nix-home"} +} +``` +Such an approach can be useful when dealing with a large number of +project repositories all requiring basically the same tool chain. +This discussion also shows how to easily provide a well-defined +remote-execution environment on Nix: local launcher and local build +root can also be set on the command line when starting `just execute`. + +## Getting Nix Paths into Justbuild Actions + +On Nix, the usual "standard" paths are pretty empty. Instead, all +packages are installed into paths in the nix store containing a recursive +hash of the full build description. So a path to the nix store brings +a well-defined dependency and we're only left with the problem of +getting the right paths into the actions. + +Justbuild deliberately ignores any environment variables of the invocation; +the environment of an actions has to be provided through the build description. +When using [`rules-cc`](https://github.com/just-buildsystem/rules-cc), +the `"defaults"` targets can be used to set tools and paths for +targets defined by a particular rule. We link those targets to +the [`nixpkgs`](https://github.com/NixOS/nixpkgs) in a maintable +way as follows. + - All the `"defaults"` targets simply take their values from appropriate + parts of the `"TOOLCHAIN_CONFIG"` just configuration variable. + - The just configuration file is generated by a + nix [derivation](./nix-dependencies/dependencies.nix) using that + nix derivations have easy access to the needed paths in the nix + store. That derivation also generates an rc-file pointing to + that configuration. + - The precise [sources](./nix/sources.json) of the `nixpkgs` are + pinned using [niv](https://github.com/nmattia/niv). + - A [nix shell](./shell.nix) uses the derivation at the pinned + snapshot of the `nixpkgs` and sets an alias for `just-mr` to + use the derived rc-file. + +So to build with the correct dependencies for the checked out version, +simply start a `nix-shell` at the top level of this repository and +use `just-mr build` as usual. It should be noted that the `nix-shell` +does not pull in justbuild itself and instead inherits it from the +invoking shell; in that way, when going back to an old snapshot +it is still built using the currect justbuild. The reason is that, +while newer versions of justbuild can work with a local build root +generated by older versions, this is not necessarily the case the +other way round (e.g., versions before `1.3.0` are not aware of +the large-object CAS introduced in that release, and hence will +not find certain artifacts referenced in the action cache). So, +by using the current justbuild when checking out older snapshots, +we can reconstruct the old actions without the need of cleaning up +the local build root. + +## Shell commands + +The built-in rule `"generic"` allows to define a target by +executing a shell command (via `sh -c ...`). This is convenient +on systems where the shell is configured to have "the standard +tools" available by default. On Nix, one would have to set +the `"env"` appropriately on every invocation. As would be +quite cumbersome, preference is given to the +rule [`["shell", "cmds"]`](https://github.com/just-buildsystem/rules-cc?tab=readme-ov-file#rule-shell-cmds) +that is a replacement for `"generic"` honoring the shell toolchain (as +defined in the appropriate `"defaults"` target). + + +## Repository Overview + +### Version pinning and updating + +There are two files pinning dependencies + - The `nixpkgs` are pinned in [nix/sources.json](./nix/sources.json). They can + be updated using [update-nix-dependencies.sh](./update-nix-dependencies.sh) + which simply calls `niv update` in our `nix-shell`. + - The dependencies on other other justbuild projects are + pinned in [etc/repos.json](etc/repos.json). As, at the + moment, there is only one external justbuild dependency, + the [rules-cc](https://github.com/just-buildsystem/rules-cc), + this is updated by hand. For larger + projects [just-import-git](https://github.com/just-buildsystem/justbuild/blob/master/share/man/just-import-git.1.md) + would be used to generate this file out of a description of the + local repositories and the dependencies to import. + +### Logical repositories + +This project uses several logical repositories. + + - The directories `src` and `test` provide standard main and test + repositories. The logical structure is, as usual, that the test + repository has access to the main repository, but not the other + way round. In this way, the main repository can be imported + without pulling in the test dependencies. Note that in those + directories (where the main project-development work will happen) + there is nothing Nix-specific, except maybe for the choice to + refrain from using the built-in rule `"generic"`. + + - The rules `rules/nix` and `rules/nix-test` are derived from + the [rules-cc](https://github.com/just-buildsystem/rules-cc) by + setting the target-file layer to the directory `etc/defaults`. + Here the toolchains are defined in a fine-granular way. The toolchains + of `rules/nix-test` inherit from the ones in `rules/nix`. In this + way, we can bring in additional tools only for tests. Those extra + paths are set in `TOOLCHAIN_CONFIG["test"]["PATH"]`. + +This fine-granular setting of the toolchains has the effect that actions only +have the paths necessary set and in that way are not unnecessarily affected +by changes of the depedencies. A good example for this granularity can be +seen by building verbosely +``` +[nix-shell]$ just-mr --main test build hello '' --log-limit 5 +``` +and looking at the differnt values of the environment: `PKG_CONFIG_PATH` +is only set in the actions calling `pkg-config`, only the test +action has the additional test tools in `PATH`. + +### Nix dependencies + +As explained, the nix-dependencies are declared in +a [derivation](./nix-dependencies/dependencies.nix). For +libraries, which probably form the majority of dependencies +added, they way they become available is via `PKG_CONFIG_PATH` +which makes them discoverable +as [`["CC/pkgconfig", "system_library"]`](https://github.com/just-buildsystem/rules-cc?tab=readme-ov-file#rule-ccpkgconfig-system_library). +As in the `buildPhase` of a Nix derivation, the environment variable +`PKC_CONFIG_PATH` is already set appropriately, it is sufficient +to simply add new libraries to `buildInputs`. As an example, note +that `fmt` is available without being explicitly mentioned in +`buildPhase`. diff --git a/etc/defaults/CC/TARGETS b/etc/defaults/CC/TARGETS new file mode 100644 index 0000000..4ac5673 --- /dev/null +++ b/etc/defaults/CC/TARGETS @@ -0,0 +1,51 @@ +{ "defaults": + { "type": "configure" + , "target": "defaults (unconfigured)" + , "arguments_config": ["TOOLCHAIN_CONFIG"] + , "config": + { "type": "let*" + , "bindings": + [ [ "CC toolchain" + , { "type": "lookup" + , "map": + { "type": "var" + , "name": "TOOLCHAIN_CONFIG" + , "default": {"type": "empty_map"} + } + , "key": "CC" + , "default": {"type": "empty_map"} + } + ] + , [ "CC" + , { "type": "lookup" + , "map": {"type": "var", "name": "CC toolchain"} + , "key": "CC" + , "default": "cc" + } + ] + , [ "CXX" + , { "type": "lookup" + , "map": {"type": "var", "name": "CC toolchain"} + , "key": "CXX" + , "default": "c++" + } + ] + , [ "PATH" + , { "type": "lookup" + , "map": {"type": "var", "name": "CC toolchain"} + , "key": "PATH" + , "default": [] + } + ] + ] + , "body": {"type": "env", "vars": ["CC", "CXX", "PATH"]} + } + } +, "defaults (unconfigured)": + { "type": "defaults" + , "arguments_config": ["CC", "CXX", "PATH"] + , "CC": [{"type": "var", "name": "CC"}] + , "CXX": [{"type": "var", "name": "CXX"}] + , "PATH": {"type": "var", "name": "PATH"} + } +} diff --git a/etc/defaults/CC/pkgconfig/TARGETS b/etc/defaults/CC/pkgconfig/TARGETS new file mode 100644 index 0000000..63be252 --- /dev/null +++ b/etc/defaults/CC/pkgconfig/TARGETS @@ -0,0 +1,51 @@ +{ "defaults": + { "type": "configure" + , "target": "defaults (unconfigured)" + , "arguments_config": ["TOOLCHAIN_CONFIG"] + , "config": + { "type": "let*" + , "bindings": + [ [ "PKGCONFIG toolchain" + , { "type": "lookup" + , "map": + { "type": "var" + , "name": "TOOLCHAIN_CONFIG" + , "default": {"type": "empty_map"} + } + , "key": "PKGCONFIG" + , "default": {"type": "empty_map"} + } + ] + , [ "pkg-config" + , { "type": "lookup" + , "map": {"type": "var", "name": "PKGCONFIG toolchain"} + , "key": "pkg-config" + , "default": "pkg-config" + } + ] + , [ "PATH" + , { "type": "lookup" + , "map": {"type": "var", "name": "PKGCONFIG toolchain"} + , "key": "PATH" + , "default": [] + } + ] + , [ "PKG_CONFIG_PATH" + , { "type": "lookup" + , "map": {"type": "var", "name": "PKGCONFIG toolchain"} + , "key": "PKG_CONFIG_PATH" + , "default": [] + } + ] + ] + , "body": {"type": "env", "vars": ["pkg-config", "PATH", "PKG_CONFIG_PATH"]} + } + } +, "defaults (unconfigured)": + { "type": "defaults" + , "arguments_config": ["pkg-config", "PATH", "PKG_CONFIG_PATH"] + , "pkg-config": [{"type": "var", "name": "pkg-config"}] + , "PATH": {"type": "var", "name": "PATH"} + , "PKG_CONFIG_PATH": {"type": "var", "name": "PKG_CONFIG_PATH"} + } +} diff --git a/etc/defaults/CC/pkgconfig/test.TARGETS b/etc/defaults/CC/pkgconfig/test.TARGETS new file mode 100644 index 0000000..993fea3 --- /dev/null +++ b/etc/defaults/CC/pkgconfig/test.TARGETS @@ -0,0 +1,3 @@ +{ "defaults": + {"type": "defaults", "base": [["@", "base", "CC/pkgconfig", "defaults"]]} +} diff --git a/etc/defaults/CC/proto/TARGETS b/etc/defaults/CC/proto/TARGETS new file mode 100644 index 0000000..7f98293 --- /dev/null +++ b/etc/defaults/CC/proto/TARGETS @@ -0,0 +1,70 @@ +{ "defaults": + { "type": "configure" + , "target": "configure" + , "config": + {"type": "singleton_map", "key": "deps", "value": ["libprotobuf"]} + } +, "service defaults": + { "type": "configure" + , "target": "configure" + , "config": + { "type": "singleton_map" + , "key": "deps" + , "value": ["libgrpc++", "libprotobuf"] + } + } +, "configure": + { "type": "configure" + , "target": "defaults (unconfigured)" + , "arguments_config": ["TOOLCHAIN_CONFIG"] + , "config": + { "type": "let*" + , "bindings": + [ [ "proto toolchain" + , { "type": "lookup" + , "map": + { "type": "var" + , "name": "TOOLCHAIN_CONFIG" + , "default": {"type": "empty_map"} + } + , "key": "PROTO" + , "default": {"type": "empty_map"} + } + ] + , [ "PROTOC" + , { "type": "lookup" + , "map": {"type": "var", "name": "proto toolchain"} + , "key": "PROTOC" + , "default": "protoc" + } + ] + , [ "GRPC_PLUGIN" + , { "type": "lookup" + , "map": {"type": "var", "name": "proto toolchain"} + , "key": "GRPC_PLUGIN" + , "default": "grpc_cpp_plugin" + } + ] + , [ "PATH" + , { "type": "lookup" + , "map": {"type": "var", "name": "proto toolchain"} + , "key": "PATH" + , "default": [] + } + ] + ] + , "body": {"type": "env", "vars": ["PROTOC", "GRPC_PLUGIN", "PATH"]} + } + } +, "defaults (unconfigured)": + { "type": "defaults" + , "arguments_config": ["PROTOC", "GRPC_PLUGIN", "PATH", "deps"] + , "PROTOC": [{"type": "var", "name": "PROTOC"}] + , "GRPC_PLUGIN": [{"type": "var", "name": "GRPC_PLUGIN"}] + , "PATH": {"type": "var", "name": "PATH"} + , "deps": {"type": "var", "name": "deps"} + } +, "libprotobuf": + {"type": ["CC/pkgconfig", "system_library"], "name": ["protobuf"]} +, "libgrpc++": {"type": ["CC/pkgconfig", "system_library"], "name": ["grpc++"]} +} diff --git a/etc/defaults/CC/proto/test.TARGETS b/etc/defaults/CC/proto/test.TARGETS new file mode 100644 index 0000000..9f2fcae --- /dev/null +++ b/etc/defaults/CC/proto/test.TARGETS @@ -0,0 +1,5 @@ +{ "defaults": + {"type": "defaults", "base": [["@", "base", "CC/proto", "defaults"]]} +, "service defaults": + {"type": "defaults", "base": [["@", "base", "CC/proto", "defaults"]]} +} diff --git a/etc/defaults/CC/test.TARGETS b/etc/defaults/CC/test.TARGETS new file mode 100644 index 0000000..8a876f1 --- /dev/null +++ b/etc/defaults/CC/test.TARGETS @@ -0,0 +1 @@ +{"defaults": {"type": "defaults", "base": [["@", "base", "CC", "defaults"]]}} diff --git a/etc/defaults/shell/TARGETS b/etc/defaults/shell/TARGETS new file mode 100644 index 0000000..378611c --- /dev/null +++ b/etc/defaults/shell/TARGETS @@ -0,0 +1,43 @@ +{ "defaults": + { "type": "configure" + , "target": "defaults (unconfigured)" + , "arguments_config": ["TOOLCHAIN_CONFIG"] + , "config": + { "type": "let*" + , "bindings": + [ [ "shell toolchain" + , { "type": "lookup" + , "map": + { "type": "var" + , "name": "TOOLCHAIN_CONFIG" + , "default": {"type": "empty_map"} + } + , "key": "shell" + , "default": {"type": "empty_map"} + } + ] + , [ "sh" + , { "type": "lookup" + , "map": {"type": "var", "name": "shell toolchain"} + , "key": "sh" + , "default": "/bin/sh" + } + ] + , [ "PATH" + , { "type": "lookup" + , "map": {"type": "var", "name": "shell toolchain"} + , "key": "PATH" + , "default": [] + } + ] + ] + , "body": {"type": "env", "vars": ["sh", "PATH"]} + } + } +, "defaults (unconfigured)": + { "type": "defaults" + , "arguments_config": ["sh", "PATH"] + , "sh": [{"type": "var", "name": "sh"}] + , "PATH": {"type": "var", "name": "PATH"} + } +} diff --git a/etc/defaults/shell/test.TARGETS b/etc/defaults/shell/test.TARGETS new file mode 100644 index 0000000..7c099ff --- /dev/null +++ b/etc/defaults/shell/test.TARGETS @@ -0,0 +1,21 @@ +{ "defaults": + { "type": "defaults" + , "arguments_config": ["TOOLCHAIN_CONFIG"] + , "PATH": + { "type": "lookup" + , "key": "PATH" + , "default": [] + , "map": + { "type": "lookup" + , "key": "test" + , "default": {"type": "empty_map"} + , "map": + { "type": "var" + , "name": "TOOLCHAIN_CONFIG" + , "default": {"type": "empty_map"} + } + } + } + , "base": [["@", "base", "shell", "defaults"]] + } +} diff --git a/etc/defaults/shell/test/test.TARGETS b/etc/defaults/shell/test/test.TARGETS new file mode 100644 index 0000000..0967ef4 --- /dev/null +++ b/etc/defaults/shell/test/test.TARGETS @@ -0,0 +1 @@ +{} diff --git a/etc/repos.json b/etc/repos.json new file mode 100644 index 0000000..d1537db --- /dev/null +++ b/etc/repos.json @@ -0,0 +1,30 @@ +{ "repositories": + { "": + { "repository": {"type": "file", "path": "src"} + , "bindings": {"rules": "rules/nix"} + } + , "test": + { "repository": {"type": "file", "path": "test"} + , "bindings": {"rules": "rules/nix-test", "src": ""} + } + , "defaults": {"repository": {"type": "file", "path": "etc/defaults"}} + , "rules": + { "repository": + { "type": "git" + , "branch": "master" + , "commit": "0d436f26134d3fcaa695e4e6e87249d7fa381e44" + , "repository": "https://github.com/just-buildsystem/rules-cc.git" + , "subdir": "rules" + } + } + , "rules/nix": + {"repository": "rules", "target_root": "defaults", "rule_root": "rules"} + , "rules/nix-test": + { "repository": "rules" + , "target_root": "defaults" + , "rule_root": "rules" + , "target_file_name": "test.TARGETS" + , "bindings": {"base": "rules/nix"} + } + } +} diff --git a/nix-dependencies/default.nix b/nix-dependencies/default.nix new file mode 100644 index 0000000..9adde33 --- /dev/null +++ b/nix-dependencies/default.nix @@ -0,0 +1,2 @@ +{ nixpkgs ? import <nixpkgs> {} }: +nixpkgs.callPackage ./dependencies.nix {} diff --git a/nix-dependencies/dependencies.nix b/nix-dependencies/dependencies.nix new file mode 100644 index 0000000..99280cc --- /dev/null +++ b/nix-dependencies/dependencies.nix @@ -0,0 +1,58 @@ +{ stdenv +, jo +, pkg-config +, coreutils +, protobuf_25 +, grpc +, clang +, fmt + + # for tests +, gnugrep +, unixtools +}: + +stdenv.mkDerivation rec { + name = "nix-dependencies"; + + unpackPhase=''true''; + + nativeBuildInputs = [ + pkg-config + protobuf_25 + grpc + jo + ]; + + buildInputs = [ + clang + fmt + coreutils + gnugrep + unixtools.xxd + ]; + + buildPhase = '' + echo PKG_CONFIG_PATH=$PKG_CONFIG_PATH + jo TOOLCHAIN_CONFIG=$(jo \ + CC=$(jo PATH=$(jo -a ${clang}/bin ${coreutils}/bin)) \ + PROTO=$(jo PATH=$(jo -a ${protobuf_25}/bin ${grpc}/bin) \ + PROTOC=${protobuf_25}/bin/protoc \ + GRPC_PLUGIN=${grpc}/bin/grpc_cpp_plugin \ + ) \ + shell=$(jo PATH=$(jo -a ${coreutils}/bin)) \ + test=$(jo PATH=$(jo -a ${gnugrep}/bin ${unixtools.xxd}/bin)) \ + PKGCONFIG=$(jo pkg-config=${pkg-config}/bin/pkg-config \ + PKG_CONFIG_PATH=$(jo -a $PKG_CONFIG_PATH)) \ + ) > config.json + cat config.json + jo "just files"=$(jo config=$(jo -a $(jo root=system path=$out/share/config.json))) > rc.json + ''; + + installPhase = '' + mkdir -p $out/share + cp config.json $out/share + cp rc.json $out/share + ''; + +} diff --git a/nix/sources.json b/nix/sources.json new file mode 100644 index 0000000..8dfabc6 --- /dev/null +++ b/nix/sources.json @@ -0,0 +1,26 @@ +{ + "niv": { + "branch": "master", + "description": "Easy dependency management for Nix projects", + "homepage": "https://github.com/nmattia/niv", + "owner": "nmattia", + "repo": "niv", + "rev": "f7c538837892dd2eb83567c9f380a11efb59b53f", + "sha256": "0xl33k24vfc29cg9lnp95kvcq69qbq5fzb7jk9ig4lgrhaarh651", + "type": "tarball", + "url": "https://github.com/nmattia/niv/archive/f7c538837892dd2eb83567c9f380a11efb59b53f.tar.gz", + "url_template": "https://github.com/<owner>/<repo>/archive/<rev>.tar.gz" + }, + "nixpkgs-unstable": { + "branch": "nixpkgs-unstable", + "description": "Nix Packages collection", + "homepage": "", + "owner": "nixos", + "repo": "nixpkgs", + "rev": "6132b0f6e344ce2fe34fc051b72fb46e34f668e0", + "sha256": "0xh6fmkn2p8ivwqymb9gymfz07lw16fycyipgiydhxzgx6y9j7gd", + "type": "tarball", + "url": "https://github.com/nixos/nixpkgs/archive/6132b0f6e344ce2fe34fc051b72fb46e34f668e0.tar.gz", + "url_template": "https://github.com/<owner>/<repo>/archive/<rev>.tar.gz" + } +} diff --git a/nix/sources.nix b/nix/sources.nix new file mode 100644 index 0000000..9a01c8a --- /dev/null +++ b/nix/sources.nix @@ -0,0 +1,194 @@ +# This file has been generated by Niv. + +let + + # + # The fetchers. fetch_<type> fetches specs of type <type>. + # + + fetch_file = pkgs: name: spec: + let + name' = sanitizeName name + "-src"; + in + if spec.builtin or true then + builtins_fetchurl { inherit (spec) url sha256; name = name'; } + else + pkgs.fetchurl { inherit (spec) url sha256; name = name'; }; + + fetch_tarball = pkgs: name: spec: + let + name' = sanitizeName name + "-src"; + in + if spec.builtin or true then + builtins_fetchTarball { name = name'; inherit (spec) url sha256; } + else + pkgs.fetchzip { name = name'; inherit (spec) url sha256; }; + + fetch_git = name: spec: + let + ref = + if spec ? ref then spec.ref else + if spec ? branch then "refs/heads/${spec.branch}" else + if spec ? tag then "refs/tags/${spec.tag}" else + abort "In git source '${name}': Please specify `ref`, `tag` or `branch`!"; + submodules = if spec ? submodules then spec.submodules else false; + submoduleArg = + let + nixSupportsSubmodules = builtins.compareVersions builtins.nixVersion "2.4" >= 0; + emptyArgWithWarning = + if submodules == true + then + builtins.trace + ( + "The niv input \"${name}\" uses submodules " + + "but your nix's (${builtins.nixVersion}) builtins.fetchGit " + + "does not support them" + ) + {} + else {}; + in + if nixSupportsSubmodules + then { inherit submodules; } + else emptyArgWithWarning; + in + builtins.fetchGit + ({ url = spec.repo; inherit (spec) rev; inherit ref; } // submoduleArg); + + fetch_local = spec: spec.path; + + fetch_builtin-tarball = name: throw + ''[${name}] The niv type "builtin-tarball" is deprecated. You should instead use `builtin = true`. + $ niv modify ${name} -a type=tarball -a builtin=true''; + + fetch_builtin-url = name: throw + ''[${name}] The niv type "builtin-url" will soon be deprecated. You should instead use `builtin = true`. + $ niv modify ${name} -a type=file -a builtin=true''; + + # + # Various helpers + # + + # https://github.com/NixOS/nixpkgs/pull/83241/files#diff-c6f540a4f3bfa4b0e8b6bafd4cd54e8bR695 + sanitizeName = name: + ( + concatMapStrings (s: if builtins.isList s then "-" else s) + ( + builtins.split "[^[:alnum:]+._?=-]+" + ((x: builtins.elemAt (builtins.match "\\.*(.*)" x) 0) name) + ) + ); + + # The set of packages used when specs are fetched using non-builtins. + mkPkgs = sources: system: + let + sourcesNixpkgs = + import (builtins_fetchTarball { inherit (sources.nixpkgs) url sha256; }) { inherit system; }; + hasNixpkgsPath = builtins.any (x: x.prefix == "nixpkgs") builtins.nixPath; + hasThisAsNixpkgsPath = <nixpkgs> == ./.; + in + if builtins.hasAttr "nixpkgs" sources + then sourcesNixpkgs + else if hasNixpkgsPath && ! hasThisAsNixpkgsPath then + import <nixpkgs> {} + else + abort + '' + Please specify either <nixpkgs> (through -I or NIX_PATH=nixpkgs=...) or + add a package called "nixpkgs" to your sources.json. + ''; + + # The actual fetching function. + fetch = pkgs: name: spec: + + if ! builtins.hasAttr "type" spec then + abort "ERROR: niv spec ${name} does not have a 'type' attribute" + else if spec.type == "file" then fetch_file pkgs name spec + else if spec.type == "tarball" then fetch_tarball pkgs name spec + else if spec.type == "git" then fetch_git name spec + else if spec.type == "local" then fetch_local spec + else if spec.type == "builtin-tarball" then fetch_builtin-tarball name + else if spec.type == "builtin-url" then fetch_builtin-url name + else + abort "ERROR: niv spec ${name} has unknown type ${builtins.toJSON spec.type}"; + + # If the environment variable NIV_OVERRIDE_${name} is set, then use + # the path directly as opposed to the fetched source. + replace = name: drv: + let + saneName = stringAsChars (c: if isNull (builtins.match "[a-zA-Z0-9]" c) then "_" else c) name; + ersatz = builtins.getEnv "NIV_OVERRIDE_${saneName}"; + in + if ersatz == "" then drv else + # this turns the string into an actual Nix path (for both absolute and + # relative paths) + if builtins.substring 0 1 ersatz == "/" then /. + ersatz else /. + builtins.getEnv "PWD" + "/${ersatz}"; + + # Ports of functions for older nix versions + + # a Nix version of mapAttrs if the built-in doesn't exist + mapAttrs = builtins.mapAttrs or ( + f: set: with builtins; + listToAttrs (map (attr: { name = attr; value = f attr set.${attr}; }) (attrNames set)) + ); + + # https://github.com/NixOS/nixpkgs/blob/0258808f5744ca980b9a1f24fe0b1e6f0fecee9c/lib/lists.nix#L295 + range = first: last: if first > last then [] else builtins.genList (n: first + n) (last - first + 1); + + # https://github.com/NixOS/nixpkgs/blob/0258808f5744ca980b9a1f24fe0b1e6f0fecee9c/lib/strings.nix#L257 + stringToCharacters = s: map (p: builtins.substring p 1 s) (range 0 (builtins.stringLength s - 1)); + + # https://github.com/NixOS/nixpkgs/blob/0258808f5744ca980b9a1f24fe0b1e6f0fecee9c/lib/strings.nix#L269 + stringAsChars = f: s: concatStrings (map f (stringToCharacters s)); + concatMapStrings = f: list: concatStrings (map f list); + concatStrings = builtins.concatStringsSep ""; + + # https://github.com/NixOS/nixpkgs/blob/8a9f58a375c401b96da862d969f66429def1d118/lib/attrsets.nix#L331 + optionalAttrs = cond: as: if cond then as else {}; + + # fetchTarball version that is compatible between all the versions of Nix + builtins_fetchTarball = { url, name ? null, sha256 }@attrs: + let + inherit (builtins) lessThan nixVersion fetchTarball; + in + if lessThan nixVersion "1.12" then + fetchTarball ({ inherit url; } // (optionalAttrs (!isNull name) { inherit name; })) + else + fetchTarball attrs; + + # fetchurl version that is compatible between all the versions of Nix + builtins_fetchurl = { url, name ? null, sha256 }@attrs: + let + inherit (builtins) lessThan nixVersion fetchurl; + in + if lessThan nixVersion "1.12" then + fetchurl ({ inherit url; } // (optionalAttrs (!isNull name) { inherit name; })) + else + fetchurl attrs; + + # Create the final "sources" from the config + mkSources = config: + mapAttrs ( + name: spec: + if builtins.hasAttr "outPath" spec + then abort + "The values in sources.json should not have an 'outPath' attribute" + else + spec // { outPath = replace name (fetch config.pkgs name spec); } + ) config.sources; + + # The "config" used by the fetchers + mkConfig = + { sourcesFile ? if builtins.pathExists ./sources.json then ./sources.json else null + , sources ? if isNull sourcesFile then {} else builtins.fromJSON (builtins.readFile sourcesFile) + , system ? builtins.currentSystem + , pkgs ? mkPkgs sources system + }: rec { + # The sources, i.e. the attribute set of spec name to spec + inherit sources; + + # The "pkgs" (evaluated nixpkgs) to use for e.g. non-builtin fetchers + inherit pkgs; + }; + +in +mkSources (mkConfig {}) // { __functor = _: settings: mkSources (mkConfig settings); } diff --git a/shell.nix b/shell.nix new file mode 100644 index 0000000..b89488b --- /dev/null +++ b/shell.nix @@ -0,0 +1,23 @@ +let + + sources = import ./nix/sources.nix; + + nixpkgs = sources."nixpkgs-unstable"; + + pkgs = import nixpkgs {}; + + nix-dependencies = import ./nix-dependencies { nixpkgs = pkgs; } ; + +in pkgs.mkShell rec { + + name = "hello-nix"; + + buildInputs = with pkgs; [ + niv + ]; + + shellHook = '' + alias just-mr="just-mr --rc ${nix-dependencies}/share/rc.json" + ''; + +} diff --git a/src/hello/TARGETS b/src/hello/TARGETS new file mode 100644 index 0000000..80380c7 --- /dev/null +++ b/src/hello/TARGETS @@ -0,0 +1,9 @@ +{ "": + { "type": ["@", "rules", "CC", "binary"] + , "name": ["hello"] + , "srcs": ["hello.cpp"] + , "private-deps": ["fmt"] + } +, "fmt": + {"type": ["@", "rules", "CC/pkgconfig", "system_library"], "name": ["fmt"]} +} diff --git a/src/hello/hello.cpp b/src/hello/hello.cpp new file mode 100644 index 0000000..0c9df99 --- /dev/null +++ b/src/hello/hello.cpp @@ -0,0 +1,11 @@ +#include <iostream> + +#include <fmt/format.h> +#include <string> + +int main(int argc, char **argv) { + std::cout << fmt::format("Hello {}!", + argc > 1 ? std::string{argv[1]} : "World") + << std::endl; + return 0; +} diff --git a/src/proto/TARGETS b/src/proto/TARGETS new file mode 100644 index 0000000..512d7c2 --- /dev/null +++ b/src/proto/TARGETS @@ -0,0 +1,18 @@ +{ "example": + { "type": ["@", "rules", "proto", "library"] + , "name": ["example"] + , "srcs": ["example.proto"] + } +, "write": + { "type": ["@", "rules", "CC", "binary"] + , "name": ["write"] + , "srcs": ["write.cc"] + , "private-proto": ["example"] + } +, "read": + { "type": ["@", "rules", "CC", "binary"] + , "name": ["read"] + , "srcs": ["read.cc"] + , "private-proto": ["example"] + } +} diff --git a/src/proto/example.proto b/src/proto/example.proto new file mode 100644 index 0000000..69b5953 --- /dev/null +++ b/src/proto/example.proto @@ -0,0 +1,7 @@ +syntax = "proto3"; +package sample; + +message Example { + string foo = 1; + int32 bar = 2; +} diff --git a/src/proto/read.cc b/src/proto/read.cc new file mode 100644 index 0000000..f97a39a --- /dev/null +++ b/src/proto/read.cc @@ -0,0 +1,18 @@ +#include <fstream> +#include <iostream> + +#include "example.pb.h" + +int main(int argc, char **argv) { + sample::Example example; + + { + std::fstream input(argv[1], std::ios::in | std::ios::binary); + example.ParseFromIstream(&input); + } + + std::cout << "foo=" << example.foo() << "\n"; + std::cout << "bar=" << example.bar() << std::endl; + + return 0; +} diff --git a/src/proto/write.cc b/src/proto/write.cc new file mode 100644 index 0000000..2e206e5 --- /dev/null +++ b/src/proto/write.cc @@ -0,0 +1,17 @@ +#include <fstream> +#include <string> + +#include "example.pb.h" + +int main(int argc, char **argv) { + sample::Example example; + example.set_foo(std::string{argv[1]}); + example.set_bar(atoi(argv[2])); + { + std::fstream output(argv[3], + std::ios::out | std::ios::trunc | std::ios::binary); + example.SerializeToOstream(&output); + } + + return 0; +} diff --git a/src/shell/TARGETS b/src/shell/TARGETS new file mode 100644 index 0000000..f4b2dbf --- /dev/null +++ b/src/shell/TARGETS @@ -0,0 +1,13 @@ +{ "out": + { "type": ["@", "rules", "shell", "cmds"] + , "outs": ["out.txt"] + , "cmds": ["./hello > out.txt"] + , "deps": [["hello", ""]] + } +, "": + { "type": ["@", "rules", "shell", "cmds"] + , "outs": ["wc.txt", "tr.txt"] + , "cmds": ["wc out.txt > wc.txt", "cat out.txt | tr a-z A-Z > tr.txt"] + , "deps": ["out"] + } +} diff --git a/src/withExtendedPath/TARGETS b/src/withExtendedPath/TARGETS new file mode 100644 index 0000000..e226541 --- /dev/null +++ b/src/withExtendedPath/TARGETS @@ -0,0 +1,6 @@ +{ "": + { "type": ["@", "rules", "CC", "binary"] + , "name": ["withExtendedPath"] + , "srcs": ["extend-path.cc"] + } +} diff --git a/src/withExtendedPath/extend-path.cc b/src/withExtendedPath/extend-path.cc new file mode 100644 index 0000000..a466728 --- /dev/null +++ b/src/withExtendedPath/extend-path.cc @@ -0,0 +1,37 @@ +#include <algorithm> +#include <cstdlib> +#include <string> +#include <unistd.h> +#include <vector> + +int main(int argc, char* argv[]) { + constexpr int kForkFailed = 65; + constexpr int kPrerequisiteError = 97; + auto kEnv = std::string{"/usr/bin/env"}; + + if (argc < 3) { + return kPrerequisiteError; + } + + // compute PATH + auto path = std::string{argv[1]}; + auto const* env_path = std::getenv("PATH"); + if (env_path && (*env_path != '\0')) { + path = std::string{env_path} + std::string{":"} + path; + } + auto path_arg = std::string{"PATH="} + path; + + // create new argument vector + std::vector<char*> nargv{}; + nargv.reserve(argc+3); + nargv.push_back(kEnv.data()); + nargv.push_back(path_arg.data()); + for (int i=2; i < argc; i++) { + nargv.push_back(argv[i]); + } + nargv.push_back(nullptr); + + // exec + (void) execvp(nargv[0], static_cast<char* const*>(nargv.data())); + return kForkFailed; +} diff --git a/test/hello/TARGETS b/test/hello/TARGETS new file mode 100644 index 0000000..d01c382 --- /dev/null +++ b/test/hello/TARGETS @@ -0,0 +1,7 @@ +{ "": + { "type": ["@", "rules", "shell/test", "script"] + , "name": ["hello"] + , "test": ["test_hello.sh"] + , "deps": [["@", "src", "hello", ""]] + } +} diff --git a/test/hello/test_hello.sh b/test/hello/test_hello.sh new file mode 100644 index 0000000..08347e8 --- /dev/null +++ b/test/hello/test_hello.sh @@ -0,0 +1,5 @@ +set -e + +./hello | grep -i world +./hello Universe | grep Universe +./hello Universe | grep -i world && exit 1 || : diff --git a/test/proto/TARGETS b/test/proto/TARGETS new file mode 100644 index 0000000..50d5195 --- /dev/null +++ b/test/proto/TARGETS @@ -0,0 +1,7 @@ +{ "": + { "type": ["@", "rules", "shell/test", "script"] + , "name": ["proto"] + , "test": ["test.sh"] + , "deps": [["@", "src", "proto", "write"], ["@", "src", "proto", "read"]] + } +} diff --git a/test/proto/test.sh b/test/proto/test.sh new file mode 100644 index 0000000..fbd946b --- /dev/null +++ b/test/proto/test.sh @@ -0,0 +1,12 @@ +set -e + +./write FOO 42 data.bin 2>&1 + +echo +echo Binary encoding +xxd data.bin + +echo +echo Verifying read +./read data.bin | grep foo=FOO +./read data.bin | grep bar=42 diff --git a/test/test_hello.sh b/test/test_hello.sh new file mode 100644 index 0000000..08347e8 --- /dev/null +++ b/test/test_hello.sh @@ -0,0 +1,5 @@ +set -e + +./hello | grep -i world +./hello Universe | grep Universe +./hello Universe | grep -i world && exit 1 || : diff --git a/test/withExtendedPath/TARGETS b/test/withExtendedPath/TARGETS new file mode 100644 index 0000000..73a5a9e --- /dev/null +++ b/test/withExtendedPath/TARGETS @@ -0,0 +1,7 @@ +{"withExtendedPath": + { "type": ["@", "rules", "shell/test", "script"] + , "name": ["find-in-extended-path"] + , "test": ["test.sh"] + , "deps": [["@", "src", "withExtendedPath", ""]] + } +} diff --git a/test/withExtendedPath/test.sh b/test/withExtendedPath/test.sh new file mode 100644 index 0000000..de51742 --- /dev/null +++ b/test/withExtendedPath/test.sh @@ -0,0 +1,12 @@ +set -e + +TOOLS_DIR=`pwd`/additionalTools +mkdir -p ${TOOLS_DIR} +cat > ${TOOLS_DIR}/mytool <<'EOF' +#!/bin/sh + +echo using my CuStOm tool +EOF +chmod 755 ${TOOLS_DIR}/mytool + +./withExtendedPath ${TOOLS_DIR} mytool | grep CuStOm diff --git a/update-nix-dependencies.sh b/update-nix-dependencies.sh new file mode 100755 index 0000000..d4953a8 --- /dev/null +++ b/update-nix-dependencies.sh @@ -0,0 +1 @@ +nix-shell --run "niv update" |