diff options
| -rw-r--r-- | CHANGELOG.md | 3 | ||||
| -rw-r--r-- | doc/future-designs/extended-git-tree-variables.md | 78 | ||||
| -rw-r--r-- | share/man/just-mr-repository-config.5.md | 24 | ||||
| -rw-r--r-- | test/end-to-end/just-mr/TARGETS | 8 | ||||
| -rwxr-xr-x | test/end-to-end/just-mr/git-tree-env.sh | 86 |
5 files changed, 119 insertions, 80 deletions
diff --git a/CHANGELOG.md b/CHANGELOG.md index ba7b9a1d..8559f2ca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,9 @@ A feature release on top of `1.2.0`, backwards compatible. allowing to specify the invocation of the shell (defaulting to `["sh", "-c"]`). - `just describe` also shows the values of the implicit dependencies. +- When `just-mr` executes the action to generate the desired tree of a + `"git tree"` repository, it can be specified that certain variables + of the environment can be inherited. ### Fixes diff --git a/doc/future-designs/extended-git-tree-variables.md b/doc/future-designs/extended-git-tree-variables.md deleted file mode 100644 index 05c1feb2..00000000 --- a/doc/future-designs/extended-git-tree-variables.md +++ /dev/null @@ -1,78 +0,0 @@ -# Extended Variables in Actions for `"git tree"` roots - -## Background - -Justbuild is tightly coupled to `git`: it uses the same identifiers -as `git` to refer to files and directories. In this way, cache keys -for actions where the inputs come for `git` trees can be computed -without inspecting the respective files or directories. Due to -this tight coupling, `git` repositories get preferential treatment, -e.g., fetching them is built into `just-mr`. - -Of course, adding support for every version-control system that might -be relevant does not scale. Instead, `"git tree"` repositories are -supported as a generic way to support all (other) version-control -systems; they are given by a `git` tree identifier and a command -that, when executed, produces this tree somewhere in the current -working directory—typically by checking out a specific version -using a different version-control system. - -## Current Limitation - -Calls to other version-control systems often require certain -environment variables to be set. E.g., for `ssh`-based authentication -typically one would set `SSH_AUTH_SOCK`. If proxy settings should -be honored, those would also be passed via appropriate environment -variables. All these environment variables have in common that -they need different values for different users; in particular, they -cannot be set in to fixed values in the action description. - -## Proposed Change - -As for `"git tree"` repositories the result of the fetch action is -determined ahead of time, it is not necessary to isolate the fetch -actions in the same way as we do with the build actions; if the -non-isolation misguides the fetch action, the verification of the -fetched tree will notice this and an error will be reported instead -of obtaining wrong results. Therefore, the following proposed -extension is safe. - -The description of a `"git tree"` repository already contains -an optional field `"env"` (defaulting to the empty map) with a -string-string map specifying the environment in which the fetch -command is to be executed. We propose to extend the definition by an -additional optional field `"inherit env"`, that, if given contains -a list of strings thought of as names of environment variables. -For each of them, if they are set in the ambient environment in -which `just-mr` is invoked, they are passed with the value in the -ambient environment to the environment of the fetch command (taking -precedence over a possible fixed value set in the `"env"` map); -those not set in the ambient environment are ignored, leaving a -possible value set in `"env"` untouched. - -## Approach not to be taken: Reference to the main workspace - -It might seem tempting to allow a variable pointing to the -main workspace, arguing that in this way more complex fetch -logic could be achieved (by having a command like `["sh", "-c", -"${WORKSPACE}/bin/fetch.py example.com/foorepo"]`). - -Such an approach is, however, problematic, as the definition is -no longer self contained but depends on the "main" repository, and -therefore does not compose if used as a transitive dependency, e.g., -via `just-import-git`. Here it is important to note that this not -due to a limitation of the import tool, but due to the fact that -two repositories might refer to a different "main" repository in -their definition. - -For cases where a more complex fetch logic is needed, we note that -the `repos.json` can be, and often is, a generated file anyway, -e.g., for the reasons described the `just-deduplicate-repos` man -page. For generated files it is not problematic if they contain -duplicate code. Therefore, inlining the logic into the individual -commands is acceptable can can be done by the entity (typically a -script) generating the actual `repos.json` from the descriptions -of which repositories to follow at which branches. - -If complex logic is needed to compute parts of the roots, like target -files, this is better accommodated by the "computed roots" proposal. diff --git a/share/man/just-mr-repository-config.5.md b/share/man/just-mr-repository-config.5.md index 82861f69..b14023a8 100644 --- a/share/man/just-mr-repository-config.5.md +++ b/share/man/just-mr-repository-config.5.md @@ -92,8 +92,25 @@ The following fields are supported: ### *`"git tree"`* -It defines as workspace root a known Git tree obtainable by a generic -command. +It defines as workspace root as a fixed `git` tree, given by the +corresponding tree identifier. If that tree is not known already to +`just-mr`, a specified command will be executed in a fresh directory +that is expected to produce the given tree somewhere below the +working directory. + +This type of root is the way builds against sources versioned in +arbitrary version-control systems can be carried out. The command +then would be a call to the version control system requesting an +export at a particular version. + +As the root is already fully determined by the specified `git` tree +identifier, the corresponding action need not be fully isolated. +In fact, to check out a repository it might be necessary to provide +credentials (which do not matter for the checked-out tree, but +differ from user to user). To support this, the description of the +root can specify environment variables to inherit from the ambient +environment. E.g., `SSH_AUTH_SOCK` can be specified here to support +`ssh`-based authentication. The following fields are supported: @@ -108,6 +125,9 @@ The following fields are supported: - *`"env"`* provides a map of envariables to be set for executing the command. + - *`"inherit env"`* provides a list of variables to be inherited from the + environment `just-mr` is called within, if set there. + ### *`"distdir"`* It defines as workspace root a directory with the distribution archives diff --git a/test/end-to-end/just-mr/TARGETS b/test/end-to-end/just-mr/TARGETS index 0ba35cee..101f9ec9 100644 --- a/test/end-to-end/just-mr/TARGETS +++ b/test/end-to-end/just-mr/TARGETS @@ -136,6 +136,13 @@ , "tar cf src/data.tar src/*" ] } +, "git-tree-env": + { "type": ["@", "rules", "shell/test", "script"] + , "name": ["git-tree-git-tree-env"] + , "test": ["git-tree-env.sh"] + , "deps": + [["end-to-end", "mr-tool-under-test"], ["end-to-end", "tool-under-test"]] + } , "TESTS": { "type": "install" , "tainted": ["test"] @@ -153,6 +160,7 @@ [ [ "install-roots-symlinks" , "just_mr_mp" , "git-tree-verbosity" + , "git-tree-env" , "defaults" , "absent-roots" , "fetch-absent" diff --git a/test/end-to-end/just-mr/git-tree-env.sh b/test/end-to-end/just-mr/git-tree-env.sh new file mode 100755 index 00000000..d93ac454 --- /dev/null +++ b/test/end-to-end/just-mr/git-tree-env.sh @@ -0,0 +1,86 @@ +#!/bin/sh +# Copyright 2023 Huawei Cloud Computing Technology Co., Ltd. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -eu + +JUST_MR="$(pwd)/bin/mr-tool-under-test" +JUST="$(pwd)/bin/tool-under-test" +LBR="${TEST_TMPDIR}/local-build-root" +LBR2="${TEST_TMPDIR}/new-local-build-root" +OUT="${TEST_TMPDIR}/out" + +mkdir -p bin +cat > bin/mock-vcs <<'EOF' +#!/bin/sh +if [ "$(cat ${CREDENTIAL_PATH:-/dev/null})" = "sEcReT" ] +then + echo 'checked-out sources' > sources.txt + echo '{}' > TARGETS +else + echo 'not enough credentials available' +fi +EOF +chmod 755 bin/mock-vcs +export PATH="$(pwd)/bin:${PATH}" + +mkdir -p etc +echo -n sEcReT > etc/pass +export CREDENTIAL_PATH="$(pwd)/etc/pass" + +# Compute tree of our mock checkout +mkdir work +( cd work + mock-vcs 2>&1 + git init 2>&1 + git config user.email "nobody@example.org" 2>&1 + git config user.name "Nobody" 2>&1 + git add . 2>&1 + git commit -m "Sample output" 2>&1 +) +readonly TREE_ID=$(cd work && git log -n 1 --format="%T") +rm -rf work +echo "Tree of checkout is ${TREE_ID}." + +# Setup repo config +mkdir repo +cd repo +touch ROOT +cat > repos.json <<EOF +{ "repositories": + { "": + { "repository": + { "type": "git tree" + , "id": "${TREE_ID}" + , "cmd": ["mock-vcs"] + , "inherit env": ["PATH", "CREDENTIAL_PATH"] + } + } + } +} +EOF +cat repos.json + +# Succesfull build + +"${JUST_MR}" --norc --just "${JUST}" --local-build-root "${LBR}" \ + install -o "${OUT}" '' sources.txt 2>&1 +grep checked-out "${OUT}/sources.txt" + +# Verify the environment is needed +export CREDENTIALS_PATH=/dev/null +"${JUST_MR}" --norc --just "${JUST}" --local-build-root "${LBR2}" \ + build 2>&1 && exit 1 || : + +echo DONE |