summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
Diffstat
-rw-r--r--CHANGELOG.md2
-rw-r--r--share/man/just-mr.1.md10
-rw-r--r--share/man/just.1.md40
-rw-r--r--src/buildtool/common/cli.hpp28
-rw-r--r--src/buildtool/main/TARGETS1
-rw-r--r--src/buildtool/main/cli.cpp20
-rw-r--r--src/buildtool/main/cli.hpp2
-rw-r--r--src/buildtool/main/main.cpp6
-rw-r--r--src/other_tools/just_mr/utils.hpp9
9 files changed, 103 insertions, 15 deletions
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3b292c66..b1cfdc2d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -20,6 +20,8 @@ A feature release on top of `1.2.0`, backwards compatible.
- New script `just-deduplicate-repos` to avoid blow up of the
`repos.json` in the case of chained imports with common dependencies.
+- New subcommand `add-to-cas` to add files and directories to the local
+ CAS and optionally also copy them to the remote-execution endpoint.
- The built-in `"generic"` rule now supports an argument `"sh -c"`,
allowing to specify the invocation of the shell (defaulting to
`["sh", "-c"]`).
diff --git a/share/man/just-mr.1.md b/share/man/just-mr.1.md
index 7a92466d..7fa02076 100644
--- a/share/man/just-mr.1.md
+++ b/share/man/just-mr.1.md
@@ -13,7 +13,7 @@ SYNOPSIS
**`just-mr`** \[*`OPTION`*\]... **`fetch`** \[**`--all`**\] \[**`--backup-to-remote`**] \[**`-o`** *`fetch-dir`*\] \[*`main-repo`*\]
**`just-mr`** \[*`OPTION`*\]... **`update`** \[*`repo`*\]...
**`just-mr`** \[*`OPTION`*\]... **`do`** \[*`JUST_ARG`*\]...
-**`just-mr`** \[*`OPTION`*\]... {**`version`**|**`describe`**|**`analyse`**|**`build`**|**`install`**|**`install-cas`**|**`rebuild`**|**`gc`**} \[*`JUST_ARG`*\]...
+**`just-mr`** \[*`OPTION`*\]... {**`version`**|**`describe`**|**`analyse`**|**`build`**|**`install`**|**`install-cas`**|**`add-to-cas`**|**`rebuild`**|**`gc`**} \[*`JUST_ARG`*\]...
DESCRIPTION
===========
@@ -277,7 +277,7 @@ This subcommand is used as the canonical way of specifying just
arguments and calling **`just`** via **`execvp`**(2). Any subsequent argument
is unconditionally forwarded to **`just`**. For *known* subcommands
(**`version`**, **`describe`**, **`analyse`**, **`build`**, **`install`**,
-**`install-cas`**, **`rebuild`**, **`gc`**), the
+**`install-cas`**, **`add-to-cas`**, **`rebuild`**, **`gc`**), the
**`just-mr setup`** step is performed first for those commands accepting a
configuration and the produced configuration is prefixed to the provided
arguments. The main repository for the **`setup`** step can be provided in
@@ -297,14 +297,14 @@ rebuild).
The **`--remote-execution-address`**, **`--compatible`**, and
**`--remote-serve-address`** arguments are passed to **`just`** as early
arguments for those *known* subcommands that accept them
-(analyse, build, install-cas, install, rebuild, traverse).
+(analyse, build, install-cas, add-to-cas, install, rebuild, traverse).
The *authentication options* given to **`just-mr`** are passed to **`just`** as
early arguments for those *known* subcommands that accept them, according to
**`just`**(1).
-**`version`**|**`describe`**|**`analyse`**|**`build`**|**`install`**|**`install-cas`**|**`rebuild`**|**`gc`**
--------------------------------------------------------------------------------------------------------------
+**`version`**|**`describe`**|**`analyse`**|**`build`**|**`install`**|**`install-cas`**|**`add-to-cas`**|**`rebuild`**|**`gc`**
+------------------------------------------------------------------------------------------------------------------------------
This subcommand is the explicit way of specifying *known* just
subcommands and calling **`just`** via **`execvp`**(2). The same description
diff --git a/share/man/just.1.md b/share/man/just.1.md
index 8f3e9436..de84f44b 100644
--- a/share/man/just.1.md
+++ b/share/man/just.1.md
@@ -12,6 +12,7 @@ SYNOPSIS
**`just`** {**`analyse`**|**`build`**} \[*`OPTION`*\]... \[\[*`module`*\] *`target`*\]
**`just`** **`install`** \[*`OPTION`*\]... **`-o`** *`OUTPUT_DIR`* \[\[*`module`*\] *`target`*\]
**`just`** **`install-cas`** \[*`OPTION`*\]... *`OBJECT_ID`*
+**`just`** **`add-to-cas`** \[*`OPTION`*\]... *`PATH`*
**`just`** **`describe`** \[*`OPTION`*\]... \[\[*`module`*\] *`target`*\]
**`just`** **`rebuild`** \[*`OPTION`*\]... \[\[*`module`*\] *`target`*\]
**`just`** **`traverse`** \[*`OPTION`*\]... **`-o`** *`OUTPUT_DIR`* **`-g`** *`GRAPH_FILE`*
@@ -248,6 +249,17 @@ description.
If the artifact is a file, it will replace the existing file. If the
artifact is a tree, it will cause an error.
+**`add-to-cas`**
+----------------
+
+**`add-to-cas`** adds a file or directory to the local CAS and
+reports the hash (without size or type information) on stdout. If a
+remote endpoint is given, the object is also uploaded there. A main
+use case of this command is to simplify the setup of `"git tree"`
+repositories, where it can also avoid checking out a repository of
+a foreign version-control system twice.
+
+
**`traverse`**
--------------
@@ -300,7 +312,7 @@ At increased computational effort, be compatible with the original
remote build execution protocol. As the change affects identifiers, the
flag must be used consistently for all related invocations.
Supported by:
-analyse|build|describe|install-cas|install|rebuild|traverse|execute.
+add-to-cas|analyse|build|describe|install-cas|install|rebuild|traverse|execute.
Build configuration options
---------------------------
@@ -365,7 +377,7 @@ Supported by: build|install|rebuild|traverse|execute.
**`--local-build-root`** *`PATH`*
Root for local CAS, cache, and build directories. The path will be
created if it does not exist already.
-Supported by: build|describe|install-cas|install|rebuild|traverse|gc|execute.
+Supported by: add-to-cas|build|describe|install-cas|install|rebuild|traverse|gc|execute.
**`--main`** *`NAME`*
The repository to take the target from.
@@ -421,22 +433,22 @@ Path to local log file. **`just`** will store the information printed on
stderr in the log file along with the thread id and timestamp when the
output has been generated.
Supported by:
-analyse|build|describe|install|install-cas|rebuild|traverse|gc|execute.
+add-to-cas|analyse|build|describe|install|install-cas|rebuild|traverse|gc|execute.
**`--log-limit`** *`NUM`*
Log limit (higher is more verbose) in interval \[0,6\] (Default: 3).
Supported by:
-analyse|build|describe|install|install-cas|rebuild|traverse|gc|execute.
+add-to-cas|analyse|build|describe|install|install-cas|rebuild|traverse|gc|execute.
**`--plain-log`**
Do not use ANSI escape sequences to highlight messages.
Supported by:
-analyse|build|describe|install|install-cas|rebuild|traverse|gc|execute.
+add-to-cas|analyse|build|describe|install|install-cas|rebuild|traverse|gc|execute.
**`--log-append`**
Append messages to log file instead of overwriting existing.
Supported by:
-analyse|build|describe|install|install-cas|rebuild|traverse|gc|execute.
+add-to-cas|analyse|build|describe|install|install-cas|rebuild|traverse|gc|execute.
**`--expression-log-limit`** *`NUM`*
In error messages, truncate the entries in the enumeration of the active
@@ -526,7 +538,7 @@ Supported by: analyse|build|install|rebuild|traverse.
**`-r`**, **`--remote-execution-address`** *`NAME`*:*`PORT`*
Address of the remote execution service.
-Supported by: analyse|build|describe|install-cas|install|rebuild|traverse.
+Supported by: add-to-cas|analyse|build|describe|install-cas|install|rebuild|traverse.
**`--endpoint-configuration`** FILE
File containing a description on how to dispatch to different
@@ -573,17 +585,17 @@ Only TLS and mutual TLS (mTLS) are supported.
**`--tls-ca-cert`** *`PATH`*
Path to a TLS CA certificate that is trusted to sign the server
certificate.
-Supported by: analyse|build|describe|install-cas|install|rebuild|traverse|execute.
+Supported by: add-to-cas|analyse|build|describe|install-cas|install|rebuild|traverse|execute.
**`--tls-client-cert`** *`PATH`*
Path to a TLS client certificate to enable mTLS. It must be passed in
conjunction with **`--tls-client-key`** and **`--tls-ca-cert`**.
-Supported by: analyse|build|describe|install-cas|install|rebuild|traverse.
+Supported by: add-to-cas|analyse|build|describe|install-cas|install|rebuild|traverse.
**`--tls-client-key`** *`PATH`*
Path to a TLS client key to enable mTLS. It must be passed in
conjunction with **`--tls-client-cert`** and **`--tls-ca-cert`**.
-Supported by: analyse|build|describe|install-cas|install|rebuild|traverse.
+Supported by: add-to-cas|analyse|build|describe|install-cas|install|rebuild|traverse.
**`analyse`** specific options
------------------------------
@@ -684,6 +696,14 @@ Cache endpoint to compare against (use *`"local"`* for local cache).
**`--dump-flaky`** *`PATH`*
Dump flaky actions to file.
+**`add-to-cas`** specific options
+---------------------------------
+
+**`--follow-symlinks`**
+Resolve the positional argument to not be a symbolic link by following
+symbolic links. The default is to add the link itself, i.e., the string
+obtained by **`readlink`**(2), as blob.
+
**`traverse`** specific options
-------------------------------
diff --git a/src/buildtool/common/cli.hpp b/src/buildtool/common/cli.hpp
index 7b4f3987..6b6be63a 100644
--- a/src/buildtool/common/cli.hpp
+++ b/src/buildtool/common/cli.hpp
@@ -607,6 +607,34 @@ static inline auto SetupFetchArguments(
"--remember", clargs->remember, "Copy object to local CAS first");
}
+static inline auto SetupToAddArguments(
+ gsl::not_null<CLI::App*> const& app,
+ gsl::not_null<ToAddArguments*> const& clargs) {
+ app->add_option_function<std::string>(
+ "location",
+ [clargs](auto const& path_raw) {
+ std::filesystem::path in = ToNormalPath(path_raw);
+ if (not in.is_absolute()) {
+ try {
+ in = std::filesystem::absolute(in);
+ } catch (std::exception const& e) {
+ Logger::Log(LogLevel::Error,
+ "Failed to convert input path {} ({})",
+ path_raw,
+ e.what());
+ throw e;
+ }
+ }
+ clargs->location = in;
+ },
+ "The path on the local file system to be added to CAS")
+ ->required();
+ app->add_flag("--follow-symlinks",
+ clargs->follow_symlinks,
+ "Resolve the positional argument to not be a symbolic link "
+ "before adding it to CAS.");
+}
+
static inline auto SetupGraphArguments(
gsl::not_null<CLI::App*> const& app,
gsl::not_null<GraphArguments*> const& clargs) {
diff --git a/src/buildtool/main/TARGETS b/src/buildtool/main/TARGETS
index 7f293ee5..1dc35eaa 100644
--- a/src/buildtool/main/TARGETS
+++ b/src/buildtool/main/TARGETS
@@ -37,6 +37,7 @@
, "cli"
, "version"
, "analyse"
+ , "add_to_cas"
, "install_cas"
, "describe"
, "diagnose"
diff --git a/src/buildtool/main/cli.cpp b/src/buildtool/main/cli.cpp
index 449e12fd..39f0a17e 100644
--- a/src/buildtool/main/cli.cpp
+++ b/src/buildtool/main/cli.cpp
@@ -105,6 +105,20 @@ auto SetupInstallCasCommandArguments(
SetupRetryArguments(app, &clargs->retry);
}
+/// \brief Setup arguments for sub command "just install-cas".
+auto SetupAddToCasCommandArguments(
+ gsl::not_null<CLI::App*> const& app,
+ gsl::not_null<CommandLineArguments*> const& clargs) {
+ SetupCompatibilityArguments(app);
+ SetupCacheArguments(app, &clargs->endpoint);
+ SetupExecutionEndpointArguments(app, &clargs->endpoint);
+ SetupCommonAuthArguments(app, &clargs->auth);
+ SetupClientAuthArguments(app, &clargs->cauth);
+ SetupLogArguments(app, &clargs->log);
+ SetupRetryArguments(app, &clargs->retry);
+ SetupToAddArguments(app, &clargs->to_add);
+}
+
/// \brief Setup arguments for sub command "just traverse".
auto SetupTraverseCommandArguments(
gsl::not_null<CLI::App*> const& app,
@@ -173,6 +187,8 @@ auto ParseCommandLineArguments(int argc, char const* const* argv)
"rebuild", "Rebuild and compare artifacts to cached build.");
auto* cmd_install_cas =
app.add_subcommand("install-cas", "Fetch and stage artifact from CAS.");
+ auto* cmd_add_to_cas = app.add_subcommand(
+ "add-to-cas", "Add a local file or directory to CAS.");
auto* cmd_gc =
app.add_subcommand("gc", "Trigger garbage collection of local cache.");
auto* cmd_execution = app.add_subcommand(
@@ -192,6 +208,7 @@ auto ParseCommandLineArguments(int argc, char const* const* argv)
SetupInstallCommandArguments(cmd_install, &clargs);
SetupRebuildCommandArguments(cmd_rebuild, &clargs);
SetupInstallCasCommandArguments(cmd_install_cas, &clargs);
+ SetupAddToCasCommandArguments(cmd_add_to_cas, &clargs);
SetupTraverseCommandArguments(cmd_traverse, &clargs);
SetupGcCommandArguments(cmd_gc, &clargs);
SetupExecutionServiceCommandArguments(cmd_execution, &clargs);
@@ -226,6 +243,9 @@ auto ParseCommandLineArguments(int argc, char const* const* argv)
else if (*cmd_install_cas) {
clargs.cmd = SubCommand::kInstallCas;
}
+ else if (*cmd_add_to_cas) {
+ clargs.cmd = SubCommand::kAddToCas;
+ }
else if (*cmd_traverse) {
clargs.cmd = SubCommand::kTraverse;
}
diff --git a/src/buildtool/main/cli.hpp b/src/buildtool/main/cli.hpp
index 04ae2a16..e03e9bec 100644
--- a/src/buildtool/main/cli.hpp
+++ b/src/buildtool/main/cli.hpp
@@ -26,6 +26,7 @@ enum class SubCommand {
kInstall,
kRebuild,
kInstallCas,
+ kAddToCas,
kTraverse,
kGc,
kExecute,
@@ -53,6 +54,7 @@ struct CommandLineArguments {
ServeArguments serve;
RetryArguments retry;
GcArguments gc;
+ ToAddArguments to_add;
};
auto ParseCommandLineArguments(int argc, char const* const* argv)
diff --git a/src/buildtool/main/main.cpp b/src/buildtool/main/main.cpp
index ea17e9ea..0c5c0c15 100644
--- a/src/buildtool/main/main.cpp
+++ b/src/buildtool/main/main.cpp
@@ -41,6 +41,7 @@
#include "src/buildtool/logging/log_sink_cmdline.hpp"
#include "src/buildtool/logging/log_sink_file.hpp"
#include "src/buildtool/logging/logger.hpp"
+#include "src/buildtool/main/add_to_cas.hpp"
#include "src/buildtool/main/analyse.hpp"
#include "src/buildtool/main/build_utils.hpp"
#include "src/buildtool/main/cli.hpp"
@@ -953,6 +954,11 @@ auto main(int argc, char* argv[]) -> int {
? kExitSuccess
: kExitFailure;
}
+ if (arguments.cmd == SubCommand::kAddToCas) {
+ return AddArtifactsToCas(arguments.to_add, traverser.GetRemoteApi())
+ ? kExitSuccess
+ : kExitFailure;
+ }
#endif // BOOTSTRAP_BUILD_TOOL
auto [main_repo, main_ws_root] =
diff --git a/src/other_tools/just_mr/utils.hpp b/src/other_tools/just_mr/utils.hpp
index f35495c8..909918bd 100644
--- a/src/other_tools/just_mr/utils.hpp
+++ b/src/other_tools/just_mr/utils.hpp
@@ -111,6 +111,15 @@ std::map<std::string, JustSubCmdFlags> const kKnownJustSubcommands{
.remote_props = true,
.serve = true,
.dispatch = true}},
+ {"add-to-cas",
+ {.config = false,
+ .build_root = true,
+ .launch = false,
+ .defines = false,
+ .remote = true,
+ .remote_props = false,
+ .serve = false,
+ .dispatch = false}},
{"install-cas",
{.config = false,
.build_root = true,
generated by cgit v1.2.3 (git 2.39.1) at 2025-07-15 05:54:34 +0000