summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2024-02-26Add split-splice end-to-end testSascha Roloff
2024-02-26Employ blob splitting and splicing for endpoint dispatchingSascha Roloff
2024-02-26Implement blob splitting in local APISascha Roloff
2024-02-26Implement blob splicing protocol at just client sideSascha Roloff
2024-02-26Implement blob splicing protocol at just server sideSascha Roloff
2024-02-26Implement blob chunking algorithm negotiationSascha Roloff
2024-02-26Update remote execution protocol to support blob splicingSascha Roloff
The latest remote execution protocol version includes a blob splicing RPC and allows for the negotiation of the used chunking algorithm between client and server.
2024-02-26Unify error logging in just execute BatchUpdateBlobsSascha Roloff
2024-02-26Refactor split and splice implementations.Sascha Roloff
Currently, the implementations of the split and splice operation are both hidden behind the Bazel API implementation. This was sufficient to implement splitting at the server and splicing at the client. In order to support the other direction of splitting at the client and splicing at the server while reusing their implementations, the code needs to be refactored. First, the functionality of split and splice are explicitly exposed at the general execution API interface and implemented in the sub APIs. Second, the implementations of split and splice are factored into a separate utils class.
2024-02-26Rename constant for just-internal hash lengthSascha Roloff
2024-02-26Remove upper and lower bounds for FastCDC random-number generationSascha Roloff
2024-02-26Initialize file chunker in mainSascha Roloff
2024-02-26Add a design document for foreign-file rootsKlaus Aehlig
2024-02-22Add test verifying error reporting on failing actionsKlaus Aehlig
Add an end-to-end test verifying that we report enough useful information in case of an action failing.
2024-02-22Executor: in case of a failing action also report originKlaus Aehlig
... if provided. This might help users to find the correct place in their code base causing the action to fail.
2024-02-21Add test verifying the annotations added by just-import-gitKlaus Aehlig
... i.e., the values for "mirror" and "inherit env"
2024-02-21just-import-git: support setting "inherit env"Klaus Aehlig
When importing a repository via just-import-git, allow to specify the value for the "inherit env" property for the repository just being imported.
2024-02-20Add test verifying that "inherit env" is honored for git fetchesKlaus Aehlig
2024-02-20git repo fetch: support "inherit env"Klaus Aehlig
When fetching git repositories, just-mr routinely shells out to git. In this case, allow the user to specify via "inherit env", which environment variables from the host environment should be made available in this action. Typical variables to inherit are ones providing credentials, like SSH_AUTH_SOCK. As the repository description specifies the commit that will be taken, and hence the resulting tree, correctness is not affected by the environement leaking in here.
2024-02-20Git URL scheme handling: whitelist path instead of non-path protocolsKlaus Aehlig
At some point we have to decide if a given git repository URL is a path. So far, we assumed that anything not starting with ssh://, http://, or https:// is a path. This ignores the facts that - the file:// scheme, while referring to a file, does not denote a relative path starting file://, - the [user@]host:path scheme is not a path on the local machine, - there exist the URL schemes git://, ftp://, and ftps://, and - future extension might add additional schemes. To also correctly handle new schemes that git might add (which we indeed can handle, as we simply shell out to the git binary), we reverse the approach: we give the user the means to unambigiously specify that they refer to a path on the local machine, by either - using the file:// scheme, - providing an absolute path starting with /, or - providing a relative path starting with ./ All other schemes will not be modified. The file scheme, as well as the git://, http://, and https:// scheme, are handled interally using libgit2; all others are passed on to git in an unmodified form.
2024-02-16export targets: Enforce the invariant when writing target cache entriesPaul Cristian Sarbu
We ensure that for each export target to be written to the target cache all its implied export targets are written to the target cache first. This ensures that the target cache maintains its consistency at all times with respect to export target dependencies.
2024-02-16async maps: Create utility library to handle cycle detectionPaul Cristian Sarbu
2024-02-16ArtifactDigest: Hashing the object should not take size into accountPaul Cristian Sarbu
As it is common to use ArtifactDigest objects with a default value of 0, the std::hash implementation should not take the size into account. The is_tree member however should be kept in as it always needs to have a value and has no sensible default.
2024-02-16serve target: Upload also the artifacts to remote CAS after orchestrated buildPaul Cristian Sarbu
2024-02-15absent-archives test: also verify fetching as distfileKlaus Aehlig
2024-02-15git-gc design: fix typosKlaus Aehlig
2024-02-15defaults: Support fully static linkingOliver Reiche
... of all produced binaries, including the intermediate ones: protoc and grpc_cpp_plugin.
2024-02-15Replace glibc sync stubs when linking pthreadsOliver Reiche
... glibc provides synchronization stubs for single-threaded environments as weak symobls. When linking pthreads, these weak symbols must be replaced by the strong symbols provided by the pthread library. For dynamically linking pthreads, this is done automatically. However, to support this for static linking, we must ensure to link the whole archive.
2024-02-15atomic: Fix potential raceOliver Reiche
... as any unguarded access to non-const members of the same shared_ptr instance require the use of `atomic_load` and `atomic_store`.
2024-02-15Remove unnecessarily quadratic expressionsKlaus Aehlig
... not that everyone has updated to 1.2 or later we can use the built-in expressions "reverse" and "set".
2024-02-14expression map: properly wrap logger for import callsKlaus Aehlig
... to provide an informative error message on how a rule is related to a particular import and, in particularly, at which expression a problem with the import occurred. While there, also improve the message in the other error case to follow our standard line-breaking scheme.
2024-02-13with-serve test runner: do not rely on readdir orderKlaus Aehlig
2024-02-13executor-api test: also set env for echoKlaus Aehlig
... in order to not rely on the default launcher to pull it in.
2024-02-13local-execution test: set launcherKlaus Aehlig
... to not rely on env implicitly pulling in echo to PATH.
2024-02-13execution-api test: also set the launcherKlaus Aehlig
... in order to not assume echo to be on the standard search path pulled in by env.
2024-02-13Add test that also for trees in the git cas we can follow pathsKlaus Aehlig
For a user `just install-cas` will show an entry without revealing where it found it---as it is content-addressable, it does not matter. Therefore, verify that accessing paths of a tree object also works regardless of where the tree is stored.
2024-02-13local_api: also for retrieve to memory, consult git api as fallbackKlaus Aehlig
2024-02-08rule_map: improve error reportingKlaus Aehlig
By showing the full entity name and also adding the usual newline character after every "While ..." clause.
2024-02-08NamedTarget: support ToString()Klaus Aehlig
2024-02-08JsonFileMap: refuse to read absent rootKlaus Aehlig
... instead of erroring on missing file. In this way, whenever a rule or expression from an absent root would have to be read, we get a meaningful error message and not a complaint about a file not being there.
2024-02-08Target map: on error in rule look up, report the rule being looked forKlaus Aehlig
2024-02-07graph_traverser test: provide env trough launcherKlaus T. Aehlig
... instead of hard-coding it in the example graphs. In this way, we can honor a provided PATH.
2024-02-07executor tests: only use hard-coded paths if no PATH is providedKlaus T. Aehlig
2024-02-07execution-api tests: provide PATH where neededKlaus T. Aehlig
... to make tests pass also on systems where /bin/sh does not, by default, pull in a path to "the standard tools".
2024-02-05end-to-end tests: compare json values with jqKlaus T. Aehlig
... and only let test do the check on the final resulting boolean, where the string representation is canonical. In this way, we avoid having to rely on the string representation of numbers, where, e.g., 1 and 1.0 are equally valid representations of the same number.
2024-02-05end-to-end tests with remote (and serve) endpoint: inhert PATHKlaus T. Aehlig
... in the local launcher for the execution end point. In this way, the tests also work on systems where sh does not pull in enough paths to have all the "usual" tools available. To also get a correct path on systems relying on sh pulling in enough paths, take what sh, launched with the default launcher, would believe the path is.
2024-02-05end-to-end tests: inherit pathKlaus T. Aehlig
... for test actions, by setting an appropriate local launcher. In this way, the tests can also be run on systems where sh does not pull in enough paths to have all the "usual" tools available.
2024-02-02Fix export of remote APIKlaus T. Aehlig
As we patch that API, it also can implicitly depend on the toolchain (and hence its config), if we use a compiled implementation of patch. Making the TOOLCHAIN_CONFIG a flexible part of the config allow to, again, build static binaries as usual.
2024-02-01Add end-to-end test describing absent export targetsKlaus Aehlig
2024-02-01describe: support -RKlaus Aehlig
... as, for absent repositories, we need to get the description from the serve end point. As a consequence, also support -r and --compatible, as the remote-execution endpoint needs to fit with the one for the serve endpoint.