summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2023-11-14just_mr utils: Add missing nodiscard attributesPaul Cristian Sarbu
2023-11-14fs_utils: Only accept strings as 'checkouts' map valuesPaul Cristian Sarbu
While we don't want to fail if the 'checkouts' map values are not strings, we shouldn't just accept non-string values either, instead we should warn the user and continue without them.
2023-11-14commit_git_map fix: Fetch from correct location if fetch URL is a pathPaul Cristian Sarbu
2023-11-14alternative-mirrors design doc: Update local specification key namesPaul Cristian Sarbu
This improves clarity while maintaining specificity of the JSON file entries. Also fixes the typo in the design document name.
2023-11-13test: Verify unlink of symlinksOliver Reiche
2023-11-13bugfix: Also unlink symlinks before installingOliver Reiche
Make sure that all CopyFile, WriteFile, and CreateSymlink functions properly unlink the target file (if it exists and overwrite requested) to avoid interferences of the install command. With this change, the clean up step for install-cas and the within GraphTraverser can new be omitted.
2023-11-13Add end-to-end test for environment handling of "git tree" repositoriesKlaus T. Aehlig
... ensuring that just-mr passes on the specified environment variables (and only those) to the tree-generating action.
2023-11-13"git tree" repositories: honor "inherit env"Klaus Aehlig
In the specification of the action generating a fixed git tree, also honor "inherit env", i.e., inherit the environment variables specified in this field from the environment just-mr is invoked in. As the expected output is fixed ahead of time anyway, this lack of isolation does not affect correctness.
2023-11-09["CC/auto", "config_file"] runner: let /usr/bin/env find python3Klaus T. Aehlig
... instead of assuming it to be installed under /usr/bin as we also do in other places. In this way, we do not make any additional assumptions on top of the already existing one on the path of env.
2023-11-07Bazel API: implement ParallelRetrieveToCasKlaus Aehlig
... using thread-based parallelism for the blobs of each tree.
2023-11-07artifacts sync: use parallelism if providedKlaus Aehlig
Use parallelism if provided by the build API when synchronizing artifacts of export targets and when synchronizing artifacts due to the --remember option. Do so at build parallelism as this the parallelism suitable for the build API.
2023-11-07remote api: support otpional parallel CAS synchronisationKlaus Aehlig
Allow implementations to use a given number of threads to carry out the synchronisation. In this way parallelism can be achieved even in situations where batch reading degrades as objects of unknown size have to be fetched; this situation typically occurs if a tree object has a large number of direct children that are blobs. If no implementation is provided, the default implementation is to fall back to the normal (sequential) CAS synchronisation.
2023-11-07Fetch export targets sequentially when creating local backupKlaus Aehlig
Typically, the number of export targets is small compared to the size of the respective export targets. Moreover, export targets are often chained, resulting in overlapping artifacts that we want to fetch only once. Therefore, fetch export targets sequentially, giving room to introduce parallelism in the individual fetch steps later. While there, also log the beginning of the artifact synchronisation for the export targets.
2023-11-02Document the new featuresPaul Cristian Sarbu
2023-11-02just-mr fetch: Add handling of --fetch-absent optionPaul Cristian Sarbu
2023-11-02test: Add simple check that 'just serve' resolves symlinks as expectedPaul Cristian Sarbu
2023-11-02serve test runner: Fix style and missing counter incrementPaul Cristian Sarbu
2023-11-02Add end-to-end test for fetching absent archivesKlaus Aehlig
Includes check to ensure we do not cache subtrees unnecessarily. Co-authored-by: Paul Cristian Sarbu <paul.cristian.sarbu@huawei.com>
2023-11-02content git map: Add handling of --fetch-absent optionPaul Cristian Sarbu
2023-11-02content git map: Move handling of --fetch-absent flag from checkout to the ↵Paul Cristian Sarbu
map itself
2023-11-02Move content archive helper functions in own libraryPaul Cristian Sarbu
2023-11-02remote serve: Extend client and API with tree-of-archive service callsPaul Cristian Sarbu
2023-11-02just serve: Implement service to get Git tree of an archive contentPaul Cristian Sarbu
Also adds missing TARGETS file in serve_api folder and ensures code comments are consistent with the proto file.
2023-11-02just serve: Fix inconsistenciesPaul Cristian Sarbu
- add missing serve_api TARGETS file - rename service client to align with server naming scheme - fix inconsistencies in comments between implementation and protocol
2023-11-02Decoupling symlinks map and CAS utilities from just-mrPaul Cristian Sarbu
This is required in order to make them available to 'just serve' in a minimal just installation.
2023-11-02ArchiveOps: Move libarchive utilities outside other_toolsPaul Cristian Sarbu
This way they can be used by 'just serve'.
2023-11-02GitRepo: Add method for async fetch from local repositoryPaul Cristian Sarbu
This avoids using the more geenric GitRepoRemote method which has libcurl as a dependency, something that is not needed for this Git operation.
2023-11-02GitRepo: Add blob existence checkerPaul Cristian Sarbu
Also fixes a small typo in tree existence checker log messages.
2023-11-02just serve: add a request to get the tree of an archiveKlaus Aehlig
Co-authored-by: Paul Cristian Sarbu <paul.cristian.sarbu@huawei.com>
2023-11-02content CAS map: Check remote CAS only after local distfilesPaul Cristian Sarbu
2023-11-02bootstrap: Add libarchive dependencyPaul Cristian Sarbu
This is brought in by the tree-of-archive rpc of just serve. Also adds lzma and bzip2 as transitive dependencies.
2023-10-27Add a test for similar but different exportsKlaus Aehlig
Export the same target from different repositories, as well as an explicit file reference with the same name and verify that no spourious cache hits occur.
2023-10-27Base export target chache key on the exported targetKlaus Aehlig
The cache key for an export target should contain as target name that of the export target (and its effective configuration) rather than the exported target. As we computed the repository part of the cache key for the target included in the key, this was still a correct cache key except in the case an explicit file reference was exported (as here, the information that the file was to be taken rather than the target of the same name got lost). We still fix this issue by making the implementation match our design (rather than by including the file-reference bit in the cache key), as the original design gives the cleaner protocol for target-level caching as a service.
2023-10-27extended-git-tree-variables doc: fix typoPaul Cristian Sarbu
2023-10-27Add design about having additional variables available in git-tree actionsKlaus Aehlig
2023-10-25just execute: fix race condition during garbage collection...Alberto Sartori
...of the internal cache used for keeping track of running operations.
2023-10-25just serve: document the changes required in the analysis of an export targetAlberto Sartori
Co-authored-by: Paul Cristian Sarbu <paul.cristian.sarbu@huawei.com>
2023-10-24just-execute tutorial: recommend statically linked binaresKlaus Aehlig
... and point to the repository with machine-readable instructions on how to properly build them.
2023-10-23Add a design for supporting additional mirrors in just-mrKlaus Aehlig
2023-10-20with_serve_test_runner.py: terminate serve process at the endAlberto Sartori
2023-10-16curl_easy_handle: Ensure we report a fail on HTTP codes >=400Paul Cristian Sarbu
...which signal either cient- or server-side failures. This ensures we exit with a failed network fetch early in cases where it is clear we won't receive useful data.
2023-10-16just-mr fetch: Fix exception on fetched data hash mismatchPaul Cristian Sarbu
After successfully fetching data over the network for an archive, if the optional checksums are not provided, the code will throw if there is a mismatch between the hash of the fetched data stream and the provided content hash. This commit fixes the issue by adding an additional check which properly handles the possible mismatch.
2023-10-06CreateExecutionApi: define a new library to avoid code duplicationAlberto Sartori
2023-09-29rule ["patch", "defaults"]: remove unused config variable "TARGET_ARCH"Klaus Aehlig
2023-09-28Avoid conflicts on case-insensitive file systemsKlaus Aehlig
... by renaming test/end-to-end/targets.
2023-09-28["CC/pkgconfig", "system_library"] Add -rpathKlaus Aehlig
When a shared library is picked up from the host system via pkg-config that is located at a non-standard location, binaries linked against this library by a standard linker either need LD_LIBRARY_PATH set appropriately, or -rpath set at link time. However, not all custom installations set -rpath in the link flags in their pkg-config file. Therefore, in order to get working binaries (and not have to set custom paths in tests), add a -rpath for every -L option found in the ldflags provided by pkg-config.
2023-09-27Use standard tools for pkg-configKlaus Aehlig
2023-09-26pkg-config files: properly terminate last lineKlaus Aehlig
2023-09-25Fix pkg-config name for libprotobufOliver Reiche
2023-09-25Use pkgconfig for default proto libsOliver Reiche