summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2023-11-15source_tree: add missing depAlberto Sartori
2023-11-15amend service-target-cache.mdAlberto Sartori
Remove wrong (un-needed) design specifications. When we compute the TargetCacheKey of an export target, there is no need to check if any of (recursively) referenced trees is absent. As soon as we deal with an "absent target" we query just serve. If it is not able to provide the target cache value, we error out.
2023-11-15artifact.hpp: make ToJson and FromJson consistentAlberto Sartori
The function ToJson used `file_type` key to express the type of the artifact, on the other hand, FromJson was expecting `type`. This patch makes the two functions consistent, prefering `file_type` for historical reasons.
2023-11-15target-cache.md: fix a typoAlberto Sartori
2023-11-15test ["end-to-end/just-mr", "git-tree-env"]: fix negative test conditionKlaus Aehlig
This test was only working by accident, as the missing target specification ["", "sources.txt"] was making the build fail, not the (supposedly) unset variable. Fix this by looking only at the setup command and expect failure there---and also unset the correct variable.
2023-11-15git tree repository: Creating absent roots should not run the commandPaul Cristian Sarbu
Also extended absent-roots test with this scenario.
2023-11-15just-execute: verify the validity of all the hashes received over the wireAlberto Sartori
2023-11-15utils/cpp: add IsAHash functionAlberto Sartori
This function is mainly used to check that the hash of a Digest received over the wire is a real hash, to prevent a malicious attack.
2023-11-14Update ChangelogKlaus Aehlig
... clarifying the effect of the target-cache-key change. In particular, explain that, as the cache-key format has changed in a non-overlapping way, no care has to be taken on update, but old target-cache entries will not be used by the newer version of the tool. While there, fix some typos and properly break lines.
2023-11-14Fix serialization of the target cache keyOliver Reiche
... which was accidentially a list of (a single) object, instead of only a single JSON object.
2023-11-14test: Set a fixed number of retries for starting Python servers...Paul Cristian Sarbu
...to avoid waiting for the test timeout on an internal error.
2023-11-14alternative-mirrors design doc: Move to concepts folderPaul Cristian Sarbu
...and reword content to move from proposal to implemented state.
2023-11-14man: Add documentation for specifying alternative mirrorsPaul Cristian Sarbu
2023-11-14just-import-git.1: Fix wordingPaul Cristian Sarbu
2023-11-14just-import-git: Add --mirror optionPaul Cristian Sarbu
This allows to specify upfront alternative locations from where the imported repository may be fetched. No action is taken in the import, instead the argument gets passed intot he resulting just-mr configuration.
2023-11-14test: Fetch from mirrors given in just's specification filePaul Cristian Sarbu
2023-11-14just-mr archives fetch: Add logic for local mirrors and preferred hostnamesPaul Cristian Sarbu
2023-11-14just-mr git fetch: Add logic for local mirrors and preferred hostnamesPaul Cristian Sarbu
2023-11-14curl_url_handle: Add method to replace the hostname of a URLPaul Cristian Sarbu
Also adds a section in the curl_url test suite.
2023-11-14curl_url_handle: Allow permissive parsing and URL extraction to be non-fatalPaul Cristian Sarbu
2023-11-14curl_url_handle: Improve documentationPaul Cristian Sarbu
2023-11-14just-mr: Add utility getters for alternative mirrors specificationsPaul Cristian Sarbu
2023-11-14just-mr: Parse mirrors specification from checkout locations filePaul Cristian Sarbu
2023-11-14test: Add check for 'mirrors' field in just-mr repositoriesPaul Cristian Sarbu
2023-11-14just-mr: Add 'mirrors' field to archive-like repositoriesPaul Cristian Sarbu
Also extends 'distdir' repositories logic accordingly.
2023-11-14curl_easy_handle: Allow non-fatal logging of errors in curl operationsPaul Cristian Sarbu
In order to allow non-fatal retries of fetches, be it from same remote or not (e.g., mirrors), the handle now reports with a caller-defined LogLevel.
2023-11-14DistdirCheckout: Improve log messagesPaul Cristian Sarbu
2023-11-14just-mr: Add 'mirrors' field to 'git' repositoriesPaul Cristian Sarbu
2023-11-14just_mr utils: Add missing nodiscard attributesPaul Cristian Sarbu
2023-11-14fs_utils: Only accept strings as 'checkouts' map valuesPaul Cristian Sarbu
While we don't want to fail if the 'checkouts' map values are not strings, we shouldn't just accept non-string values either, instead we should warn the user and continue without them.
2023-11-14commit_git_map fix: Fetch from correct location if fetch URL is a pathPaul Cristian Sarbu
2023-11-14alternative-mirrors design doc: Update local specification key namesPaul Cristian Sarbu
This improves clarity while maintaining specificity of the JSON file entries. Also fixes the typo in the design document name.
2023-11-13test: Verify unlink of symlinksOliver Reiche
2023-11-13bugfix: Also unlink symlinks before installingOliver Reiche
Make sure that all CopyFile, WriteFile, and CreateSymlink functions properly unlink the target file (if it exists and overwrite requested) to avoid interferences of the install command. With this change, the clean up step for install-cas and the within GraphTraverser can new be omitted.
2023-11-13Add end-to-end test for environment handling of "git tree" repositoriesKlaus T. Aehlig
... ensuring that just-mr passes on the specified environment variables (and only those) to the tree-generating action.
2023-11-13"git tree" repositories: honor "inherit env"Klaus Aehlig
In the specification of the action generating a fixed git tree, also honor "inherit env", i.e., inherit the environment variables specified in this field from the environment just-mr is invoked in. As the expected output is fixed ahead of time anyway, this lack of isolation does not affect correctness.
2023-11-09["CC/auto", "config_file"] runner: let /usr/bin/env find python3Klaus T. Aehlig
... instead of assuming it to be installed under /usr/bin as we also do in other places. In this way, we do not make any additional assumptions on top of the already existing one on the path of env.
2023-11-07Bazel API: implement ParallelRetrieveToCasKlaus Aehlig
... using thread-based parallelism for the blobs of each tree.
2023-11-07artifacts sync: use parallelism if providedKlaus Aehlig
Use parallelism if provided by the build API when synchronizing artifacts of export targets and when synchronizing artifacts due to the --remember option. Do so at build parallelism as this the parallelism suitable for the build API.
2023-11-07remote api: support otpional parallel CAS synchronisationKlaus Aehlig
Allow implementations to use a given number of threads to carry out the synchronisation. In this way parallelism can be achieved even in situations where batch reading degrades as objects of unknown size have to be fetched; this situation typically occurs if a tree object has a large number of direct children that are blobs. If no implementation is provided, the default implementation is to fall back to the normal (sequential) CAS synchronisation.
2023-11-07Fetch export targets sequentially when creating local backupKlaus Aehlig
Typically, the number of export targets is small compared to the size of the respective export targets. Moreover, export targets are often chained, resulting in overlapping artifacts that we want to fetch only once. Therefore, fetch export targets sequentially, giving room to introduce parallelism in the individual fetch steps later. While there, also log the beginning of the artifact synchronisation for the export targets.
2023-11-02Document the new featuresPaul Cristian Sarbu
2023-11-02just-mr fetch: Add handling of --fetch-absent optionPaul Cristian Sarbu
2023-11-02test: Add simple check that 'just serve' resolves symlinks as expectedPaul Cristian Sarbu
2023-11-02serve test runner: Fix style and missing counter incrementPaul Cristian Sarbu
2023-11-02Add end-to-end test for fetching absent archivesKlaus Aehlig
Includes check to ensure we do not cache subtrees unnecessarily. Co-authored-by: Paul Cristian Sarbu <paul.cristian.sarbu@huawei.com>
2023-11-02content git map: Add handling of --fetch-absent optionPaul Cristian Sarbu
2023-11-02content git map: Move handling of --fetch-absent flag from checkout to the ↵Paul Cristian Sarbu
map itself
2023-11-02Move content archive helper functions in own libraryPaul Cristian Sarbu
2023-11-02remote serve: Extend client and API with tree-of-archive service callsPaul Cristian Sarbu