summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2024-03-07exceptions handling: small improvementsPaul Cristian Sarbu
It is better to avoid empty catches when we have a reasonable way to log an (even unlikely) exception, even more so if memory allocations are involved.
2024-03-07just-mr fetch: Correctly report if no fetch requiredPaul Cristian Sarbu
2024-03-07absent_target_map: Improve log messagesPaul Cristian Sarbu
2024-03-07just-mr: Correctly pass client-side auth args to 'just'...Paul Cristian Sarbu
...also for 'analyse' and 'describe'.
2024-03-07just: 'analyse' and 'describe' require client-side authenticationPaul Cristian Sarbu
As these subcommands can access the serve endpoint, it is not enough anymore to just know the remote-execution endpoint, but one must also try to connect to it. Thus, the client-side authentication arguments need to be supported and read. The just.1 man page is also updated accordingly.
2024-03-05end-to-end test: also test foreign-file setup in compatible modeKlaus Aehlig
2024-03-05test distdir serving: verify the generic caseKlaus Aehlig
... where the files are known to serve, but not the resulting directory. We achieve this, by adding an unrealted file to the repository exposed to the serve server.
2024-03-05distdir serving: do not confuse hashesKlaus Aehlig
When just serve has to create a distdir from files known to it, there are two kind of hashes involved. - The requests, as well as the resulting tree always refer to git identifiers. - Additionally, files can be stored in the local CAS of the serve instance. This CAS might use a different hash, e.g., plain sha256. Therefore, we cannot simply forward the hash from the request to a lookup in our local CAS; fortunately, when we add a file to the local CAS, we are told the value of the applicable hash, represented as bazel digest, hence we can simply remember essentially this value (converting it to unprefixed form when running in native mode, as in the following we need the hash as it is part of an artifact digest).
2024-03-05foreign_file_git_map: add missing returnKlaus Aehlig
... after handling absent roots. Otherwise, we violate the condition of calling the setter precisely once.
2024-03-05test summarizer: in case of FLAKY tests, also report failure rateKlaus Aehlig
2024-03-05Update documentation, now that "foreign file" roots are implementedKlaus Aehlig
2024-03-05Add a test verifying that foreign files correctly use serveKlaus Aehlig
... by using the distdir request of the serve API.
2024-03-05Add a test for foreign-file rootsKlaus Aehlig
2024-03-05Support fetching of foreign filesKlaus Aehlig
2024-03-05Support checkout of foreign-file reposKlaus Aehlig
2024-03-05Add a root_map for foreign-file reposKlaus Aehlig
2024-03-05serve client: add custom function to ask for a foreign-file rootKlaus Aehlig
2024-03-05fs_utils: specify path for foreign-file root cacheKlaus Aehlig
As the internal distdir data structure now supports the executable bit, it is also expressive enough to support foreign-file repositories. Hence we can use this cache, getting potentially more cache hits.
2024-03-05Add data structure for foreign-file rootsKlaus Aehlig
2024-03-04serve protocol: support executable distfilesKlaus Aehlig
As serve and just-mr share their caching of description-tree association, we also have to change this cache. Thanks to json encoding before hashing, we know that the old and new hash keys do not overlap, so the change is save. As distdirs also keep the respective files in the git root, no new downlaoding will happen either, hence no warning in the CHANGELOG is needed.
2024-02-29expressions: add kFalseKlaus Aehlig
2024-02-29archive_fetch_map: only have the actual archive content in the interfaceKlaus Aehlig
... as this is the only part this map depends on, and not the information on what to do with the archive, like style of unpacking, etc. In this way, this map will also be reusable for fetching foreign files.
2024-02-29just-mr and serve: Extend use of noexcept specifierPaul Cristian Sarbu
The just serve client-side and API methods, as well as just-mr utilities should use the noexcept specifier.
2024-02-29end-to-end: Print envariables in all just serve testsPaul Cristian Sarbu
...to help with debugging.
2024-02-29verify_hash utils: Clarify return value meaningPaul Cristian Sarbu
2024-02-29serve source tree: Do not sync trees with the remote if in compatible modePaul Cristian Sarbu
2024-02-29serve source tree: Fix bugs and inconsistencies in remote syncPaul Cristian Sarbu
Firstly, in ServeDistdirTree the tree of a distdir should be synced from the corresponding witnessing Git repository (as is the case with all root trees), but was wrongly trying to sync from the local CAS. Secondly, a status of SYNC_ERROR, according to the protocol, must always ensure the tree field is set, but some setter calls were missing. Thirdly, ServeContent and ServeTree failed to report on successfully syncing to remote if the blob or tree, respetively, were uploaded from the local CAS. Lastly, ServeDistdirTree contained some legacy code sections and comments that were superfluous. This commit fixes these issues.
2024-02-28Deduplicate parsing of archive descriptionsKlaus Aehlig
2024-02-27["end-to-end/git-import", "annotations"]: propagate PATHKlaus Aehlig
... to generic actions, in order to not rely on "standard paths" being pulled in.
2024-02-27["end-to-end/remote-execution", "split-splice"] set default TIMEOUT_SCALEKlaus Aehlig
As this test takes significantly more time than the other end-to-end tests, set a suitable default value for the configuration variable TIMEOUT_SCALE.
2024-02-27with_{remote,serve}: honor TIMEOUT_SCALEKlaus Aehlig
Especially the end-to-end tests that use "remote" end points can grow large; therefore, honor the configuration variable TIMEOUT_SCALE, to allow the author to specify that this test takes more time.
2024-02-27test just-mr and serve: Check distdir repository root creationPaul Cristian Sarbu
Extends test coverage for the interaction between 'just-mr setup' and the serve endpoint for distdir repositories.
2024-02-27test just-mr and serve: Check git-tree repository root creationPaul Cristian Sarbu
Extends test coverage for the interaction between 'just-mr setup' and the serve endpoint for git-tree repositories.
2024-02-27test just-mr and serve: Check file repository to_git root creationPaul Cristian Sarbu
Extends test coverage for the interaction between 'just-mr setup' and the serve endpoint for file repositories with to_git pragma.
2024-02-27test just-mr and serve: Check git repository root creationPaul Cristian Sarbu
Extends test coverage for the interaction between 'just-mr setup' and the serve endpoint for git repositories.
2024-02-27test just-mr and serve: Check archive repository root creationPaul Cristian Sarbu
Extends test coverage for the interaction between 'just-mr setup' and the serve endpoint for archive repositories.
2024-02-27just-mr: Correct handling of remotes with respect to compatibility modePaul Cristian Sarbu
In order to set up roots, just-mr is able to interrogate, if given, serve and/or remote-execution endpoints. However, just-mr operates only with Git hashes, i.e., with a native mode CAS. This commit ensures the correct interactions occur between just-mr and the provided endpoints not only in native mode, but also in comaptible mode, where a serve endpoint might be present even if one cannot make use of its associated remote-exection endpoint. The user always gets informed if any incompatibilities are detected.
2024-02-27just serve: Add request to report compatibility status of endpointPaul Cristian Sarbu
This request is needed by just-mr in order to verify that it can use a provided serve endpoint when setting up the repository roots.
2024-02-27just-mr: Fix correct CAS storage location is usedPaul Cristian Sarbu
just-mr should always operate with the CAS location corresponding to the native protocol, i.e., using Git hashes. This way all the checks and transactions between local CAS and the Git cache are correct. The commit fixes the issue by ensuring we don't set globally the compatibility mode or hashing function if being passed the --compatible flag, as this flag should only be used to check comaptibility with any given remote endpoint and not affect the local CAS location used by just-mr.
2024-02-27commit_git_map: Proper handling of file associationPaul Cristian Sarbu
This commit fixes the invariant that a file association between a Git commit and the root tree should only be set if that tree is found in our own Git cache. This ensures consistency between present and absent roots and in the interaction with the serve endpoint.
2024-02-27just-mr setup archive: Local roots require the archive blob to be local tooPaul Cristian Sarbu
For archive repositories we need to ensure that a non-absent root is backed by an archive content blob in the local CAS, in order to also keep the proper root tree file associations. This change also simplifies the content_cas_map logic by removing the previous separation of implementation logic between fetching and setting up the workspace root.
2024-02-27symlinks design doc: Fix typoPaul Cristian Sarbu
2024-02-26Add split-splice end-to-end testSascha Roloff
2024-02-26Employ blob splitting and splicing for endpoint dispatchingSascha Roloff
2024-02-26Implement blob splitting in local APISascha Roloff
2024-02-26Implement blob splicing protocol at just client sideSascha Roloff
2024-02-26Implement blob splicing protocol at just server sideSascha Roloff
2024-02-26Implement blob chunking algorithm negotiationSascha Roloff
2024-02-26Update remote execution protocol to support blob splicingSascha Roloff
The latest remote execution protocol version includes a blob splicing RPC and allows for the negotiation of the used chunking algorithm between client and server.
2024-02-26Unify error logging in just execute BatchUpdateBlobsSascha Roloff