summaryrefslogtreecommitdiff
path: root/src/buildtool/execution_api
AgeCommit message (Collapse)Author
2023-03-10execution service: implement WaitExecution and ↵Alberto Sartori
google::longrunning::Operations::GetOperation For each action that is executed, an entry is added to a shared thread safe cache. Once the number of operations stored exceeds twice 2^n, where n is given by the option --log-operations-threshold, at most 2^n operations will be removed, in a FIFO scheme.
2023-03-06Execution server: Fix OutputDirectory generation in compatible modeSascha Roloff
2023-02-28just execute: indicate the mode started inKlaus Aehlig
2023-02-28execution service: BatchUpdateBlobs check consitency of requested digestsAlberto Sartori
2023-02-28Various typo fixesPaul Cristian Sarbu
Co-authored-by: Sascha Roloff <sascha.roloff@huawei.com>
2023-02-22cleanup: Fix typos in code line comments and logging messagesPaul Cristian Sarbu
2023-02-22ExecutionApi: Retrieve to correct CAS directlyOliver Reiche
... by keeping track of each blob being a file or executable and storing it to the correct local physical CAS directory. The new flag is merely a hint and only used by the local execution API. Leaving it out will still correctly transfer the blob but may cause unnecessary duplicates in file CAS.
2023-02-15execution service: use alias namespaceAlberto Sartori
2023-02-15just-execute: report un-tagged hashesAlberto Sartori
2023-02-15ExecutionServiceImpl: refactor ExecuteAlberto Sartori
2023-02-15just execute: add more logs during executionAlberto Sartori
2023-02-15just execute: add action's stdout and stderr to cas...Alberto Sartori
so that they can also be reported on the client side
2023-02-14targets: fix deps structurePaul Cristian Sarbu
2023-02-10just execute: bugfix: honor do_not_cache booleanAlberto Sartori
2023-02-02execution-service: add new subcommand executeAlberto Sartori
This subcommand starts a single node remote execution service honoring the just native remote protocol. If the flag --compatible is provided, the execution service will honor the original remote build execution protocol. New command line args supported by this subcommand: -p,--port INT: Execution service will listen to this port. If unset, the service will listen to the first available one. --info-file TEXT: Write the used port, interface, and pid to this file in JSON format. If the file exists, it will be overwritten. -i,--interface TEXT: Interface to use. If unset, the loopback device is used. --pid-file TEXT Write pid to this file in plain txt. If the file exists, it will be overwritten. --tls-server-cert TEXT: Path to the TLS server certificate. --tls-server-key TEXT: Path to the TLS server key. Co-authored by: Klaus Aehlig <klaus.aehlig@huawei.com>
2023-02-02LocalStorage: add StoreTree(std::filesystem::path const &)Alberto Sartori
2023-02-02bytestream_client: use common header with settings shared by both server and ↵Alberto Sartori
client
2023-02-02remote/config.hpp: make ParsePort re-usable for execution-serviceAlberto Sartori
2023-02-02bytestream_client: report failures on LogLevel::ErrorAlberto Sartori
2023-02-01Update fmtlib to version 9.1.0Klaus Aehlig
Co-authored-by: Michael Thies <mail@mhthies.de>
2023-01-24BazelMsgFactory: Improve error reporting for creating treesOliver Reiche
2023-01-24FileSystemManager: Do not follow symlinksOliver Reiche
... and ensure that cascades of checks are performed with only a single filesystem stat per method.
2023-01-20Add local garbage collectionSascha Roloff
2023-01-20Move execution-backend-id calculation from target-level cache to local configSascha Roloff
This code movement is required to break a cyclic dependency coming with the introduction of the garbage collector. target_cache depends on garbage_collector and garbage_collector would depend on target_cache to determine the target-level-cache directory. After moving this calculation to a more general location, the cycle is broken.
2023-01-20Introduce multiple cache generationsSascha Roloff
2023-01-20Add check for same CAS in CAS-to-CAS artifact retrievalSascha Roloff
2023-01-20Move file_storage.hpp to file_system subdirectorySascha Roloff
2023-01-18authentication: add TLS and mTLS supportAlberto Sartori
2022-12-21Just-MR: Add utilities library for just-mrPaul Cristian Sarbu
2022-12-21Git CAS: Move Git tree ops to fake repo wrapper classPaul Cristian Sarbu
2022-11-24Fix root directory upload to honor tree invariant.Sascha Roloff
2022-11-21Use the newly-added concept of private-depsKlaus Aehlig
While there, also add all direct dependencies explicitly; using directly dependencies that are pulled in only indireclty causes problems from a maintainability point of view.
2022-11-04Remove unwanted native protocol prefixes from user-visible locationsSascha Roloff
2022-10-19Apply sharding as used in git to files stored in the just cache directory.Sascha Roloff
This commit also introduces an incompatible change, since it modifies the way how files are stored in the just cache directory. This modification reduces the number of files per directory and only introduces a maximum number new directories to avoid possible performance bottlenecks.
2022-10-19Add generation directory as part of the just cache root.Sascha Roloff
This change is introduced to be prepared for future changes such as garbage collection. It is an incompatible change compared to earlier just versions since it modifies the local path to the just cache directory, where among others the CASes, action cache, target-level cache are located.
2022-10-14Remove default value of ArtifactDigest::Create template parameterSascha Roloff
This enforces the explicit specification, which object type, either file or tree, should be used to create an artifact digest. This also avoids subtile errors at locations as in the previous commit, where files as well as trees are supposed to be handled, but digest creation mistakenly defaults to file object type.
2022-10-14Fix missing tree handling in BazelCasClient::ReadSingleBlob functionSascha Roloff
The initial assumption before introducing the native remote execution protocol was that only blobs were transmitted over the wire and put in a blob store. With the native remote execution protocol, also trees can be transmitted over the wire and they are put in a different store than blobs. Different hashing functions are used for blobs and trees and need to be calculated accordingly for data to be put in the correct store. This distinction of blobs and trees is now implemented. Additionally, a warning is added that checks equality between the requested digest and the digest computed by the received content. This check makes sense to detect, e.g., corrupted messages, unexpected transmission termination, or remote server failure.
2022-10-12Add copyright and license notice to all source and header filesKlaus Aehlig
Signed-off-by: Goetz Brasche <goetz.brasche@huawei.com>
2022-10-11Refactor ReadTreeInfos function names and add documentationSascha Roloff
2022-10-07Fix upload of known source treesOliver Reiche
When a tree is taken from a git root, it is not necessarily known on the remote site. So, as any missing artifact it has to be uploaded, recursively uploading the parts to keep the tree invariant. The function RetrieveToCas was doing the correct recursiv pattern, however inspecting trees incorrectly using the function ReadTreeInfos; the latter function, however, was obtaining all the leafs of the tree as is needed for a compatible action-input description. Add and use a function that reads the direct contents of a tree.
2022-10-07LocalTreeMap: Drop the use of the map entirelyOliver Reiche
... as for remote execution, the map entries are only used for the `install` subcommand. For local execution, much less tree objects are read from CAS when using this map. However, the performance benefit is barely measurable and therefore we rather remove this map entirely to reduce complexity.
2022-10-07LocalTreeMap: Prevent tree objects from being storedOliver Reiche
... to align with the original idea of caching a flat list of blob objects, without the need to recursively traverse any trees. Consequently, we cannot create any map entry in places where we do not have all sub-tree entries at hand (e.g., LocalAPI, BazelAPI, BazelResponse).
2022-10-07BazelNetwork: Fix use of DirectoryMap in native modeOliver Reiche
... which is not using `Directory` messages nor does it support the `GetTree()` rpc.
2022-10-07Increase log level for grpc-related failures to warningKlaus Aehlig
... so that any infra structure problem on the remote execution is reported by default.
2022-10-07Improve error message on filure to execute a remote actionKlaus Aehlig
... to emphasize that this is the extensional action identifier, not the intensional one.
2022-10-05LocalAction: Improve error message for stagingOliver Reiche
2022-09-13Fix build with gcc/g++Oliver Reiche
2022-09-12Introduced RetrieveToCas function in IExecutionApi to synchronize artifacts ↵Sascha Roloff
between different CASes
2022-09-12BazelApi chunk availability check returns provided ArtifactDigestsSascha Roloff
2022-09-12Removed ArtifactDigest is_tree() predicateSascha Roloff