summaryrefslogtreecommitdiff
path: root/src/buildtool
AgeCommit message (Collapse)Author
2023-11-15just-execute: verify the validity of all the hashes received over the wireAlberto Sartori
2023-11-14Fix serialization of the target cache keyOliver Reiche
... which was accidentially a list of (a single) object, instead of only a single JSON object.
2023-11-14fs_utils: Only accept strings as 'checkouts' map valuesPaul Cristian Sarbu
While we don't want to fail if the 'checkouts' map values are not strings, we shouldn't just accept non-string values either, instead we should warn the user and continue without them.
2023-11-13bugfix: Also unlink symlinks before installingOliver Reiche
Make sure that all CopyFile, WriteFile, and CreateSymlink functions properly unlink the target file (if it exists and overwrite requested) to avoid interferences of the install command. With this change, the clean up step for install-cas and the within GraphTraverser can new be omitted.
2023-11-07Bazel API: implement ParallelRetrieveToCasKlaus Aehlig
... using thread-based parallelism for the blobs of each tree.
2023-11-07artifacts sync: use parallelism if providedKlaus Aehlig
Use parallelism if provided by the build API when synchronizing artifacts of export targets and when synchronizing artifacts due to the --remember option. Do so at build parallelism as this the parallelism suitable for the build API.
2023-11-07remote api: support otpional parallel CAS synchronisationKlaus Aehlig
Allow implementations to use a given number of threads to carry out the synchronisation. In this way parallelism can be achieved even in situations where batch reading degrades as objects of unknown size have to be fetched; this situation typically occurs if a tree object has a large number of direct children that are blobs. If no implementation is provided, the default implementation is to fall back to the normal (sequential) CAS synchronisation.
2023-11-07Fetch export targets sequentially when creating local backupKlaus Aehlig
Typically, the number of export targets is small compared to the size of the respective export targets. Moreover, export targets are often chained, resulting in overlapping artifacts that we want to fetch only once. Therefore, fetch export targets sequentially, giving room to introduce parallelism in the individual fetch steps later. While there, also log the beginning of the artifact synchronisation for the export targets.
2023-11-02remote serve: Extend client and API with tree-of-archive service callsPaul Cristian Sarbu
2023-11-02just serve: Implement service to get Git tree of an archive contentPaul Cristian Sarbu
Also adds missing TARGETS file in serve_api folder and ensures code comments are consistent with the proto file.
2023-11-02just serve: Fix inconsistenciesPaul Cristian Sarbu
- add missing serve_api TARGETS file - rename service client to align with server naming scheme - fix inconsistencies in comments between implementation and protocol
2023-11-02Decoupling symlinks map and CAS utilities from just-mrPaul Cristian Sarbu
This is required in order to make them available to 'just serve' in a minimal just installation.
2023-11-02GitRepo: Add method for async fetch from local repositoryPaul Cristian Sarbu
This avoids using the more geenric GitRepoRemote method which has libcurl as a dependency, something that is not needed for this Git operation.
2023-11-02GitRepo: Add blob existence checkerPaul Cristian Sarbu
Also fixes a small typo in tree existence checker log messages.
2023-11-02just serve: add a request to get the tree of an archiveKlaus Aehlig
Co-authored-by: Paul Cristian Sarbu <paul.cristian.sarbu@huawei.com>
2023-10-27Base export target chache key on the exported targetKlaus Aehlig
The cache key for an export target should contain as target name that of the export target (and its effective configuration) rather than the exported target. As we computed the repository part of the cache key for the target included in the key, this was still a correct cache key except in the case an explicit file reference was exported (as here, the information that the file was to be taken rather than the target of the same name got lost). We still fix this issue by making the implementation match our design (rather than by including the file-reference bit in the cache key), as the original design gives the cleaner protocol for target-level caching as a service.
2023-10-25just execute: fix race condition during garbage collection...Alberto Sartori
...of the internal cache used for keeping track of running operations.
2023-10-06CreateExecutionApi: define a new library to avoid code duplicationAlberto Sartori
2023-09-22RemoteServeConfig: Remove problematic inheritancePaul Cristian Sarbu
This was causing the remote serve address to overwrite the one set for remote execution. Also, to keep things clean, some common remote server-related methods and definitions were moved into their own library.
2023-09-19just serve protocol: clean upKlaus Aehlig
Just like the remote-execution protocol has several services (Execution, ActionCache, ContentAddressableStorage, etc), so will the serve protocol: the actual target-level caching, as well auxilliary services, like the service to obtain the tree for a given root. Already follow that scheme, before the protocol gets part of any release. Also, move the status enum into the respective answer messages. In this way, we can have different enums for different requests without causing conflicts on the named enum constants.
2023-09-19ServeServerImpl: create and `git-init` StorageConfig::GitRoot() when the ↵Alberto Sartori
server is started
2023-09-15just serve: add remote execution endpoint and --fetch-absent optionPaul Cristian Sarbu
The serve service will communicate with this endpoint when needed, as well as ensure artifacts it provides are synced with the remote execution CAS, if requested by the client. If just-mr is given the --fetch-absent option, it Always produce present roots irrespective of the 'absent' pragma. For Git repositories marked with the 'absent' pragma, first try to fetch any commit trees provided by the serve endpoint from the execution endpoint CAS, before reverting to a network fetch. Co-authored-by: Klaus Aehlig <klaus.aehlig@huawei.com> Co-authored-by: Alberto Sartori <alberto.sartori@huawei.com>
2023-09-15GitApi: implement RetrieveToCASAlberto Sartori
This functionality will be needed to upload git trees to a remote-execution end point by `just serve.
2023-09-13just serve: Add endpoint argumentsPaul Cristian Sarbu
2023-09-13just serve: Implement basic API handling service requestsPaul Cristian Sarbu
2023-09-13just serve: Implement initial serve clientPaul Cristian Sarbu
...able to request the tree of a commit known to the remote.
2023-09-13just serve: Basic service implementation with commit-of-tree RPCPaul Cristian Sarbu
Initial version, to be extended later with other RPCs.
2023-09-13just: Add handling of 'just serve' configurationPaul Cristian Sarbu
2023-09-13remote: Extract port parsing in own libraryPaul Cristian Sarbu
2023-09-13remote: Extract common client implementation utils in own libraryPaul Cristian Sarbu
2023-09-13targets: Add missing explicit dependencies and fix library namePaul Cristian Sarbu
2023-09-01fmt: Fix includes to only bring in the core APIPaul Cristian Sarbu
There is no need to use the full format API, so avoid increasing the compile time gratuitously.
2023-08-28Extend built-in "generic" rule to allow setting "sh"Klaus Aehlig
2023-08-25just describe: also show the implict dependenciesKlaus Aehlig
... in the human-readable description. More precisely, for any implicit target field, show - the field documentation given (if any), and - the resolved targets this field contains. However, do not show the actual name of the field, as this name is internal to the rule and not part of the interface.
2023-08-25Start 1.3 development cycleKlaus Aehlig
2023-08-25Release 1.2.0v1.2.0Klaus Aehlig
2023-08-24LinkedMap: Simplify ::Find() methodsOliver Reiche
... to silence false-positive 'possible dangling reference' warning produced by gcc 13.2.0.
2023-08-24Execution: only take complete actions from cacheKlaus Aehlig
... which are only actions that, besides giving exit code 0 also created all the outputs they promised to.
2023-08-23target_map: Use iterators instead of pointer arithmeticOliver Reiche
2023-08-23just: split main codeKlaus Aehlig
... by moving the diagnose and cli related code into to separate libraries. In this way, we hopefully improve maintainability of the code base.
2023-08-22Release 1.2.0~beta3v1.2.0-beta3Oliver Reiche
2023-08-18Release 1.2.0~beta2v1.2.0-beta2Klaus Aehlig
2023-08-16Release 1.2.0~beta1v1.2.0-beta1Klaus Aehlig
2023-08-14expression: add new built in "reverse"Klaus Aehlig
While a foldl is enough to implement a reverse functionality, adding it as a built in allows doing so in linear time.
2023-08-14expression: add new built in "set"Klaus Aehlig
... to obtain from a list of strings a map with those entries as keys and true as value. In this way, repeated membership tests in lists can be implemented more efficiently.
2023-08-14expressions: add constant for trueKlaus Aehlig
2023-08-11just build: allow -P to reach into treesKlaus Aehlig
2023-08-11Move downstepping to subobjects to a libraryKlaus Aehlig
... in order to make that functionality reusable
2023-08-10built-in: Add 'symlink' rulePaul Cristian Sarbu
The rule generates a non-upwards symbolic link with given target path.
2023-08-09Update fmt libraryKlaus Aehlig