summaryrefslogtreecommitdiff
path: root/src/buildtool
AgeCommit message (Collapse)Author
2023-03-14GitRepo: Fix memory leak in keep tag operationPaul Cristian Sarbu
2023-03-13Storage: Reworked storage and garbage collectionOliver Reiche
The improved GC implementation uses refactored storage classes instead of directly accessing "unknown" file paths. The required storage class refactoring is quite substantial and outlined in the following paragraphs. The module `buildtool/file_system` was extended by: - `ObjectCAS`: a plain CAS implementation for reading/writing blobs and computing digests for a given `ObjectType`. Depending on that type, files written to the file system may have different properties (e.g., the x-bit set) or the digest may be computed differently (e.g., tree digests in non-compatible mode). A new module `buildtool/storage` was introduced containing: - `LocalCAS`: provides a common interface for the "logical CAS", which internally combines three `ObjectCAS`s, one for each `ObjectType` (file, executable, tree). - `LocalAC`: implements the action cache, which needs the `LocalCAS` for storing cache values. - `TargetCache`: implements the high-level target cache, which also needs the `LocalCAS` for storing cache values. - `LocalStorage`: combines the storage classes `LocalCAS`, `LocalAC`, and `TargetCache`. Those are initialized with settings from `StorageConfig`, such as the build root base path or number of generations for the garbage collector. `LocalStorage` is templated with a Boolean parameter `kDoGlobalUplink`, which indicates that, on every read/write access, the garbage collector should be used for uplinking across all generations (global). - `GarbageCollector`: responsible for garbage collection and the global uplinking across all generations. To do so, it employs instances of `LocalStorage` with `kDoGlobalUplink` set to false, in order to avoid endless recursion. The actual (local) uplinking within two single generations is performed by the corresponding storage class (e.g., `TargetCache` implements uplinking of target cache entries between two target cache generations etc.). Thereby, the actual knowledge how data should be uplinked is implemented by the instance that is responsible for creating the data in the first place.
2023-03-13GC: Fix uplinking of ActionResults in compatibleOliver Reiche
... as internally we do not do the Tree-dance. Instead, we directly store the root Directory digest for ActionResult's OutputDirectories.
2023-03-13TargetCache: Drop dependency on execution apiOliver Reiche
2023-03-13TargetCache: Drop dependency on repository configOliver Reiche
2023-03-13Artifact: No user error on object info parse failOliver Reiche
2023-03-10Update nlohmann::json to 3.11.2Klaus Aehlig
2023-03-10Move common defaults for just and just-mr into a separate libraryKlaus Aehlig
... and, in this way, avoid duplication of these constants that might lead to inconsistencies between those two tools later.
2023-03-10bytestream server: bugfix: if a tree is requested, look for it *only*Alberto Sartori
in the tree CAS. Before this patch, after the checking in the tree cas, also the blob cas was checked leading to a NOT-FOUND error.
2023-03-10execution service: implement WaitExecution and ↵Alberto Sartori
google::longrunning::Operations::GetOperation For each action that is executed, an entry is added to a shared thread safe cache. Once the number of operations stored exceeds twice 2^n, where n is given by the option --log-operations-threshold, at most 2^n operations will be removed, in a FIFO scheme.
2023-03-08GitRepo: Add method to check existence of a Git treePaul Cristian Sarbu
2023-03-07Git: Move 'fake' repository log messages to more appropriate reporting levelPaul Cristian Sarbu
2023-03-06Execution server: Fix OutputDirectory generation in compatible modeSascha Roloff
2023-03-06GitRepo: Add missing retval check for git oid libgit2 callsPaul Cristian Sarbu
2023-03-06Targets: Fix minor deps structure issuePaul Cristian Sarbu
2023-03-03GitRepoRemote: Correctly honor SSL certification settings in fetch and ↵Paul Cristian Sarbu
commit update Uses the SSL certification utility method to correctly set the certification check options for the remote URL libgit2 calls. Due to the fact that remote operations are done via a temporary repository to allow concurrent work, the correct repository configuration needs to be interrogated. Thankfully, libgit2 provides a thread safe config snapshot object to be used in such scenarios. Also updates the existing GitRepoRemote tests accordingly.
2023-03-03GitRepoRemote: Add getter for config snapshotPaul Cristian Sarbu
2023-03-02just: make -D options overlayKlaus Aehlig
... instead of only honoring the latest argument. In this way, standard overlays coming from the launcher configuration can be amended in individual invocations.
2023-02-28just execute: indicate the mode started inKlaus Aehlig
2023-02-28execution service: BatchUpdateBlobs check consitency of requested digestsAlberto Sartori
2023-02-28expression map: improve error messageKlaus Aehlig
... by providing the fully-qualified name of an expression that could not be found.
2023-02-28Various typo fixesPaul Cristian Sarbu
Co-authored-by: Sascha Roloff <sascha.roloff@huawei.com>
2023-02-27Progress reporting: Extract generic logic from just base progress reporter classSascha Roloff
2023-02-27Progress reporting: Extract generic logic from just progress classSascha Roloff
2023-02-27Logging: Add --log-append clarg to justPaul Cristian Sarbu
2023-02-22cleanup: Fix typos in code line comments and logging messagesPaul Cristian Sarbu
2023-02-22ExecutionApi: Retrieve to correct CAS directlyOliver Reiche
... by keeping track of each blob being a file or executable and storing it to the correct local physical CAS directory. The new flag is merely a hint and only used by the local execution API. Leaving it out will still correctly transfer the blob but may cause unnecessary duplicates in file CAS.
2023-02-20Add logging command-line arguments to gc subcommandSascha Roloff
2023-02-17structure cleanup: move remote operations of GitRepo to other_tools...Paul Cristian Sarbu
...in order to not include unwanted dependencies in just proper. The new class extends the GitRepo class used for just's Git tree operations and gets used in all of just-mr's async maps.
2023-02-15CLI: enforce maximal number of positional argumentsKlaus Aehlig
..., i.e., the requirement that the positional arguments form a syntactically valid target name. An explicit error message (created by the target-name parsing) is preferrable over tacitly dropping all but the last up to two arguments (which is the meaning of CLI11's expected(2)). While there, drop duplicate full stop in error message.
2023-02-15execution service: use alias namespaceAlberto Sartori
2023-02-15just-execute: report un-tagged hashesAlberto Sartori
2023-02-15ExecutionServiceImpl: refactor ExecuteAlberto Sartori
2023-02-15just execute: add more logs during executionAlberto Sartori
2023-02-15just execute: add action's stdout and stderr to cas...Alberto Sartori
so that they can also be reported on the client side
2023-02-14targets: fix deps structurePaul Cristian Sarbu
2023-02-10just execute: bugfix: honor do_not_cache booleanAlberto Sartori
2023-02-02execution-service: add new subcommand executeAlberto Sartori
This subcommand starts a single node remote execution service honoring the just native remote protocol. If the flag --compatible is provided, the execution service will honor the original remote build execution protocol. New command line args supported by this subcommand: -p,--port INT: Execution service will listen to this port. If unset, the service will listen to the first available one. --info-file TEXT: Write the used port, interface, and pid to this file in JSON format. If the file exists, it will be overwritten. -i,--interface TEXT: Interface to use. If unset, the loopback device is used. --pid-file TEXT Write pid to this file in plain txt. If the file exists, it will be overwritten. --tls-server-cert TEXT: Path to the TLS server certificate. --tls-server-key TEXT: Path to the TLS server key. Co-authored by: Klaus Aehlig <klaus.aehlig@huawei.com>
2023-02-02LocalStorage: add StoreTree(std::filesystem::path const &)Alberto Sartori
2023-02-02bytestream_client: use common header with settings shared by both server and ↵Alberto Sartori
client
2023-02-02remote/config.hpp: make ParsePort re-usable for execution-serviceAlberto Sartori
2023-02-02bytestream_client: report failures on LogLevel::ErrorAlberto Sartori
2023-02-01Update fmtlib to version 9.1.0Klaus Aehlig
Co-authored-by: Michael Thies <mail@mhthies.de>
2023-01-31Improve error message if git tree walk failedOliver Reiche
2023-01-31Fix segfault if reading git tree failedOliver Reiche
2023-01-26Fix boostrapping on less optimizing compilersKlaus T. Aehlig
While compilers are allowed to drop unused functions in anonymous name spaces, and in this way also the open linker symbols referenced there, they are not obliged to do so. Not optimizing away such unused functions when compiled with -DBOOTSTRAP_BUILD_TOOL causes the linking fail in the initial phase of the boostrap process where libgit2 is not yet available (nor really needed). Therefore, ensure that those dead functions are absent in the initial bootstrap phase using appropriate preprocessor directives. Signed-off-by: Klaus T. Aehlig <aehlig@linta.de>
2023-01-24BazelMsgFactory: Improve error reporting for creating treesOliver Reiche
2023-01-24FileSystemManager: Do not follow symlinksOliver Reiche
... and ensure that cascades of checks are performed with only a single filesystem stat per method.
2023-01-24FileSystemManager: Pass ReadDirectory error to callerOliver Reiche
2023-01-24Just-MR: Remove obsolete GET_BRANCH_REFNAME critical Git operationPaul Cristian Sarbu
As now all remote Git operations in GitRepo require at most just the branch name, there is no more need to inquire the repository about branch refspecs.
generated by cgit v1.2.3 (git 2.39.1) at 2025-07-16 04:11:47 +0000