summaryrefslogtreecommitdiff
path: root/src
AgeCommit message (Collapse)Author
2023-03-30Improve error messages on reading invalid target filesKlaus Aehlig
... by including the details of the parse error.
2023-03-30GitRepo: Guard fake repository odb wrappingPaul Cristian Sarbu
In the current libgit2 implementation, a fake repository wrapped around an existing odb is being registered as owner the same way as a normal repository object. Therefore, one has to guard both the creation and destruction of the fake repository against all other git operations that might access the internal cache during this transfer of ownership.
2023-03-24libgit2: Remove use of deprecated functions...Paul Cristian Sarbu
...and enforce this through the build description.
2023-03-24just install{,-cas}: add --remember optionKlaus Aehlig
... asking just to transfer everyting installed to the local CAS first.
2023-03-23just-mr: extend self-descriptionKlaus Aehlig
2023-03-23just install{,-cas}: offer local CAS as preferred CASKlaus Aehlig
... to avoid unnecessary downloads and hence speed things up.
2023-03-23remote api: honor alternative for retrieving pathsKlaus Aehlig
2023-03-23execution api: support retrieving from more than one APIKlaus Aehlig
2023-03-23GitRepo: Make tag creation operation more robustPaul Cristian Sarbu
Use a similar logic as for repository initialisation: first check if tag has not already been created in another process, and only then try creation; make more tries with more wait in between; only retry if failure was due to internal locking.
2023-03-23GitRepo: Add proper error message for keep tag operationPaul Cristian Sarbu
2023-03-23GitRepo: Make repository initialisation more robustPaul Cristian Sarbu
As the initialisation of Git repositories is something that only takes place once, we should check early and cheaply whether the repository is already there before trying to initialize it. If we do need to initilize a repo, we can afford more attempts and longer wait times between tries to initalize if the failure to initialize happens due to the internal Git locking mechanism.
2023-03-23GitRepo: Add proper error message for repository initialisationPaul Cristian Sarbu
2023-03-23GitRepo: Make repository path usage explicitPaul Cristian Sarbu
Opening a repo should not check parent directories, only try to open at given path.
2023-03-23targets: Fix deps structurePaul Cristian Sarbu
2023-03-23GitRepoRemote: Improve logging messagesPaul Cristian Sarbu
Make capitalization consistent and log as debug whenever we revert to shelling out to git.
2023-03-23GitRepoRemote: Fix missing string in log messagePaul Cristian Sarbu
2023-03-23curl_easy_handle: fix unused return value...Alberto Sartori
...depending on the compiler and/or c++lib version, std::fread may warn about unused return value triggering a compile error, due to our compile flags.
2023-03-23just-mr: Add git binary optionPaul Cristian Sarbu
In the rare cases that we need to shell out to git, let the user configure what binary to use. Option resolves in the same way as the just executable, including allowing it to be set via just-mrrc. Updates all cases of shelling out to git (fetch and commit update). Update just-mr and just-mrrc docs accordingly.
2023-03-23just-mr: Update calls to fetch and update operations...Paul Cristian Sarbu
...to ensure the temporary directories exist before they are needed, as expected by those methods. This way, the TmpDir class takes care to also clean up after itself. Also, pass the local launcher to the methods that now shell out.
2023-03-23just-mr: Shell out to system Git for update commit from SSH remote...Paul Cristian Sarbu
...due to limitations in SSH support in libgit2. In this case, we simply execute 'git ls-remote <repo> [<branch>]' and then parse the output. Remote interogation requires no local repository, so it is an asynchronious operation by default.
2023-03-22just-mr: Shell out to system Git for fetches over SSH...Paul Cristian Sarbu
...due to limited SSH support in libgit2. In order to allow the fetches to still be parallel, we execute: git fetch --no-auto-gc --no-write-fetch-head <repo> [<branch>] This only fetches the packs without updating any refs, at the slight cost of sometimes fetching some redundant information, which for our purposes is practically a non-issue. (If really needed, a 'git gc' call can be done eventually to try to compact the fetched packs, although a save in disk space is not actually guaranteed.)
2023-03-22Git operations: Remove obsolete method declarationPaul Cristian Sarbu
2023-03-22just-mr: support log options in rc fileKlaus Aehlig
2023-03-22just-mrrc: allow setting a default for the launcher the rc fileKlaus Aehlig
2023-03-21LogLevel: support conversion from floating-point numbersKlaus Aehlig
... so that, e.g., we can set the logging from an expression value.
2023-03-20just-mr: report command output if desired tree was not createdKlaus Aehlig
2023-03-20just-mr: Fix fetch cache conditionPaul Cristian Sarbu
2023-03-20just-mr: rework progress reporting and statisticsKlaus Aehlig
To avoid unnecessary work, just-mr uses on-disk caches, including for the mapping of a distdir to the corresponding git tree. This, however, implies that the number of repositories that are actually considered varies: in order to fetch a distdir repository, all involved archives have to be fetched, but if we have a cache hit none of them is even looked at. So, in order to have a consistent reporting only count top-level targets (i.e., the reachable repositories) in the statistics, not the archives implicitly contained in a distdir, nor low-level sub tasks. The actual fetch acitvity is shown separately by the task tracker.
2023-03-20task tracker: support sampling the sizeKlaus Aehlig
2023-03-20just-mr: avoid fetching unnecessary repositoriesKlaus Aehlig
... by only taking the additional roots from the reachable repositories, not also from the additional roots discovered that way. Co-authored-by: Paul Cristian Sarbu <paul.cristian.sarbu@huawei.com>
2023-03-20just: only report export targets if there are anyKlaus Aehlig
By default, messages at level "progress" and above are shown. So, messages at that level should be useful for the user in the typical case. For the count of export targets, this information is only useful if the build involves export targets at all; the absence of export targets is typically only used when debugging the build structure.
2023-03-15just-mr: follow symlinks to the rc-file, also on the default locationKlaus Aehlig
2023-03-15bazel api: fix include of proto files...Alberto Sartori
...bazel_types.hpp just requires the messages, not the grpc related services and classes
2023-03-15bazel client: remove BazelAcClient::UpdateActionResult...Alberto Sartori
...both buildbarn and just execute do not allow a direct upload to the action cache. Moreover, our tool does not support it, anyway, meaning it was dead code.
2023-03-15add missing ldflags -pthread and use -pthread consistentlyAlberto Sartori
2023-03-14GitRepo: Fix memory leak in keep tag operationPaul Cristian Sarbu
2023-03-14CurlUrlHandle: Fix memory leak in no_proxy string matching methodPaul Cristian Sarbu
2023-03-13Storage: Reworked storage and garbage collectionOliver Reiche
The improved GC implementation uses refactored storage classes instead of directly accessing "unknown" file paths. The required storage class refactoring is quite substantial and outlined in the following paragraphs. The module `buildtool/file_system` was extended by: - `ObjectCAS`: a plain CAS implementation for reading/writing blobs and computing digests for a given `ObjectType`. Depending on that type, files written to the file system may have different properties (e.g., the x-bit set) or the digest may be computed differently (e.g., tree digests in non-compatible mode). A new module `buildtool/storage` was introduced containing: - `LocalCAS`: provides a common interface for the "logical CAS", which internally combines three `ObjectCAS`s, one for each `ObjectType` (file, executable, tree). - `LocalAC`: implements the action cache, which needs the `LocalCAS` for storing cache values. - `TargetCache`: implements the high-level target cache, which also needs the `LocalCAS` for storing cache values. - `LocalStorage`: combines the storage classes `LocalCAS`, `LocalAC`, and `TargetCache`. Those are initialized with settings from `StorageConfig`, such as the build root base path or number of generations for the garbage collector. `LocalStorage` is templated with a Boolean parameter `kDoGlobalUplink`, which indicates that, on every read/write access, the garbage collector should be used for uplinking across all generations (global). - `GarbageCollector`: responsible for garbage collection and the global uplinking across all generations. To do so, it employs instances of `LocalStorage` with `kDoGlobalUplink` set to false, in order to avoid endless recursion. The actual (local) uplinking within two single generations is performed by the corresponding storage class (e.g., `TargetCache` implements uplinking of target cache entries between two target cache generations etc.). Thereby, the actual knowledge how data should be uplinked is implemented by the instance that is responsible for creating the data in the first place.
2023-03-13GC: Fix uplinking of ActionResults in compatibleOliver Reiche
... as internally we do not do the Tree-dance. Instead, we directly store the root Directory digest for ActionResult's OutputDirectories.
2023-03-13TargetCache: Drop dependency on execution apiOliver Reiche
2023-03-13TargetCache: Drop dependency on repository configOliver Reiche
2023-03-13Artifact: No user error on object info parse failOliver Reiche
2023-03-13just-mr: remove unnecessary double quotesKlaus T. Aehlig
... on reporting the exec command; json encoding is already quoting enough.
2023-03-10Update nlohmann::json to 3.11.2Klaus Aehlig
2023-03-10just-mr: do not unnecessarily forward log-limitKlaus Aehlig
The forwarding of the log-limit argument is only useful, if the value is different from the default.
2023-03-10just-mr: fix launcher forwardingKlaus Aehlig
A local launcher should be forwarded, if it is different to the default launcher (which is ["env", "--"]); in particular, an explictly empy launcher should be forwarded.
2023-03-10Move common defaults for just and just-mr into a separate libraryKlaus Aehlig
... and, in this way, avoid duplication of these constants that might lead to inconsistencies between those two tools later.
2023-03-10bytestream server: bugfix: if a tree is requested, look for it *only*Alberto Sartori
in the tree CAS. Before this patch, after the checking in the tree cas, also the blob cas was checked leading to a NOT-FOUND error.
2023-03-10execution service: implement WaitExecution and ↵Alberto Sartori
google::longrunning::Operations::GetOperation For each action that is executed, an entry is added to a shared thread safe cache. Once the number of operations stored exceeds twice 2^n, where n is given by the option --log-operations-threshold, at most 2^n operations will be removed, in a FIFO scheme.
2023-03-09Just-MR: Improve logging messagesPaul Cristian Sarbu
... by reporting unsupported config values explicitly and fixing inconsistencies.