| Age | Commit message (Collapse) | Author |
|---|
|
|
|
|
|
|
|
There is no need to use the full format API, so avoid increasing
the compile time gratuitously.
|
|
... which are only actions that, besides giving exit code 0 also
created all the outputs they promised to.
|
|
... in order to make that functionality reusable
|
|
|
|
Upwards symlinks should still be collected from actions, even if
only the non-upwards symlinks are supported artifact types. The
client side is thus the one responsible with enforcing the
non-upwardness condition.
|
|
|
|
|
|
Before this patch, when creating an action directory, symlinks were
staged as regular files.
|
|
|
|
The bytestream server implementation (deployed by just execute) now
stores the temporary files under
$local_build_root/protocol-depenedent/generation-0
so that they can be garbage collected if "just exectue" is terminated
before they are cleaned up.
|
|
Extend the configuration data structure by a dispatch list of endpoints
to chose based on the first match of the execution properties.
|
|
... in a way consistent with what we're using at other places
already.
|
|
|
|
A recent change made it so that the artifact type was wrongly being
taken into account when deciding whether the CAS entry of the
artifact would be read or not. This meant that non-file artifacts
would be skipped from their content being read, resulting in
missing CAS artifacts being reported.
This fixes the issue by reverting the offending change.
|
|
In preparation for the introduction of our blob splitting protocol as extension
to the remote execution api, we need to update the used remote execution api to
a more recent version than v2.0.0. Since no new tags are available right now,
we update to the preliminary protocol version v2.3 according to the following
discussion: https://github.com/bazelbuild/remote-apis/issues/253
|
|
|
|
|
|
|
|
|
|
|
|
...as early as possible. This ensures that callers always receive
only the tree entries for the supported object types.
For the symlinks non-upwardness check we pass a lambda capturing
the real backend of the tree entries, such that the symlinks can
be read.
Updates git_tree tests accordingly.
|
|
This feature has been introduced with C++20.
|
|
As local execution is tightly coupled to storage, also specify the
layout in the storage configuration. In this way, we have a central
place specifying the layout of just's cache directory and avoid
accidentally getting into conflicting situations. While there, also
move the execution root under the generation regime, to ensure that
left-over execution directories (e.g., after a forceful termination
of the program) eventually get cleaned up by garbage collection.
|
|
|
|
... with two minor code base changes compared to previous
use of gsl-lite:
- dag.hpp: ActionNode::Ptr and ArtifactNode::Ptr are not
wrapped in gsl::not_null<> anymore, due to lack of support
for wrapping std::unique_ptr<>. More specifically, the
move constructor is missing, rendering it impossible to
use std::vector<>::emplace_back().
- utils/cpp/gsl.hpp: New header file added to implement the
macros ExpectsAudit() and EnsureAudit(), asserts running
only in debug builds, which were available in gsl-lite but
are missing in MS GSL.
|
|
|
|
|
|
From a git CAS crate an execution API unable to execute
actions or to store anything. This implementaiton of the
common interface is still useful as the execution API
is the interface used for tranfering artifacts.
|
|
As those functions indicate success, it is up to the caller to
decide if the error was fatal or not. Reporting an error nevertheless
might result in error messages on successful operaitons, which is
confusing for the user.
|
|
- deduplicate dependencies
- remove unused dependency
|
|
|
|
|
|
|
|
...bazel_types.hpp just requires the messages, not the grpc related
services and classes
|
|
...both buildbarn and just execute do not allow a direct upload to the
action cache. Moreover, our tool does not support it, anyway, meaning
it was dead code.
|
|
The improved GC implementation uses refactored storage
classes instead of directly accessing "unknown" file paths.
The required storage class refactoring is quite substantial
and outlined in the following paragraphs.
The module `buildtool/file_system` was extended by:
- `ObjectCAS`: a plain CAS implementation for
reading/writing blobs and computing digests for a given
`ObjectType`. Depending on that type, files written to the
file system may have different properties (e.g., the x-bit
set) or the digest may be computed differently (e.g., tree
digests in non-compatible mode).
A new module `buildtool/storage` was introduced containing:
- `LocalCAS`: provides a common interface for the "logical
CAS", which internally combines three `ObjectCAS`s, one
for each `ObjectType` (file, executable, tree).
- `LocalAC`: implements the action cache, which needs the
`LocalCAS` for storing cache values.
- `TargetCache`: implements the high-level target cache,
which also needs the `LocalCAS` for storing cache values.
- `LocalStorage`: combines the storage classes `LocalCAS`,
`LocalAC`, and `TargetCache`. Those are initialized with
settings from `StorageConfig`, such as the build root base
path or number of generations for the garbage collector.
`LocalStorage` is templated with a Boolean parameter
`kDoGlobalUplink`, which indicates that, on every
read/write access, the garbage collector should be used
for uplinking across all generations (global).
- `GarbageCollector`: responsible for garbage collection and
the global uplinking across all generations. To do so, it
employs instances of `LocalStorage` with `kDoGlobalUplink`
set to false, in order to avoid endless recursion. The
actual (local) uplinking within two single generations is
performed by the corresponding storage class (e.g.,
`TargetCache` implements uplinking of target cache entries
between two target cache generations etc.). Thereby, the
actual knowledge how data should be uplinked is
implemented by the instance that is responsible for
creating the data in the first place.
|
|
... as internally we do not do the Tree-dance. Instead, we
directly store the root Directory digest for ActionResult's
OutputDirectories.
|
|
in the tree CAS.
Before this patch, after the checking in the tree cas, also the blob
cas was checked leading to a NOT-FOUND error.
|
|
google::longrunning::Operations::GetOperation
For each action that is executed, an entry is added to a shared thread
safe cache. Once the number of operations stored exceeds twice 2^n,
where n is given by the option --log-operations-threshold, at most 2^n
operations will be removed, in a FIFO scheme.
|
|
|
|
|
|
|
|
Co-authored-by: Sascha Roloff <sascha.roloff@huawei.com>
|
|
|
|
... by keeping track of each blob being a file or executable
and storing it to the correct local physical CAS directory.
The new flag is merely a hint and only used by the local
execution API. Leaving it out will still correctly transfer
the blob but may cause unnecessary duplicates in file CAS.
|
|
|
|
|
