| Age | Commit message (Collapse) | Author |
|---|
|
|
|
|
|
IExecutionApi::UploadFile was introduced to handle adding of a temporary file to the storage. The call was redirected to LocalCAS internal methods that do the similar thing, so this virtual method can be removed.
|
|
Update logic populating containers to use the new method which
is aware of the maximum transfer limit.
|
|
...to reduce the "price" of copying.
|
|
...instead of BazelBlobContainer to not bring bazel_re::Digest to IExecutionApi.
|
|
|
|
|
|
...instead of various iterators.
|
|
...in LocalApi and BazelApi.
|
|
...in LocalApi and BazelApi.
|
|
This reduces the code duplication between the local and bazel APIs
and improves code maintainability.
|
|
Once a RepositoryConfig instance gets populated, it must never be
changed again. Therefore, all functions accepting these instances
should only take them as pointers to const.
|
|
|
|
Main culprits:
- std::size_t, std::nullptr_t, and NULL require <cstddef>
- std::move and std::forward require <utility>
- unordered maps and sets require respective includes
- std::for_each and std::all_of require <algorithm>
|
|
Some of the more specific issues addressed:
- missing log_level target/include
- header-only libs wrongly marking deps as private
- missing/misplaced gsl includes
|
|
|
|
|
|
|
|
Currently, the implementations of the split and splice operation are both
hidden behind the Bazel API implementation. This was sufficient to implement
splitting at the server and splicing at the client. In order to support the
other direction of splitting at the client and splicing at the server while
reusing their implementations, the code needs to be refactored. First, the
functionality of split and splice are explicitly exposed at the general
execution API interface and implemented in the sub APIs. Second, the
implementations of split and splice are factored into a separate utils class.
|
|
|
|
...in accordance to our coding style.
|
|
... and improve log messages in case of failure.
|
|
|
|
... overriding the default implementation. In this way, files can
be added directly to the local CAS without having to completely
reside in memory.
|
|
With the introduction of 'just serve', export targets can now be
built also independently from one another based on their
corresponding minimal repository configuration, as stored in the
target cache key.
In this context, this commit changes the RepositoryConfig usage
from one global (static) instance to pointers passed as necessary
throughout the code.
|
|
|
|
|
|
... which are only actions that, besides giving exit code 0 also
created all the outputs they promised to.
|
|
Upwards symlinks should still be collected from actions, even if
only the non-upwards symlinks are supported artifact types. The
client side is thus the one responsible with enforcing the
non-upwardness condition.
|
|
|
|
Before this patch, when creating an action directory, symlinks were
staged as regular files.
|
|
A recent change made it so that the artifact type was wrongly being
taken into account when deciding whether the CAS entry of the
artifact would be read or not. This meant that non-file artifacts
would be skipped from their content being read, resulting in
missing CAS artifacts being reported.
This fixes the issue by reverting the offending change.
|
|
In preparation for the introduction of our blob splitting protocol as extension
to the remote execution api, we need to update the used remote execution api to
a more recent version than v2.0.0. Since no new tags are available right now,
we update to the preliminary protocol version v2.3 according to the following
discussion: https://github.com/bazelbuild/remote-apis/issues/253
|
|
|
|
|
|
|
|
This feature has been introduced with C++20.
|
|
As local execution is tightly coupled to storage, also specify the
layout in the storage configuration. In this way, we have a central
place specifying the layout of just's cache directory and avoid
accidentally getting into conflicting situations. While there, also
move the execution root under the generation regime, to ensure that
left-over execution directories (e.g., after a forceful termination
of the program) eventually get cleaned up by garbage collection.
|
|
... with two minor code base changes compared to previous
use of gsl-lite:
- dag.hpp: ActionNode::Ptr and ArtifactNode::Ptr are not
wrapped in gsl::not_null<> anymore, due to lack of support
for wrapping std::unique_ptr<>. More specifically, the
move constructor is missing, rendering it impossible to
use std::vector<>::emplace_back().
- utils/cpp/gsl.hpp: New header file added to implement the
macros ExpectsAudit() and EnsureAudit(), asserts running
only in debug builds, which were available in gsl-lite but
are missing in MS GSL.
|
|
|
|
As those functions indicate success, it is up to the caller to
decide if the error was fatal or not. Reporting an error nevertheless
might result in error messages on successful operaitons, which is
confusing for the user.
|
|
- deduplicate dependencies
- remove unused dependency
|
|
|
|
|
|
The improved GC implementation uses refactored storage
classes instead of directly accessing "unknown" file paths.
The required storage class refactoring is quite substantial
and outlined in the following paragraphs.
The module `buildtool/file_system` was extended by:
- `ObjectCAS`: a plain CAS implementation for
reading/writing blobs and computing digests for a given
`ObjectType`. Depending on that type, files written to the
file system may have different properties (e.g., the x-bit
set) or the digest may be computed differently (e.g., tree
digests in non-compatible mode).
A new module `buildtool/storage` was introduced containing:
- `LocalCAS`: provides a common interface for the "logical
CAS", which internally combines three `ObjectCAS`s, one
for each `ObjectType` (file, executable, tree).
- `LocalAC`: implements the action cache, which needs the
`LocalCAS` for storing cache values.
- `TargetCache`: implements the high-level target cache,
which also needs the `LocalCAS` for storing cache values.
- `LocalStorage`: combines the storage classes `LocalCAS`,
`LocalAC`, and `TargetCache`. Those are initialized with
settings from `StorageConfig`, such as the build root base
path or number of generations for the garbage collector.
`LocalStorage` is templated with a Boolean parameter
`kDoGlobalUplink`, which indicates that, on every
read/write access, the garbage collector should be used
for uplinking across all generations (global).
- `GarbageCollector`: responsible for garbage collection and
the global uplinking across all generations. To do so, it
employs instances of `LocalStorage` with `kDoGlobalUplink`
set to false, in order to avoid endless recursion. The
actual (local) uplinking within two single generations is
performed by the corresponding storage class (e.g.,
`TargetCache` implements uplinking of target cache entries
between two target cache generations etc.). Thereby, the
actual knowledge how data should be uplinked is
implemented by the instance that is responsible for
creating the data in the first place.
|
|
... as internally we do not do the Tree-dance. Instead, we
directly store the root Directory digest for ActionResult's
OutputDirectories.
|
|
... by keeping track of each blob being a file or executable
and storing it to the correct local physical CAS directory.
The new flag is merely a hint and only used by the local
execution API. Leaving it out will still correctly transfer
the blob but may cause unnecessary duplicates in file CAS.
|
|
|
|
|
