| Age | Commit message (Collapse) | Author |
|---|
|
... and prepare local execution for clients
using only RBEv2.1 (setting only output_paths).
|
|
|
|
- in sequence containers, use operator[] instead of .at() when
accessing indices guaranteed to be in bound;
- in associative containers, prefer .find() and reusing the
returned const iterator to using .contains() and .at(); while
there, make any so obtained iterators const if they are read-only.
|
|
|
|
|
|
|
|
|
|
... and rename appropriately to reflect contents more precisely
than the generic "common". This separation also disentangles
dependencies a bit.
|
|
|
|
...and private members using lower_case_
|
|
|
|
|
|
Invalid entries, currently all upwards symlinks (pending
implementation of a better way of handling them), are now
identified and handled similarly to remote execution: in compatible
mode on the client side, during handling of local response, and in
native mode during the population of the local action result.
|
|
|
|
...and move it to the common stage.
|
|
|
|
|
|
...to simplify further refactoring.
|
|
...with ArtifactDigest.
|
|
...with ArtifactDigest.
|
|
... following the remote-execution standard that all output paths (but
none of the input paths) are relative to the working directory.
Therefore, the executor has to do the path translation. For our
implementation of the API interface
- the local API now handles cwd correctly,
- the remote API forwards cwd correctly, and
- the git API continues to report actions as not implemented.
|
|
The context is passed by not_null const pointer to avoid binding to
temporaries. The LocalApi also stores the context as const ref for
further access and passing it to LocalAction.
|
|
...and replace it with passed instances created early via a builder
pattern.
|
|
|
|
|
|
...instead of dereferencing nullptr.
|
|
... and continue with the newly created copy as target for the next
hard links. In this way, we get rid of the restriction we used to
have that the number of identical inputs be not greater than the
hardlink limit.
|
|
|
|
...to track changes during refactoring easier.
|
|
|
|
...in LocalApi and BazelApi.
|
|
Main culprits:
- std::size_t, std::nullptr_t, and NULL require <cstddef>
- std::move and std::forward require <utility>
- unordered maps and sets require respective includes
- std::for_each and std::all_of require <algorithm>
|
|
Some of the more specific issues addressed:
- missing log_level target/include
- header-only libs wrongly marking deps as private
- missing/misplaced gsl includes
|
|
|
|
|
|
... which are only actions that, besides giving exit code 0 also
created all the outputs they promised to.
|
|
Upwards symlinks should still be collected from actions, even if
only the non-upwards symlinks are supported artifact types. The
client side is thus the one responsible with enforcing the
non-upwardness condition.
|
|
Before this patch, when creating an action directory, symlinks were
staged as regular files.
|
|
|
|
|
|
As local execution is tightly coupled to storage, also specify the
layout in the storage configuration. In this way, we have a central
place specifying the layout of just's cache directory and avoid
accidentally getting into conflicting situations. While there, also
move the execution root under the generation regime, to ensure that
left-over execution directories (e.g., after a forceful termination
of the program) eventually get cleaned up by garbage collection.
|
|
... with two minor code base changes compared to previous
use of gsl-lite:
- dag.hpp: ActionNode::Ptr and ArtifactNode::Ptr are not
wrapped in gsl::not_null<> anymore, due to lack of support
for wrapping std::unique_ptr<>. More specifically, the
move constructor is missing, rendering it impossible to
use std::vector<>::emplace_back().
- utils/cpp/gsl.hpp: New header file added to implement the
macros ExpectsAudit() and EnsureAudit(), asserts running
only in debug builds, which were available in gsl-lite but
are missing in MS GSL.
|
|
The improved GC implementation uses refactored storage
classes instead of directly accessing "unknown" file paths.
The required storage class refactoring is quite substantial
and outlined in the following paragraphs.
The module `buildtool/file_system` was extended by:
- `ObjectCAS`: a plain CAS implementation for
reading/writing blobs and computing digests for a given
`ObjectType`. Depending on that type, files written to the
file system may have different properties (e.g., the x-bit
set) or the digest may be computed differently (e.g., tree
digests in non-compatible mode).
A new module `buildtool/storage` was introduced containing:
- `LocalCAS`: provides a common interface for the "logical
CAS", which internally combines three `ObjectCAS`s, one
for each `ObjectType` (file, executable, tree).
- `LocalAC`: implements the action cache, which needs the
`LocalCAS` for storing cache values.
- `TargetCache`: implements the high-level target cache,
which also needs the `LocalCAS` for storing cache values.
- `LocalStorage`: combines the storage classes `LocalCAS`,
`LocalAC`, and `TargetCache`. Those are initialized with
settings from `StorageConfig`, such as the build root base
path or number of generations for the garbage collector.
`LocalStorage` is templated with a Boolean parameter
`kDoGlobalUplink`, which indicates that, on every
read/write access, the garbage collector should be used
for uplinking across all generations (global).
- `GarbageCollector`: responsible for garbage collection and
the global uplinking across all generations. To do so, it
employs instances of `LocalStorage` with `kDoGlobalUplink`
set to false, in order to avoid endless recursion. The
actual (local) uplinking within two single generations is
performed by the corresponding storage class (e.g.,
`TargetCache` implements uplinking of target cache entries
between two target cache generations etc.). Thereby, the
actual knowledge how data should be uplinked is
implemented by the instance that is responsible for
creating the data in the first place.
|
|
|
|
|
|
|
|
Signed-off-by: Goetz Brasche <goetz.brasche@huawei.com>
|
|
|
|
... to align with the original idea of caching a flat list
of blob objects, without the need to recursively traverse
any trees. Consequently, we cannot create any map entry in
places where we do not have all sub-tree entries at hand
(e.g., LocalAPI, BazelAPI, BazelResponse).
|
|
|
