| Age | Commit message (Collapse) | Author |
|---|
|
|
|
In order to stay backwards compatible, the "tree_overlays" entry
in action-graph descriptions is optional.
|
|
|
|
|
|
Requires the use of a pragma to avoid wrong removal suggestion for
path_hash.hpp.
Co-authored-by: Maksim Denisov <denisov.maksim@huawei.com>
|
|
|
|
... instead of relying on those dependencies being pulled in
indirectly.
|
|
|
|
|
|
...with ArtifactDigestFactory::HashDataAs
|
|
... and, in particular, do not include headers of other libraries
that are not even needed.
|
|
...to track changes during refactoring easier.
|
|
|
|
Commit f5f9be5bc07b16807aceac86fba9212e3889762a changed from
tracking progress by cache key to tracking by configured target;
however, the absent-target map was forgotten. Fix this while also
switching to the shortend export-target representation introduced
in the previous commit.
|
|
As we always analyse export targets in their canonical configuration (and
do count in the progress the mapping from obtained configuraiton
to canonical one), the shortend name (i.e., the representation
with null values dropped form the configuration) is still a unique
representation of the target. Use this in progress reporting to
simplify reading the progress sample.
|
|
... with only the non-null entries of the configuration. This
information is enough for the user to build this target, e.g.,
when searching for the cause of a build failure.
|
|
For an absent export target, the first step of analysis is to ask
serve for the flexible variables. The answer to this request is,
however, independent of the configuration for this target. So we
can avoid calls by caching the answer in an additional map.
|
|
...by allowing a Logger instance to be provided.
|
|
Some of the more specific issues addressed:
- missing log_level target/include
- header-only libs wrongly marking deps as private
- missing/misplaced gsl includes
|
|
During analysis it is useful to track and report the progress for
all export targets. This is not exclusively linked to a serve
endpoint being present, despite most of the time being expected to
be spent in export targets being served from the remote endpoint.
This commit refactors the current implementation to give proper
feedback to the user on the progress of the analysis phase.
|
|
...with regular instances that have controlled life-times.
This avoids race conditions in tracking and reporting the results
of analysis and build, as the serve endpoint can orchestrate
multiple builds at the same time asynchronously. As a bonus
side-effect this also ensures the correctness of the progress
reporting per orchestrated build.
|
|
The serve endpoint always has to access the correctly sharded
target cache, including during analysis. For this purpose, the
target cache instance interrogated during analysis has to be
explicitly provided.
|
|
|
|
|
|
With the introduction of 'just serve', export targets can now be
built also independently from one another based on their
corresponding minimal repository configuration, as stored in the
target cache key.
In this context, this commit changes the RepositoryConfig usage
from one global (static) instance to pointers passed as necessary
throughout the code.
|
|
|
|
...and make the use of std::hash consistent.
This will make it easier to remove the fix once the libc
implementation we use catches up with the C++ standard.
|
|
... with two minor code base changes compared to previous
use of gsl-lite:
- dag.hpp: ActionNode::Ptr and ArtifactNode::Ptr are not
wrapped in gsl::not_null<> anymore, due to lack of support
for wrapping std::unique_ptr<>. More specifically, the
move constructor is missing, rendering it impossible to
use std::vector<>::emplace_back().
- utils/cpp/gsl.hpp: New header file added to implement the
macros ExpectsAudit() and EnsureAudit(), asserts running
only in debug builds, which were available in gsl-lite but
are missing in MS GSL.
|
|
The improved GC implementation uses refactored storage
classes instead of directly accessing "unknown" file paths.
The required storage class refactoring is quite substantial
and outlined in the following paragraphs.
The module `buildtool/file_system` was extended by:
- `ObjectCAS`: a plain CAS implementation for
reading/writing blobs and computing digests for a given
`ObjectType`. Depending on that type, files written to the
file system may have different properties (e.g., the x-bit
set) or the digest may be computed differently (e.g., tree
digests in non-compatible mode).
A new module `buildtool/storage` was introduced containing:
- `LocalCAS`: provides a common interface for the "logical
CAS", which internally combines three `ObjectCAS`s, one
for each `ObjectType` (file, executable, tree).
- `LocalAC`: implements the action cache, which needs the
`LocalCAS` for storing cache values.
- `TargetCache`: implements the high-level target cache,
which also needs the `LocalCAS` for storing cache values.
- `LocalStorage`: combines the storage classes `LocalCAS`,
`LocalAC`, and `TargetCache`. Those are initialized with
settings from `StorageConfig`, such as the build root base
path or number of generations for the garbage collector.
`LocalStorage` is templated with a Boolean parameter
`kDoGlobalUplink`, which indicates that, on every
read/write access, the garbage collector should be used
for uplinking across all generations (global).
- `GarbageCollector`: responsible for garbage collection and
the global uplinking across all generations. To do so, it
employs instances of `LocalStorage` with `kDoGlobalUplink`
set to false, in order to avoid endless recursion. The
actual (local) uplinking within two single generations is
performed by the corresponding storage class (e.g.,
`TargetCache` implements uplinking of target cache entries
between two target cache generations etc.). Thereby, the
actual knowledge how data should be uplinked is
implemented by the instance that is responsible for
creating the data in the first place.
|
|
|
|
|
|
|
|
|
|
This code movement is required to break a cyclic dependency coming with the
introduction of the garbage collector. target_cache depends on
garbage_collector and garbage_collector would depend on target_cache to
determine the target-level-cache directory. After moving this calculation to a
more general location, the cycle is broken.
|
|
This usage of a different header file is required to break a cyclic dependency
coming with the introduction of the garbage collector. entity_name implicitly
depends on local_cas, which will get a new dependency to garbage_collector.
garbage_collector depends on target_cache_entry, which transitively depends on
entity_name. entity_name_data instead does not depend on local_cas and breaks
the cycle.
|
|
|
|
|
|
While there, also add all direct dependencies explicitly; using
directly dependencies that are pulled in only indireclty causes
problems from a maintainability point of view.
|
|
... as this is relevant for performance of analysis. We log the total
numer of trees at performance level and the individual directories
at debug level, if requested.
|
|
|
|
|
|
|
|
|
|
|
|
Not only trees, but also regular files can disallow paths reaching
into them. If we have a file at a/b then another file at a/b/c
is a staging conflict as well. Make our tool recognize this.
|
|
... as it will be required outside the target map.
|
|
For a user, an important information is to know which actions
are currently running and, more importantly, the target that
caused them. To do so, we need a bit of infrastructure.
- We have to keep track of begin and end of running actions,
as well as the order in which they were started. That has
to happen efficiently and in a thread-safe way.
- We have to compute and keep the origin map for actions,
even if we don't serialize the action graph.
|
|
|
|
This is the initial version of our tool that is able to
build itself. In can be bootstrapped by
./bin/bootstrap.py
Co-authored-by: Oliver Reiche <oliver.reiche@huawei.com>
Co-authored-by: Victor Moreno <victor.moreno1@huawei.com>
|
