| Age | Commit message (Collapse) | Author |
|---|
|
|
|
|
|
...instead of EnsureAbsentRootOnServe.
And remove unused arguments.
|
|
|
|
|
|
|
|
...besides the simple Git hash, if syncing was done. This way one
can know what digest to ask for from the remote. The serve client
also needs to now know what hash function the remote expects.
The serve service proto file is updated accordingly.
|
|
In just-mr: to instantiate the new Git api instance, both storage
configs, as well as the compatible storage, need to be passed to
the maps. While there, use more explicit naming schemes for the
storage and CAS instances used.
In serve: also acquire gc locks for the local storages when needed
to instantiate the new Git api, which now has access to the CAS.
In all these instances we also pass, as needed, the local api, which
currently still operates only in native mode. This makes no
difference currently, but will ensure less changes needed when the
future compatible-aware local api will be used instead.
|
|
|
|
...to create ArtifactDigests.
|
|
While there, ensure optional Git operation parameters are checked
before use for the operations that require them.
|
|
The 'branch' field is deprecated, not being used by any of the
critical Git operations, thus it can be removed.
|
|
Instead of returning a plain boolean, return an expected with the same
boolean value that in case of an error indicates the error code. In this
way, an error-specific handling is possible by consumers. While there,
also add proper quoting of the involved file names.
|
|
|
|
|
|
...to track changes during refactoring easier.
|
|
|
|
...since it is used for capturing in lambdas for AsyncMaps and mustn't be temporary.
|
|
...instead of std::optional<gsl::not_null<IExecutionApi const*>>
|
|
...and replace verbose constructions.
|
|
|
|
...instead of not_null const ptr.
|
|
...and adjust interfaces.
|
|
...instead of using singleton calls.
|
|
...to track changes during refactoring easier.
|
|
The gsl-lite implementation is slightly more picky in terms of
type conversions and constness resolution in initializers,
therefore small changes were needed.
|
|
... as the fs_utils have a lot more dependencies making them usable
in less places. Moreover, this function also serves to shape the
layout of the local build root and hence is more appropriately
placed in the config anyway.
|
|
Asking the remote-exeuction endpoint for the tree means, that,
upon success, we fetch every archive from the remote execution
endpoint, even those that we could get from a provided distdir. As
our preference is to only use the network if necessary, simply use
the standard way of creating a distdir: create it locally and for
each file, try to get it from local resources first, then from the
remote-execution endpoint, and finally from other network resources.
|
|
In order to set up roots, just-mr is able to interrogate, if given,
serve and/or remote-execution endpoints. However, just-mr operates
only with Git hashes, i.e., with a native mode CAS.
This commit ensures the correct interactions occur between just-mr
and the provided endpoints not only in native mode, but also in
comaptible mode, where a serve endpoint might be present even if
one cannot make use of its associated remote-exection endpoint.
The user always gets informed if any incompatibilities are
detected.
|
|
As the serve service always has an associated remote-execution
endpoint, when just-mr gets passed only a serve endpoint address it
will now default to implying a remote-execution endpoint exists too
at that address. Additionally, now the implementation of the
--backup-to-remote option more clearly shows that it can only be
done in native mode.
|
|
The requests to retrieve the tree of a commit, archive, or distdir
also set up those trees in a way that the serve endpoint can later
build against them, besides allowing just-mr to set up roots
locally. Therefore, if the witnessing entity (Git commit, content
blob, or distdir, respectively) is known to the serve endpoint,
then failing to set up the root tree there should result in a
failure also of the just-mr setup on the client side.
|
|
To take advantage of absent roots, we need to ensure that a given
serve endpoint can build against the tree of this generated root.
For a 'distdir' repository we can know the resulting tree
identifier directly without actually needing to fetch anything.
Therefore, we only set the root as absent if the serve endpoint
knows already this tree, if it can set it up itself, or if
we can provide this tree to the serve endpoint from one of our
CAS locations (local or remote), based on our tree invariant
guarantee. A network fetch of the archives never gets performed
for an absent root.
If a serve endpoint is not provided, an absent root can still be
generated, but only if no network fetches are required. In this
case a warning is emitted.
|
|
This is to uphold the coding style guide we employ.
|
|
If root is marked absent and we're not asked to fetch absent, we
can compute the resulting distdir root tree in-memory, as we have
all the information. Otherwise, i.e., if we actually need to have
the archives locally, we first check if the remote serve can
provide them in the remote CAS before continuing as before to fetch
the listed archives one at a time.
|
|
This fixes the multiple instances of (mainly) missing or wrong
includes of standard, third-party, and own headers in the
other_tools source folder.
|
|
This is required in order to make them available to 'just serve'
in a minimal just installation.
|
|
...via an 'absent' pragma in repository descriptions.
For 'git'-type repositories, first interrogates a 'just serve'
remote, if given, before reverting to fetching from the network.
|
|
...instead of a regular field.
Also removes the ignore-special logic from distdir repositories,
as those will never contain symlinks, so no entries would be
ignored.
|
|
|
|
This feature has been introduced with C++20.
|
|
|
|
In this way, we have the whole layout of the local build root
consolidated in one place. Moreover, in this way, the location
of the git root is also available to the build tool itself and
can, e.g., be used as fallback CAS.
|
|
To avoid unnecessary work, just-mr uses on-disk caches, including
for the mapping of a distdir to the corresponding git tree. This,
however, implies that the number of repositories that are actually
considered varies: in order to fetch a distdir repository, all
involved archives have to be fetched, but if we have a cache hit
none of them is even looked at.
So, in order to have a consistent reporting only count top-level
targets (i.e., the reachable repositories) in the statistics,
not the archives implicitly contained in a distdir, nor low-level
sub tasks. The actual fetch acitvity is shown separately by the
task tracker.
|
|
The improved GC implementation uses refactored storage
classes instead of directly accessing "unknown" file paths.
The required storage class refactoring is quite substantial
and outlined in the following paragraphs.
The module `buildtool/file_system` was extended by:
- `ObjectCAS`: a plain CAS implementation for
reading/writing blobs and computing digests for a given
`ObjectType`. Depending on that type, files written to the
file system may have different properties (e.g., the x-bit
set) or the digest may be computed differently (e.g., tree
digests in non-compatible mode).
A new module `buildtool/storage` was introduced containing:
- `LocalCAS`: provides a common interface for the "logical
CAS", which internally combines three `ObjectCAS`s, one
for each `ObjectType` (file, executable, tree).
- `LocalAC`: implements the action cache, which needs the
`LocalCAS` for storing cache values.
- `TargetCache`: implements the high-level target cache,
which also needs the `LocalCAS` for storing cache values.
- `LocalStorage`: combines the storage classes `LocalCAS`,
`LocalAC`, and `TargetCache`. Those are initialized with
settings from `StorageConfig`, such as the build root base
path or number of generations for the garbage collector.
`LocalStorage` is templated with a Boolean parameter
`kDoGlobalUplink`, which indicates that, on every
read/write access, the garbage collector should be used
for uplinking across all generations (global).
- `GarbageCollector`: responsible for garbage collection and
the global uplinking across all generations. To do so, it
employs instances of `LocalStorage` with `kDoGlobalUplink`
set to false, in order to avoid endless recursion. The
actual (local) uplinking within two single generations is
performed by the corresponding storage class (e.g.,
`TargetCache` implements uplinking of target cache entries
between two target cache generations etc.). Thereby, the
actual knowledge how data should be uplinked is
implemented by the instance that is responsible for
creating the data in the first place.
|
|
Reporting and counting should be done as early as possible, once
known that there is nothing cached.
|
|
This includes also the setup-env command, as well as all ~just~
known subcommands that require a just-mr setup step.
Co-authored-by: Sascha Roloff <sascha.roloff@huawei.com>
|
|
|
|
This provides the fix already performed for the just-mr script in
the internal just-mr as well. In short, this fix makes sure that the
computation of the content key for distdir repositories is
independent of the presence of the respective archives in CAS.
|
|
|
