| Age | Commit message (Collapse) | Author |
|---|
|
... by including the details of the parse error.
|
|
In the current libgit2 implementation, a fake repository wrapped
around an existing odb is being registered as owner the same way
as a normal repository object. Therefore, one has to guard both
the creation and destruction of the fake repository against all
other git operations that might access the internal cache during
this transfer of ownership.
|
|
...and enforce this through the build description.
|
|
... asking just to transfer everyting installed to the local CAS first.
|
|
|
|
... to avoid unnecessary downloads and hence speed things up.
|
|
|
|
|
|
Use a similar logic as for repository initialisation: first check
if tag has not already been created in another process, and only
then try creation; make more tries with more wait in between; only
retry if failure was due to internal locking.
|
|
|
|
As the initialisation of Git repositories is something that only
takes place once, we should check early and cheaply whether the
repository is already there before trying to initialize it.
If we do need to initilize a repo, we can afford more attempts and
longer wait times between tries to initalize if the failure to
initialize happens due to the internal Git locking mechanism.
|
|
|
|
Opening a repo should not check parent directories, only try to
open at given path.
|
|
|
|
Make capitalization consistent and log as debug whenever we revert
to shelling out to git.
|
|
|
|
...depending on the compiler and/or c++lib version, std::fread may
warn about unused return value triggering a compile error, due to our
compile flags.
|
|
In the rare cases that we need to shell out to git, let the user
configure what binary to use. Option resolves in the same way as
the just executable, including allowing it to be set via just-mrrc.
Updates all cases of shelling out to git (fetch and commit update).
Update just-mr and just-mrrc docs accordingly.
|
|
...to ensure the temporary directories exist before they are needed,
as expected by those methods. This way, the TmpDir class takes care
to also clean up after itself.
Also, pass the local launcher to the methods that now shell out.
|
|
...due to limitations in SSH support in libgit2. In this case, we
simply execute 'git ls-remote <repo> [<branch>]' and then parse
the output. Remote interogation requires no local repository, so
it is an asynchronious operation by default.
|
|
...due to limited SSH support in libgit2. In order to allow the
fetches to still be parallel, we execute:
git fetch --no-auto-gc --no-write-fetch-head <repo> [<branch>]
This only fetches the packs without updating any refs, at the slight
cost of sometimes fetching some redundant information, which for our
purposes is practically a non-issue.
(If really needed, a 'git gc' call can be done eventually to try to
compact the fetched packs, although a save in disk space is not
actually guaranteed.)
|
|
|
|
|
|
|
|
... so that, e.g., we can set the logging from an expression value.
|
|
|
|
|
|
To avoid unnecessary work, just-mr uses on-disk caches, including
for the mapping of a distdir to the corresponding git tree. This,
however, implies that the number of repositories that are actually
considered varies: in order to fetch a distdir repository, all
involved archives have to be fetched, but if we have a cache hit
none of them is even looked at.
So, in order to have a consistent reporting only count top-level
targets (i.e., the reachable repositories) in the statistics,
not the archives implicitly contained in a distdir, nor low-level
sub tasks. The actual fetch acitvity is shown separately by the
task tracker.
|
|
|
|
... by only taking the additional roots from the reachable
repositories, not also from the additional roots discovered
that way.
Co-authored-by: Paul Cristian Sarbu <paul.cristian.sarbu@huawei.com>
|
|
By default, messages at level "progress" and above are shown. So,
messages at that level should be useful for the user in the typical
case. For the count of export targets, this information is only useful
if the build involves export targets at all; the absence of export
targets is typically only used when debugging the build structure.
|
|
|
|
...bazel_types.hpp just requires the messages, not the grpc related
services and classes
|
|
...both buildbarn and just execute do not allow a direct upload to the
action cache. Moreover, our tool does not support it, anyway, meaning
it was dead code.
|
|
|
|
|
|
|
|
The improved GC implementation uses refactored storage
classes instead of directly accessing "unknown" file paths.
The required storage class refactoring is quite substantial
and outlined in the following paragraphs.
The module `buildtool/file_system` was extended by:
- `ObjectCAS`: a plain CAS implementation for
reading/writing blobs and computing digests for a given
`ObjectType`. Depending on that type, files written to the
file system may have different properties (e.g., the x-bit
set) or the digest may be computed differently (e.g., tree
digests in non-compatible mode).
A new module `buildtool/storage` was introduced containing:
- `LocalCAS`: provides a common interface for the "logical
CAS", which internally combines three `ObjectCAS`s, one
for each `ObjectType` (file, executable, tree).
- `LocalAC`: implements the action cache, which needs the
`LocalCAS` for storing cache values.
- `TargetCache`: implements the high-level target cache,
which also needs the `LocalCAS` for storing cache values.
- `LocalStorage`: combines the storage classes `LocalCAS`,
`LocalAC`, and `TargetCache`. Those are initialized with
settings from `StorageConfig`, such as the build root base
path or number of generations for the garbage collector.
`LocalStorage` is templated with a Boolean parameter
`kDoGlobalUplink`, which indicates that, on every
read/write access, the garbage collector should be used
for uplinking across all generations (global).
- `GarbageCollector`: responsible for garbage collection and
the global uplinking across all generations. To do so, it
employs instances of `LocalStorage` with `kDoGlobalUplink`
set to false, in order to avoid endless recursion. The
actual (local) uplinking within two single generations is
performed by the corresponding storage class (e.g.,
`TargetCache` implements uplinking of target cache entries
between two target cache generations etc.). Thereby, the
actual knowledge how data should be uplinked is
implemented by the instance that is responsible for
creating the data in the first place.
|
|
... as internally we do not do the Tree-dance. Instead, we
directly store the root Directory digest for ActionResult's
OutputDirectories.
|
|
|
|
|
|
|
|
... on reporting the exec command; json encoding is already
quoting enough.
|
|
|
|
The forwarding of the log-limit argument is only useful, if
the value is different from the default.
|
|
A local launcher should be forwarded, if it is different to
the default launcher (which is ["env", "--"]); in particular,
an explictly empy launcher should be forwarded.
|
|
... and, in this way, avoid duplication of these constants that
might lead to inconsistencies between those two tools later.
|
|
in the tree CAS.
Before this patch, after the checking in the tree cas, also the blob
cas was checked leading to a NOT-FOUND error.
|
|
google::longrunning::Operations::GetOperation
For each action that is executed, an entry is added to a shared thread
safe cache. Once the number of operations stored exceeds twice 2^n,
where n is given by the option --log-operations-threshold, at most 2^n
operations will be removed, in a FIFO scheme.
|
|
... by reporting unsupported config values explicitly and fixing
inconsistencies.
|
