| Age | Commit message (Collapse) | Author |
|---|
|
...due to limited SSH support in libgit2. In order to allow the
fetches to still be parallel, we execute:
git fetch --no-auto-gc --no-write-fetch-head <repo> [<branch>]
This only fetches the packs without updating any refs, at the slight
cost of sometimes fetching some redundant information, which for our
purposes is practically a non-issue.
(If really needed, a 'git gc' call can be done eventually to try to
compact the fetched packs, although a save in disk space is not
actually guaranteed.)
|
|
|
|
To avoid unnecessary work, just-mr uses on-disk caches, including
for the mapping of a distdir to the corresponding git tree. This,
however, implies that the number of repositories that are actually
considered varies: in order to fetch a distdir repository, all
involved archives have to be fetched, but if we have a cache hit
none of them is even looked at.
So, in order to have a consistent reporting only count top-level
targets (i.e., the reachable repositories) in the statistics,
not the archives implicitly contained in a distdir, nor low-level
sub tasks. The actual fetch acitvity is shown separately by the
task tracker.
|
|
The improved GC implementation uses refactored storage
classes instead of directly accessing "unknown" file paths.
The required storage class refactoring is quite substantial
and outlined in the following paragraphs.
The module `buildtool/file_system` was extended by:
- `ObjectCAS`: a plain CAS implementation for
reading/writing blobs and computing digests for a given
`ObjectType`. Depending on that type, files written to the
file system may have different properties (e.g., the x-bit
set) or the digest may be computed differently (e.g., tree
digests in non-compatible mode).
A new module `buildtool/storage` was introduced containing:
- `LocalCAS`: provides a common interface for the "logical
CAS", which internally combines three `ObjectCAS`s, one
for each `ObjectType` (file, executable, tree).
- `LocalAC`: implements the action cache, which needs the
`LocalCAS` for storing cache values.
- `TargetCache`: implements the high-level target cache,
which also needs the `LocalCAS` for storing cache values.
- `LocalStorage`: combines the storage classes `LocalCAS`,
`LocalAC`, and `TargetCache`. Those are initialized with
settings from `StorageConfig`, such as the build root base
path or number of generations for the garbage collector.
`LocalStorage` is templated with a Boolean parameter
`kDoGlobalUplink`, which indicates that, on every
read/write access, the garbage collector should be used
for uplinking across all generations (global).
- `GarbageCollector`: responsible for garbage collection and
the global uplinking across all generations. To do so, it
employs instances of `LocalStorage` with `kDoGlobalUplink`
set to false, in order to avoid endless recursion. The
actual (local) uplinking within two single generations is
performed by the corresponding storage class (e.g.,
`TargetCache` implements uplinking of target cache entries
between two target cache generations etc.). Thereby, the
actual knowledge how data should be uplinked is
implemented by the instance that is responsible for
creating the data in the first place.
|
|
|
|
|
|
Co-authored-by: Sascha Roloff <sascha.roloff@huawei.com>
|
|
Co-authored-by: Sascha Roloff <sascha.roloff@huawei.com>
|
|
In order to only report actual work, the name of the repository for which
work is actually done needs to be known to certain AsyncMaps used by just-mr.
Also, fetches not originating in a distdir repo checkout need to be counted
additionally, so an additional flag was needed for the content_cas_map.
|
|
...in order to not include unwanted dependencies in just proper.
The new class extends the GitRepo class used for just's Git tree
operations and gets used in all of just-mr's async maps.
|
|
...in order to not include unwanted dependencies in just proper.
As the whole other_tools folder is meant to be excluded from
bootstrapping, also remove the bootstrap guards.
|
|
As now all remote Git operations in GitRepo require at most just
the branch name, there is no more need to inquire the repository
about branch refspecs.
|
|
...and use instead the branch name. A valid direct refspec (as those
retrieved by a remote_ls call) will always end in the branch name,
so checking the last path component ('/'-delimited substring) of a
retrieved refspec is enough.
|
|
This also removes the need to call the GET_BRANCH_REFNAME critical
operation.
|
|
|
|
Both, the python script as well as the compiled variant of just-mr
fetch try to overwrite already existing distfiles in the distdir.
While it is unclear if overwriting existing distfiles or not
copying to distdir if a file with that name already exists is the
better behaviour, we at least should not error out, as the compiled
just-mr currently does as it sets permission to 444 and hence cannot
overwrite later; fix this by addwing owner write permissions before
overwriting.
|
|
Passing the logger by reference would require the caller to be
kept alive. Also, being a shared_ptr, the logger can be passed
by value at almost no cost.
|
|
|
|
Calls libcurl to fetch the archives from the internet.
|
|
|
|
|
|
|
