| Age | Commit message (Collapse) | Author |
|---|
|
...due to limitations in SSH support in libgit2. In this case, we
simply execute 'git ls-remote <repo> [<branch>]' and then parse
the output. Remote interogation requires no local repository, so
it is an asynchronious operation by default.
|
|
...due to limited SSH support in libgit2. In order to allow the
fetches to still be parallel, we execute:
git fetch --no-auto-gc --no-write-fetch-head <repo> [<branch>]
This only fetches the packs without updating any refs, at the slight
cost of sometimes fetching some redundant information, which for our
purposes is practically a non-issue.
(If really needed, a 'git gc' call can be done eventually to try to
compact the fetched packs, although a save in disk space is not
actually guaranteed.)
|
|
|
|
|
|
|
|
|
|
|
|
To avoid unnecessary work, just-mr uses on-disk caches, including
for the mapping of a distdir to the corresponding git tree. This,
however, implies that the number of repositories that are actually
considered varies: in order to fetch a distdir repository, all
involved archives have to be fetched, but if we have a cache hit
none of them is even looked at.
So, in order to have a consistent reporting only count top-level
targets (i.e., the reachable repositories) in the statistics,
not the archives implicitly contained in a distdir, nor low-level
sub tasks. The actual fetch acitvity is shown separately by the
task tracker.
|
|
... by only taking the additional roots from the reachable
repositories, not also from the additional roots discovered
that way.
Co-authored-by: Paul Cristian Sarbu <paul.cristian.sarbu@huawei.com>
|
|
|
|
|
|
The improved GC implementation uses refactored storage
classes instead of directly accessing "unknown" file paths.
The required storage class refactoring is quite substantial
and outlined in the following paragraphs.
The module `buildtool/file_system` was extended by:
- `ObjectCAS`: a plain CAS implementation for
reading/writing blobs and computing digests for a given
`ObjectType`. Depending on that type, files written to the
file system may have different properties (e.g., the x-bit
set) or the digest may be computed differently (e.g., tree
digests in non-compatible mode).
A new module `buildtool/storage` was introduced containing:
- `LocalCAS`: provides a common interface for the "logical
CAS", which internally combines three `ObjectCAS`s, one
for each `ObjectType` (file, executable, tree).
- `LocalAC`: implements the action cache, which needs the
`LocalCAS` for storing cache values.
- `TargetCache`: implements the high-level target cache,
which also needs the `LocalCAS` for storing cache values.
- `LocalStorage`: combines the storage classes `LocalCAS`,
`LocalAC`, and `TargetCache`. Those are initialized with
settings from `StorageConfig`, such as the build root base
path or number of generations for the garbage collector.
`LocalStorage` is templated with a Boolean parameter
`kDoGlobalUplink`, which indicates that, on every
read/write access, the garbage collector should be used
for uplinking across all generations (global).
- `GarbageCollector`: responsible for garbage collection and
the global uplinking across all generations. To do so, it
employs instances of `LocalStorage` with `kDoGlobalUplink`
set to false, in order to avoid endless recursion. The
actual (local) uplinking within two single generations is
performed by the corresponding storage class (e.g.,
`TargetCache` implements uplinking of target cache entries
between two target cache generations etc.). Thereby, the
actual knowledge how data should be uplinked is
implemented by the instance that is responsible for
creating the data in the first place.
|
|
... on reporting the exec command; json encoding is already
quoting enough.
|
|
The forwarding of the log-limit argument is only useful, if
the value is different from the default.
|
|
A local launcher should be forwarded, if it is different to
the default launcher (which is ["env", "--"]); in particular,
an explictly empy launcher should be forwarded.
|
|
... and, in this way, avoid duplication of these constants that
might lead to inconsistencies between those two tools later.
|
|
... by reporting unsupported config values explicitly and fixing
inconsistencies.
|
|
|
|
|
|
|
|
... both with respect to just options and the man page specifications.
Option -L of just-mr was reassigned as a short name for
--local-launcher, matching its use in just. Its place is now
correctly held by the full name option --checkout-locations, as
per the section-1 man page.
|
|
|
|
Map for a workspace root given as a known git tree id which is
promised to be obtainable by executing a specified command.
|
|
Also update just-mr section-1 man page
|
|
Make clear what fields the defined boolean values refer to.
|
|
|
|
Reporting and counting should be done as early as possible, once
known that there is nothing cached.
|
|
|
|
|
|
|
|
|
|
commit update
Uses the SSL certification utility method to correctly set the
certification check options for the remote URL libgit2 calls.
Due to the fact that remote operations are done via a temporary
repository to allow concurrent work, the correct repository
configuration needs to be interrogated. Thankfully, libgit2
provides a thread safe config snapshot object to be used in
such scenarios.
Also updates the existing GitRepoRemote tests accordingly.
|
|
Due to the fact that the libgit2 library handles envariables and
gitconfig entries differently than git, we need to perform these
checks ourselves in order to be fully compliant with git.
This utility method returns the correct callback tat enables or
disables the SSL certificate verification step when interacting
with a remote URL via libgit2 calls.
|
|
|
|
It uses libcurl to parse URLs. The current version contains
methods needed for correctly handling the matching gitconfig
entries and the no_proxy-type envariable patterns.
|
|
When asked to setup, report the number of repositories as repositories
to "set up", not to "check out". Typically, we're not creating a
checkout of that repository in the sense that some directory would
contain the file strucutre of that repository in the file system.
Typically, fetch into our big git repo, or create an artifical
commit there to keep the resulting git tree there.
|
|
This includes also the setup-env command, as well as all ~just~
known subcommands that require a just-mr setup step.
Co-authored-by: Sascha Roloff <sascha.roloff@huawei.com>
|
|
Co-authored-by: Sascha Roloff <sascha.roloff@huawei.com>
|
|
Co-authored-by: Sascha Roloff <sascha.roloff@huawei.com>
|
|
In order to only report actual work, the name of the repository for which
work is actually done needs to be known to certain AsyncMaps used by just-mr.
Also, fetches not originating in a distdir repo checkout need to be counted
additionally, so an additional flag was needed for the content_cas_map.
|
|
|
|
|
|
|
|
When calling just from just-mr, all logging arguments are forwarded
as early arguments. If any log files are provided, an unconditional
--log-append is also prepended to ensure the contents of the log files
are not overwritten.
|
|
Both, just and the launcher just-mr are developped and released
together and hence share version numbers. Nevertheless, the tools
are used independently. In fact, just-mr has an option to specify
the just binary to be used. Hence it is useful that each tool is
aware of its own version number. As a launcher, just-mr is supposed
to forward all subcommands known to just, hence the just-mr command
to show the version information cannot be called "version"; we use
"mrversion" instead.
|
|
|
|
|
|
tracing
|
|
|
|
|
