| Age | Commit message (Collapse) | Author |
|---|
|
... and set default stack size to 8 MB.
(cherry picked from commit 9e45d1525b0fde12a08f7c376b5bf2c7bfbc9803)
|
|
...caused by incorrectly setting and resetting the library internal
state and the misuse of pthreads in libgit2.
Normally, git_libgit2_init and git_libgit2_shutdown should span the
life of a worker thread in order to be safely used. However, due to
an incorrect implementation of libgit2's threadstate with pthreads,
on unix systems there is a race condition.
Until the use of pthread_key_t is corrected in libgit2, we need to
apply a workaround by always ensuring that the main thread is the
first thread reaching the GitContext constructor.
|
|
In just-mr the config file should only be needed if:
1. We call just with a subcommand requiring a config file, or
2. We use one of just-mr's own commands (except version).
|
|
... with two minor code base changes compared to previous
use of gsl-lite:
- dag.hpp: ActionNode::Ptr and ArtifactNode::Ptr are not
wrapped in gsl::not_null<> anymore, due to lack of support
for wrapping std::unique_ptr<>. More specifically, the
move constructor is missing, rendering it impossible to
use std::vector<>::emplace_back().
- utils/cpp/gsl.hpp: New header file added to implement the
macros ExpectsAudit() and EnsureAudit(), asserts running
only in debug builds, which were available in gsl-lite but
are missing in MS GSL.
|
|
|
|
Make just-mr unconditionally support an option -D that collects a
configuration overlay and forwards it to the invocation of a just
subcommand that supports this option. This syntax-switching facility
makes it easy to embedd dynamic parts of the configuration (like
the head commit to be part of a version string) as those information
can unconditionally be the first argument to just-mr.
|
|
In this way, we have the whole layout of the local build root
consolidated in one place. Moreover, in this way, the location
of the git root is also available to the build tool itself and
can, e.g., be used as fallback CAS.
|
|
|
|
In the rare cases that we need to shell out to git, let the user
configure what binary to use. Option resolves in the same way as
the just executable, including allowing it to be set via just-mrrc.
Updates all cases of shelling out to git (fetch and commit update).
Update just-mr and just-mrrc docs accordingly.
|
|
...to ensure the temporary directories exist before they are needed,
as expected by those methods. This way, the TmpDir class takes care
to also clean up after itself.
Also, pass the local launcher to the methods that now shell out.
|
|
|
|
|
|
To avoid unnecessary work, just-mr uses on-disk caches, including
for the mapping of a distdir to the corresponding git tree. This,
however, implies that the number of repositories that are actually
considered varies: in order to fetch a distdir repository, all
involved archives have to be fetched, but if we have a cache hit
none of them is even looked at.
So, in order to have a consistent reporting only count top-level
targets (i.e., the reachable repositories) in the statistics,
not the archives implicitly contained in a distdir, nor low-level
sub tasks. The actual fetch acitvity is shown separately by the
task tracker.
|
|
... by only taking the additional roots from the reachable
repositories, not also from the additional roots discovered
that way.
Co-authored-by: Paul Cristian Sarbu <paul.cristian.sarbu@huawei.com>
|
|
|
|
The improved GC implementation uses refactored storage
classes instead of directly accessing "unknown" file paths.
The required storage class refactoring is quite substantial
and outlined in the following paragraphs.
The module `buildtool/file_system` was extended by:
- `ObjectCAS`: a plain CAS implementation for
reading/writing blobs and computing digests for a given
`ObjectType`. Depending on that type, files written to the
file system may have different properties (e.g., the x-bit
set) or the digest may be computed differently (e.g., tree
digests in non-compatible mode).
A new module `buildtool/storage` was introduced containing:
- `LocalCAS`: provides a common interface for the "logical
CAS", which internally combines three `ObjectCAS`s, one
for each `ObjectType` (file, executable, tree).
- `LocalAC`: implements the action cache, which needs the
`LocalCAS` for storing cache values.
- `TargetCache`: implements the high-level target cache,
which also needs the `LocalCAS` for storing cache values.
- `LocalStorage`: combines the storage classes `LocalCAS`,
`LocalAC`, and `TargetCache`. Those are initialized with
settings from `StorageConfig`, such as the build root base
path or number of generations for the garbage collector.
`LocalStorage` is templated with a Boolean parameter
`kDoGlobalUplink`, which indicates that, on every
read/write access, the garbage collector should be used
for uplinking across all generations (global).
- `GarbageCollector`: responsible for garbage collection and
the global uplinking across all generations. To do so, it
employs instances of `LocalStorage` with `kDoGlobalUplink`
set to false, in order to avoid endless recursion. The
actual (local) uplinking within two single generations is
performed by the corresponding storage class (e.g.,
`TargetCache` implements uplinking of target cache entries
between two target cache generations etc.). Thereby, the
actual knowledge how data should be uplinked is
implemented by the instance that is responsible for
creating the data in the first place.
|
|
... on reporting the exec command; json encoding is already
quoting enough.
|
|
The forwarding of the log-limit argument is only useful, if
the value is different from the default.
|
|
A local launcher should be forwarded, if it is different to
the default launcher (which is ["env", "--"]); in particular,
an explictly empy launcher should be forwarded.
|
|
... and, in this way, avoid duplication of these constants that
might lead to inconsistencies between those two tools later.
|
|
... by reporting unsupported config values explicitly and fixing
inconsistencies.
|
|
|
|
|
|
... both with respect to just options and the man page specifications.
Option -L of just-mr was reassigned as a short name for
--local-launcher, matching its use in just. Its place is now
correctly held by the full name option --checkout-locations, as
per the section-1 man page.
|
|
|
|
Also update just-mr section-1 man page
|
|
Make clear what fields the defined boolean values refer to.
|
|
When asked to setup, report the number of repositories as repositories
to "set up", not to "check out". Typically, we're not creating a
checkout of that repository in the sense that some directory would
contain the file strucutre of that repository in the file system.
Typically, fetch into our big git repo, or create an artifical
commit there to keep the resulting git tree there.
|
|
This includes also the setup-env command, as well as all ~just~
known subcommands that require a just-mr setup step.
Co-authored-by: Sascha Roloff <sascha.roloff@huawei.com>
|
|
Co-authored-by: Sascha Roloff <sascha.roloff@huawei.com>
|
|
Co-authored-by: Sascha Roloff <sascha.roloff@huawei.com>
|
|
In order to only report actual work, the name of the repository for which
work is actually done needs to be known to certain AsyncMaps used by just-mr.
Also, fetches not originating in a distdir repo checkout need to be counted
additionally, so an additional flag was needed for the content_cas_map.
|
|
|
|
|
|
When calling just from just-mr, all logging arguments are forwarded
as early arguments. If any log files are provided, an unconditional
--log-append is also prepended to ensure the contents of the log files
are not overwritten.
|
|
Both, just and the launcher just-mr are developped and released
together and hence share version numbers. Nevertheless, the tools
are used independently. In fact, just-mr has an option to specify
the just binary to be used. Hence it is useful that each tool is
aware of its own version number. As a launcher, just-mr is supposed
to forward all subcommands known to just, hence the just-mr command
to show the version information cannot be called "version"; we use
"mrversion" instead.
|
|
|
|
|
|
|
|
|
|
|
|
...in order to not include unwanted dependencies in just proper.
The new class extends the GitRepo class used for just's Git tree
operations and gets used in all of just-mr's async maps.
|
|
The actual error fixed in this commit was the usage of the std::vector resize
method instead of the reserve method. The resize method increases the size of
the vector by appending new default-allocated elements (empty strings in this
case), whereas the reserve method just increases the underlying capacity of the
vector without appending elements. In addition to this fix, the remaining
just-args assembly code was refactored to drop the lambda-function usage in
favor of a plain sequence of statements.
|
|
|
|
|
|
The absent value for an optional string is std::nullopt, not
the empty string.
|
|
just-mr fetch is supposed to fetch to the first (in search order)
distdir that is an existing directory (and error out if no such
directory exists). So, once a directory is found, there is no need
to create it---we already verified it is an existing directory.
|
|
... in error message reporting the lack of present distdirs.
|
|
This provides the fix already performed for the just-mr script in
the internal just-mr as well. In short, this fix makes sure that the
computation of the content key for distdir repositories is
independent of the presence of the respective archives in CAS.
|
|
Use intermediary unordered sets to store the repos to_setup and to_include
to avoid unnecessary duplicates. It is purely a performance benefit.
|
