| Age | Commit message (Collapse) | Author |
|---|
|
...caused by incorrectly setting and resetting the library internal
state and the misuse of pthreads in libgit2.
Normally, git_libgit2_init and git_libgit2_shutdown should span the
life of a worker thread in order to be safely used. However, due to
an incorrect implementation of libgit2's threadstate with pthreads,
on unix systems there is a race condition.
Until the use of pthread_key_t is corrected in libgit2, we need to
apply a workaround by always ensuring that the main thread is the
first thread reaching the GitContext constructor.
|
|
... and, in particular, do not read any rc files for just-mr
the user might have.
|
|
... in particular echo(1), which behaves different on dash
and sh.
|
|
...as required by grpc v1.53.0
|
|
... with two minor code base changes compared to previous
use of gsl-lite:
- dag.hpp: ActionNode::Ptr and ArtifactNode::Ptr are not
wrapped in gsl::not_null<> anymore, due to lack of support
for wrapping std::unique_ptr<>. More specifically, the
move constructor is missing, rendering it impossible to
use std::vector<>::emplace_back().
- utils/cpp/gsl.hpp: New header file added to implement the
macros ExpectsAudit() and EnsureAudit(), asserts running
only in debug builds, which were available in gsl-lite but
are missing in MS GSL.
|
|
endpoint
This can be useful, for example, to test if justbuild can successfully
communicate with the specified remote execution service.
|
|
|
|
|
|
Make just-mr unconditionally support an option -D that collects a
configuration overlay and forwards it to the invocation of a just
subcommand that supports this option. This syntax-switching facility
makes it easy to embedd dynamic parts of the configuration (like
the head commit to be part of a version string) as those information
can unconditionally be the first argument to just-mr.
|
|
|
|
|
|
... by including the details of the parse error.
|
|
Move the creation of a fake repository on top of an existing odb
into the individual threads, to ensure the thread-safety of the
operations on fake repositories is properly tested.
|
|
|
|
... as `git init -b ...` is a rather recent git feature.
|
|
|
|
|
|
... also verifying that the local CAS is used, even in the presence
of remote execution.
|
|
|
|
Using "test" in the name of a test target is unnecessary and
adds gratuitous inconsistencies.
|
|
In the rare cases that we need to shell out to git, let the user
configure what binary to use. Option resolves in the same way as
the just executable, including allowing it to be set via just-mrrc.
Updates all cases of shelling out to git (fetch and commit update).
Update just-mr and just-mrrc docs accordingly.
|
|
...due to limitations in SSH support in libgit2. In this case, we
simply execute 'git ls-remote <repo> [<branch>]' and then parse
the output. Remote interogation requires no local repository, so
it is an asynchronious operation by default.
|
|
...due to limited SSH support in libgit2. In order to allow the
fetches to still be parallel, we execute:
git fetch --no-auto-gc --no-write-fetch-head <repo> [<branch>]
This only fetches the packs without updating any refs, at the slight
cost of sometimes fetching some redundant information, which for our
purposes is practically a non-issue.
(If really needed, a 'git gc' call can be done eventually to try to
compact the fetched packs, although a save in disk space is not
actually guaranteed.)
|
|
|
|
While for unit tests it is best practice to replicate the layout
of the source tree, for end-to-end tests, this is less so. This
is particularly true, as the distinction between the tests for two
tools is a bit blurry; some tests for just still use just-mr as a
launcher, and, e.g., the fetch test for just-mr uses just to carry
out the garbage collection. Therefore, move all end-to-end tests
together so that we also have a joinded target for precisely the
end-to-end tests.
In this reorganisation also indicate more explicitly which tests are
also available for the bootstrap multi-repo tool. That information
was so far hidden by the fact that in the other directory the tool
dependency would not dispatch on TEST_BOOTSTRAP_JUST_MR.
|
|
|
|
|
|
|
|
tests have been updated accordingly
|
|
...both buildbarn and just execute do not allow a direct upload to the
action cache. Moreover, our tool does not support it, anyway, meaning
it was dead code.
|
|
|
|
The improved GC implementation uses refactored storage
classes instead of directly accessing "unknown" file paths.
The required storage class refactoring is quite substantial
and outlined in the following paragraphs.
The module `buildtool/file_system` was extended by:
- `ObjectCAS`: a plain CAS implementation for
reading/writing blobs and computing digests for a given
`ObjectType`. Depending on that type, files written to the
file system may have different properties (e.g., the x-bit
set) or the digest may be computed differently (e.g., tree
digests in non-compatible mode).
A new module `buildtool/storage` was introduced containing:
- `LocalCAS`: provides a common interface for the "logical
CAS", which internally combines three `ObjectCAS`s, one
for each `ObjectType` (file, executable, tree).
- `LocalAC`: implements the action cache, which needs the
`LocalCAS` for storing cache values.
- `TargetCache`: implements the high-level target cache,
which also needs the `LocalCAS` for storing cache values.
- `LocalStorage`: combines the storage classes `LocalCAS`,
`LocalAC`, and `TargetCache`. Those are initialized with
settings from `StorageConfig`, such as the build root base
path or number of generations for the garbage collector.
`LocalStorage` is templated with a Boolean parameter
`kDoGlobalUplink`, which indicates that, on every
read/write access, the garbage collector should be used
for uplinking across all generations (global).
- `GarbageCollector`: responsible for garbage collection and
the global uplinking across all generations. To do so, it
employs instances of `LocalStorage` with `kDoGlobalUplink`
set to false, in order to avoid endless recursion. The
actual (local) uplinking within two single generations is
performed by the corresponding storage class (e.g.,
`TargetCache` implements uplinking of target cache entries
between two target cache generations etc.). Thereby, the
actual knowledge how data should be uplinked is
implemented by the instance that is responsible for
creating the data in the first place.
|
|
|
|
|
|
|
|
... instead of using pipes that are prone to get full. Also
increase log level to Trace.
|
|
|
|
remote endpoint
|
|
... now that they no longer depend on an external remote
execution.
|
|
... to bring their own remote-execution endpoint.
|
|
In order to keep our tests self-contained, do not rely on
an external remote-execution service to be present; instead,
use `just execute` to provide the remote execution service.
|
|
|
|
|
|
... and, in this way, ensure the test also tests properly in
absence of an external remote execution system.
|
|
... similar to ["@", "rules", "shell/test", "script"], but implicitly
starting a remote execution in the background, passing the information
about that endpoint to the test via an environment variable.
|
|
|
|
commit update
Uses the SSL certification utility method to correctly set the
certification check options for the remote URL libgit2 calls.
Due to the fact that remote operations are done via a temporary
repository to allow concurrent work, the correct repository
configuration needs to be interrogated. Thankfully, libgit2
provides a thread safe config snapshot object to be used in
such scenarios.
Also updates the existing GitRepoRemote tests accordingly.
|
|
|
|
|
|
Co-authored-by: Sascha Roloff <sascha.roloff@huawei.com>
|
