| Age | Commit message (Collapse) | Author |
|---|
|
...due to limited SSH support in libgit2. In order to allow the
fetches to still be parallel, we execute:
git fetch --no-auto-gc --no-write-fetch-head <repo> [<branch>]
This only fetches the packs without updating any refs, at the slight
cost of sometimes fetching some redundant information, which for our
purposes is practically a non-issue.
(If really needed, a 'git gc' call can be done eventually to try to
compact the fetched packs, although a save in disk space is not
actually guaranteed.)
|
|
|
|
While for unit tests it is best practice to replicate the layout
of the source tree, for end-to-end tests, this is less so. This
is particularly true, as the distinction between the tests for two
tools is a bit blurry; some tests for just still use just-mr as a
launcher, and, e.g., the fetch test for just-mr uses just to carry
out the garbage collection. Therefore, move all end-to-end tests
together so that we also have a joinded target for precisely the
end-to-end tests.
In this reorganisation also indicate more explicitly which tests are
also available for the bootstrap multi-repo tool. That information
was so far hidden by the fact that in the other directory the tool
dependency would not dispatch on TEST_BOOTSTRAP_JUST_MR.
|
|
|
|
|
|
|
|
tests have been updated accordingly
|
|
...both buildbarn and just execute do not allow a direct upload to the
action cache. Moreover, our tool does not support it, anyway, meaning
it was dead code.
|
|
|
|
The improved GC implementation uses refactored storage
classes instead of directly accessing "unknown" file paths.
The required storage class refactoring is quite substantial
and outlined in the following paragraphs.
The module `buildtool/file_system` was extended by:
- `ObjectCAS`: a plain CAS implementation for
reading/writing blobs and computing digests for a given
`ObjectType`. Depending on that type, files written to the
file system may have different properties (e.g., the x-bit
set) or the digest may be computed differently (e.g., tree
digests in non-compatible mode).
A new module `buildtool/storage` was introduced containing:
- `LocalCAS`: provides a common interface for the "logical
CAS", which internally combines three `ObjectCAS`s, one
for each `ObjectType` (file, executable, tree).
- `LocalAC`: implements the action cache, which needs the
`LocalCAS` for storing cache values.
- `TargetCache`: implements the high-level target cache,
which also needs the `LocalCAS` for storing cache values.
- `LocalStorage`: combines the storage classes `LocalCAS`,
`LocalAC`, and `TargetCache`. Those are initialized with
settings from `StorageConfig`, such as the build root base
path or number of generations for the garbage collector.
`LocalStorage` is templated with a Boolean parameter
`kDoGlobalUplink`, which indicates that, on every
read/write access, the garbage collector should be used
for uplinking across all generations (global).
- `GarbageCollector`: responsible for garbage collection and
the global uplinking across all generations. To do so, it
employs instances of `LocalStorage` with `kDoGlobalUplink`
set to false, in order to avoid endless recursion. The
actual (local) uplinking within two single generations is
performed by the corresponding storage class (e.g.,
`TargetCache` implements uplinking of target cache entries
between two target cache generations etc.). Thereby, the
actual knowledge how data should be uplinked is
implemented by the instance that is responsible for
creating the data in the first place.
|
|
|
|
|
|
|
|
... instead of using pipes that are prone to get full. Also
increase log level to Trace.
|
|
|
|
remote endpoint
|
|
... now that they no longer depend on an external remote
execution.
|
|
... to bring their own remote-execution endpoint.
|
|
In order to keep our tests self-contained, do not rely on
an external remote-execution service to be present; instead,
use `just execute` to provide the remote execution service.
|
|
|
|
|
|
... and, in this way, ensure the test also tests properly in
absence of an external remote execution system.
|
|
... similar to ["@", "rules", "shell/test", "script"], but implicitly
starting a remote execution in the background, passing the information
about that endpoint to the test via an environment variable.
|
|
|
|
commit update
Uses the SSL certification utility method to correctly set the
certification check options for the remote URL libgit2 calls.
Due to the fact that remote operations are done via a temporary
repository to allow concurrent work, the correct repository
configuration needs to be interrogated. Thankfully, libgit2
provides a thread safe config snapshot object to be used in
such scenarios.
Also updates the existing GitRepoRemote tests accordingly.
|
|
|
|
|
|
Co-authored-by: Sascha Roloff <sascha.roloff@huawei.com>
|
|
All desired outputs of just-mr (e.g., the config file after a setup call)
are printed to stdout, while all logging messages are printed to stderr.
Therefore, one should avoid explicitly piping stderr to stdout, as this
can break the intended use of just-mr.
|
|
|
|
|
|
|
|
... instead of only using the "install-cas" subcommand,
which considers the file and executable CAS as one single
logical instance. Rather also check the CAS directories
directly to verify that files and executables are actually
downloaded to the correct CAS. Furthermore, added "main" as
an export target to produce an executable file for running
the test.
|
|
... by keeping track of each blob being a file or executable
and storing it to the correct local physical CAS directory.
The new flag is merely a hint and only used by the local
execution API. Leaving it out will still correctly transfer
the blob but may cause unnecessary duplicates in file CAS.
|
|
Only tests pertaining to methods not covered in the tests of the
inherited class were added.
|
|
...in order to not include unwanted dependencies in just proper.
The new class extends the GitRepo class used for just's Git tree
operations and gets used in all of just-mr's async maps.
|
|
...in order to not include unwanted dependencies in just proper.
As the whole other_tools folder is meant to be excluded from
bootstrapping, also remove the bootstrap guards.
|
|
...in order to not include unwanted dependencies in just proper.
As the whole other_tools folder is meant to be excluded from
bootstrapping, also remove the bootstrap guards.
|
|
|
|
|
|
|
|
|
|
Co-authored by: Klaus Aehlig <klaus.aehlig@huawei.com>
|
|
... and also fix the test if we reached the fixed point; of course,
that has to use the same configuration as we do not expect different
tool chains to produce bit-by-bit identical output.
|
|
|
|
... and ensure that cascades of checks are performed with
only a single filesystem stat per method.
|
|
Not all repositories mentioned in the repository configuration are
needed to boostrap just (e.g., because they are used for testing).
Support dropping dependencies for bootstrapping by replacing them
by a reference to an empty directory and in this way have less
files that need to be fetched.
|
|
As now all remote Git operations in GitRepo require at most just
the branch name, there is no more need to inquire the repository
about branch refspecs.
|
|
...and use instead the branch name. A valid direct refspec (as those
retrieved by a remote_ls call) will always end in the branch name,
so checking the last path component ('/'-delimited substring) of a
retrieved refspec is enough.
|
|
This also removes the need to call the GET_BRANCH_REFNAME critical
operation.
|
