| Age | Commit message (Collapse) | Author |
|---|
|
tests have been updated accordingly
|
|
...both buildbarn and just execute do not allow a direct upload to the
action cache. Moreover, our tool does not support it, anyway, meaning
it was dead code.
|
|
|
|
The improved GC implementation uses refactored storage
classes instead of directly accessing "unknown" file paths.
The required storage class refactoring is quite substantial
and outlined in the following paragraphs.
The module `buildtool/file_system` was extended by:
- `ObjectCAS`: a plain CAS implementation for
reading/writing blobs and computing digests for a given
`ObjectType`. Depending on that type, files written to the
file system may have different properties (e.g., the x-bit
set) or the digest may be computed differently (e.g., tree
digests in non-compatible mode).
A new module `buildtool/storage` was introduced containing:
- `LocalCAS`: provides a common interface for the "logical
CAS", which internally combines three `ObjectCAS`s, one
for each `ObjectType` (file, executable, tree).
- `LocalAC`: implements the action cache, which needs the
`LocalCAS` for storing cache values.
- `TargetCache`: implements the high-level target cache,
which also needs the `LocalCAS` for storing cache values.
- `LocalStorage`: combines the storage classes `LocalCAS`,
`LocalAC`, and `TargetCache`. Those are initialized with
settings from `StorageConfig`, such as the build root base
path or number of generations for the garbage collector.
`LocalStorage` is templated with a Boolean parameter
`kDoGlobalUplink`, which indicates that, on every
read/write access, the garbage collector should be used
for uplinking across all generations (global).
- `GarbageCollector`: responsible for garbage collection and
the global uplinking across all generations. To do so, it
employs instances of `LocalStorage` with `kDoGlobalUplink`
set to false, in order to avoid endless recursion. The
actual (local) uplinking within two single generations is
performed by the corresponding storage class (e.g.,
`TargetCache` implements uplinking of target cache entries
between two target cache generations etc.). Thereby, the
actual knowledge how data should be uplinked is
implemented by the instance that is responsible for
creating the data in the first place.
|
|
|
|
|
|
|
|
... instead of using pipes that are prone to get full. Also
increase log level to Trace.
|
|
|
|
remote endpoint
|
|
... now that they no longer depend on an external remote
execution.
|
|
... to bring their own remote-execution endpoint.
|
|
In order to keep our tests self-contained, do not rely on
an external remote-execution service to be present; instead,
use `just execute` to provide the remote execution service.
|
|
|
|
|
|
... and, in this way, ensure the test also tests properly in
absence of an external remote execution system.
|
|
... similar to ["@", "rules", "shell/test", "script"], but implicitly
starting a remote execution in the background, passing the information
about that endpoint to the test via an environment variable.
|
|
|
|
commit update
Uses the SSL certification utility method to correctly set the
certification check options for the remote URL libgit2 calls.
Due to the fact that remote operations are done via a temporary
repository to allow concurrent work, the correct repository
configuration needs to be interrogated. Thankfully, libgit2
provides a thread safe config snapshot object to be used in
such scenarios.
Also updates the existing GitRepoRemote tests accordingly.
|
|
|
|
|
|
Co-authored-by: Sascha Roloff <sascha.roloff@huawei.com>
|
|
All desired outputs of just-mr (e.g., the config file after a setup call)
are printed to stdout, while all logging messages are printed to stderr.
Therefore, one should avoid explicitly piping stderr to stdout, as this
can break the intended use of just-mr.
|
|
|
|
|
|
|
|
... instead of only using the "install-cas" subcommand,
which considers the file and executable CAS as one single
logical instance. Rather also check the CAS directories
directly to verify that files and executables are actually
downloaded to the correct CAS. Furthermore, added "main" as
an export target to produce an executable file for running
the test.
|
|
... by keeping track of each blob being a file or executable
and storing it to the correct local physical CAS directory.
The new flag is merely a hint and only used by the local
execution API. Leaving it out will still correctly transfer
the blob but may cause unnecessary duplicates in file CAS.
|
|
Only tests pertaining to methods not covered in the tests of the
inherited class were added.
|
|
...in order to not include unwanted dependencies in just proper.
The new class extends the GitRepo class used for just's Git tree
operations and gets used in all of just-mr's async maps.
|
|
...in order to not include unwanted dependencies in just proper.
As the whole other_tools folder is meant to be excluded from
bootstrapping, also remove the bootstrap guards.
|
|
...in order to not include unwanted dependencies in just proper.
As the whole other_tools folder is meant to be excluded from
bootstrapping, also remove the bootstrap guards.
|
|
|
|
|
|
|
|
|
|
Co-authored by: Klaus Aehlig <klaus.aehlig@huawei.com>
|
|
... and also fix the test if we reached the fixed point; of course,
that has to use the same configuration as we do not expect different
tool chains to produce bit-by-bit identical output.
|
|
|
|
... and ensure that cascades of checks are performed with
only a single filesystem stat per method.
|
|
Not all repositories mentioned in the repository configuration are
needed to boostrap just (e.g., because they are used for testing).
Support dropping dependencies for bootstrapping by replacing them
by a reference to an empty directory and in this way have less
files that need to be fetched.
|
|
As now all remote Git operations in GitRepo require at most just
the branch name, there is no more need to inquire the repository
about branch refspecs.
|
|
...and use instead the branch name. A valid direct refspec (as those
retrieved by a remote_ls call) will always end in the branch name,
so checking the last path component ('/'-delimited substring) of a
retrieved refspec is enough.
|
|
This also removes the need to call the GET_BRANCH_REFNAME critical
operation.
|
|
By default, just-mr fetches to the first existing distribution
directory; by default, the first distribution directory to consider
is ~/.distfiles. If an explicit destination is desired, it should
be specified with the -o option. Fix the invocation of just-mr in
the end-to-end test to use the specified destination; this also makes
local execution of te test hermetic (as was originally intended), even
if the user has a directory ~/.distfiles.
|
|
Run build and garbage collection end to end and verify two basic
properties.
- A target used after the the last gc is still available after
another gc, and so are all its parts, even if the output is a
single tree.
- A target not used between to gc is purged from disk and disk
space is not used any more.
|
|
|
|
|
|
|
|
|
