| Age | Commit message (Collapse) | Author |
|---|
|
Before this patch, when creating an action directory, symlinks were
staged as regular files.
|
|
...that std::filesystem::* calls produce. This is because existence
and type checks use almost exclusively std::filesystem::status,
which follows symbolic links, when being called with path arguments.
Instead, one should instead use these methods with the value
returned by a call of std::filesystem::symlink_status.
This commit also streamlines the FileSystemManager tests, as well
as replace bare calls to std::filesystem with their FileSystemManager
counterparts (where suitable).
|
|
...allowing the skipping of certain subtrees if needed. This is
useful, e.g., in simulating what a 'git add' call would do,
which ignores all '.git' subdirectories.
Also adds a corresponding test for the new method.
|
|
|
|
|
|
|
|
|
|
via a 'SYMLINK' constructor function. This works similarly to the
'FILE' construct, but the name given must point to a non-upwards
symlink and a symlink artifact is being generated from it.
Also updates the relevant tests.
|
|
...and update tests accordingly.
|
|
...as early as possible. This ensures that callers always receive
only the tree entries for the supported object types.
For the symlinks non-upwardness check we pass a lambda capturing
the real backend of the tree entries, such that the symlinks can
be read.
Updates git_tree tests accordingly.
|
|
|
|
|
|
This feature has been introduced with C++20.
|
|
...using the new ignore-special tree variant.
|
|
...by removing unused subfolders and corresponding redundant checks
in the tests using it. Also fixes misues in tests of the GetSubtree
family of methods.
|
|
|
|
This allows better separation and, in particular, repositories
needed only for tests do not have to be provided for building the
tools. This also better documents which dependencies are only needed
for testing.
|
|
... in particular echo(1), which behaves different on dash
and sh.
|
|
... with two minor code base changes compared to previous
use of gsl-lite:
- dag.hpp: ActionNode::Ptr and ArtifactNode::Ptr are not
wrapped in gsl::not_null<> anymore, due to lack of support
for wrapping std::unique_ptr<>. More specifically, the
move constructor is missing, rendering it impossible to
use std::vector<>::emplace_back().
- utils/cpp/gsl.hpp: New header file added to implement the
macros ExpectsAudit() and EnsureAudit(), asserts running
only in debug builds, which were available in gsl-lite but
are missing in MS GSL.
|
|
|
|
Move the creation of a fake repository on top of an existing odb
into the individual threads, to ensure the thread-safety of the
operations on fake repositories is properly tested.
|
|
|
|
... as `git init -b ...` is a rather recent git feature.
|
|
|
|
Using "test" in the name of a test target is unnecessary and
adds gratuitous inconsistencies.
|
|
|
|
tests have been updated accordingly
|
|
...both buildbarn and just execute do not allow a direct upload to the
action cache. Moreover, our tool does not support it, anyway, meaning
it was dead code.
|
|
|
|
The improved GC implementation uses refactored storage
classes instead of directly accessing "unknown" file paths.
The required storage class refactoring is quite substantial
and outlined in the following paragraphs.
The module `buildtool/file_system` was extended by:
- `ObjectCAS`: a plain CAS implementation for
reading/writing blobs and computing digests for a given
`ObjectType`. Depending on that type, files written to the
file system may have different properties (e.g., the x-bit
set) or the digest may be computed differently (e.g., tree
digests in non-compatible mode).
A new module `buildtool/storage` was introduced containing:
- `LocalCAS`: provides a common interface for the "logical
CAS", which internally combines three `ObjectCAS`s, one
for each `ObjectType` (file, executable, tree).
- `LocalAC`: implements the action cache, which needs the
`LocalCAS` for storing cache values.
- `TargetCache`: implements the high-level target cache,
which also needs the `LocalCAS` for storing cache values.
- `LocalStorage`: combines the storage classes `LocalCAS`,
`LocalAC`, and `TargetCache`. Those are initialized with
settings from `StorageConfig`, such as the build root base
path or number of generations for the garbage collector.
`LocalStorage` is templated with a Boolean parameter
`kDoGlobalUplink`, which indicates that, on every
read/write access, the garbage collector should be used
for uplinking across all generations (global).
- `GarbageCollector`: responsible for garbage collection and
the global uplinking across all generations. To do so, it
employs instances of `LocalStorage` with `kDoGlobalUplink`
set to false, in order to avoid endless recursion. The
actual (local) uplinking within two single generations is
performed by the corresponding storage class (e.g.,
`TargetCache` implements uplinking of target cache entries
between two target cache generations etc.). Thereby, the
actual knowledge how data should be uplinked is
implemented by the instance that is responsible for
creating the data in the first place.
|
|
... to bring their own remote-execution endpoint.
|
|
|
|
... by keeping track of each blob being a file or executable
and storing it to the correct local physical CAS directory.
The new flag is merely a hint and only used by the local
execution API. Leaving it out will still correctly transfer
the blob but may cause unnecessary duplicates in file CAS.
|
|
...in order to not include unwanted dependencies in just proper.
The new class extends the GitRepo class used for just's Git tree
operations and gets used in all of just-mr's async maps.
|
|
|
|
... and ensure that cascades of checks are performed with
only a single filesystem stat per method.
|
|
...and use instead the branch name. A valid direct refspec (as those
retrieved by a remote_ls call) will always end in the branch name,
so checking the last path component ('/'-delimited substring) of a
retrieved refspec is enough.
|
|
This also removes the need to call the GET_BRANCH_REFNAME critical
operation.
|
|
|
|
|
|
Needed by the content-in-CAS git map to check fetched archives' checksums.
SHA1 and SHA256 hashers are already implemented.
|
|
|
|
|
|
|
|
|
|
|
|
Compared to the previous commit, the action graphs for just
and its unit tests are unchanged.
- Git hash of action graph for ["",""]:
c6e75f17abd7ffaab6ff9bb725ad67ec0bf6c973
- Git hash of action graph for ["test/buildtool","TESTS"]:
8063dfb3dd7daa9ae01d95c177e14946f785c57e
Refactor:
- "local cflags" to "private-cflags"
- "local defines" to "private-defines"
- "link externals" to "private-ldflags"
- "deps" to "private-deps" for (test) binaries
- "proto" to "private-proto" for binaries
Improvements:
- consistent variable declaration order:
OS, ARCH, HOST_ARCH, TARGET_ARCH,
CC, CXX, CFLAGS, CXXFLAGS, ADD_CFLAGS, ADD_CXXFLAGS,
AR, ENV, PATH
- use fields close to their definition (in RULES)
- use common expression for binaries and test binaries
- split expression "flags" and "compiler"
... to separate ones for CC and CXX.
- rename "transition" to "deps-transition"
... to avoid conflicts with other transitions.
- support "defaults-transition" for CC expressions
Implement:
- "cflags" for libraries
- "private-cflags" for (test) binaries
- "private-defines" for test binaries
- "private-ldflags" for test binaries
- (public) "defines" for libraries
|
|
|
|
While there, also add all direct dependencies explicitly; using
directly dependencies that are pulled in only indireclty causes
problems from a maintainability point of view.
|
|
This enforces the explicit specification, which object type, either file or
tree, should be used to create an artifact digest. This also avoids subtile
errors at locations as in the previous commit, where files as well as trees are
supposed to be handled, but digest creation mistakenly defaults to file object
type.
|
