| Age | Commit message (Collapse) | Author |
|---|
|
|
|
|
|
Also adds an appropriate test for this method.
|
|
In certain cases, e.g., on the serve endpoint, an unresolved tree
might lie in a repository other than the Git cache, therefore we
cannot create any new entries there, as it would violate our
guarantee that we only write under our local build root.
Therefore, the resolve_symlinks_map now receives pointers to both
the source and target Git databases and ensures that:
1. any tree created on-the-fly is stored exclusively in the target
repository, and
2. any other entry required for those trees is made available in
the target repository by copying it from the source repository.
Note that in our use case the target repository is always our Git
cache and passing a pointer to that object database is done to
avoid the overhead of otherwise opening the database very often.
|
|
Also extends the tests accordingly.
|
|
|
|
|
|
|
|
This is needed for LocalCAS's splice routines.
|
|
Main culprits:
- std::size_t, std::nullptr_t, and NULL require <cstddef>
- std::move and std::forward require <utility>
- unordered maps and sets require respective includes
- std::for_each and std::all_of require <algorithm>
|
|
Some of the more specific issues addressed:
- missing log_level target/include
- header-only libs wrongly marking deps as private
- missing/misplaced gsl includes
|
|
Commit 2ebf355989eb92ac9967eceee0af14d39477afe0 moved the tmpdir
creation for various tasks into the task itself. In doing so, TmpDir
was called with a relative path; that was, however, is interpreted
relative to the working directory, violating the property that
our tool never write anything outside the local build root unless
explicitly asked to do so (by specifying the output path in an
install or install-cas invocation). Fix this, by calling the the
tmp-dir function that is storage-layout aware.
|
|
... by avoiding reusing temp dirs for execute. While we are
at it, also refactor LocalFetchViaTmpRepo() to create its
own empty temp dirs, that cannot be reused by the caller.
|
|
|
|
For archives and Git repositories we should ensure that not finding
the witnessing entity (archive content blob or Git commit,
respectively) results in a distinct status in the response to a
request that sets up roots on the serve endpoint. This will allow
just-mr to better handle its interaction with the serve endpoint.
|
|
|
|
... and improve log messages in case of failure.
|
|
|
|
...to make it available also for setting up 'just serve' builds.
|
|
|
|
|
|
|
|
Ignore-special git-tree-based roots are still content defined, so
one should use the correct marker in the JSON description of the
root that will be stored in the repository description of target
cache keys. This commit fixes the issue and improves documentation.
|
|
The man page for open(2) says the following to the O_SYNC flag: 'O_SYNC
provides synchronized I/O file integrity completion, meaning write operations
will flush data and all associated metadata to the underlying hardware.' This
flag results in a high delay when files are stored in casx, e.g., several
seconds for medium-sized files such as 23 MB. Since just does not care about
persistency, this strong synchronization mechanism is not required and is
deactivated.
|
|
Absent roots are characterised only by a Git tree hash, so a new
variant of the underlying stored information was added in the form
of a plain string.
In order to avoid unwanted implicit conversions when instantiating
via literal strings, we force callers of the constructors to
explicitly differentiate between plain strings and filesystem
paths. Existing tests were updated to reflect this.
Co-authored-by: Alberto Sartori <alberto.sartori@huawei.com>
|
|
TargetCache...
...backed by the same CAS, but the FileStorage uses the given
shard. This is particularly useful for the just-serve server
implementation, since the sharding must be performed according to the
client's request and not following the server configuration.
|
|
Make sure that all CopyFile, WriteFile, and CreateSymlink
functions properly unlink the target file (if it exists and
overwrite requested) to avoid interferences of the install
command. With this change, the clean up step for install-cas
and the within GraphTraverser can new be omitted.
|
|
This is required in order to make them available to 'just serve'
in a minimal just installation.
|
|
This avoids using the more geenric GitRepoRemote method which
has libcurl as a dependency, something that is not needed for this
Git operation.
|
|
Also fixes a small typo in tree existence checker log messages.
|
|
Upwards symlinks should still be collected from actions, even if
only the non-upwards symlinks are supported artifact types. The
client side is thus the one responsible with enforcing the
non-upwardness condition.
|
|
|
|
...that std::filesystem::* calls produce. This is because existence
and type checks use almost exclusively std::filesystem::status,
which follows symbolic links, when being called with path arguments.
Instead, one should instead use these methods with the value
returned by a call of std::filesystem::symlink_status.
This commit also streamlines the FileSystemManager tests, as well
as replace bare calls to std::filesystem with their FileSystemManager
counterparts (where suitable).
|
|
The introduction of non-upwards symlinks as first-class objects
should have updated the handling of known git tree artifacts
containing symlinks. In particular, one should consider trees in
their entirety when uploading (irrespective of the ignore_special
flag), and git trees should only be reported as known only if
the ignore_special flag is set to false.
|
|
The command 'git add .' does not include paths found in .gitignore
files in the directory tree where the command is issued. This is
not the desired behaviour, as we expect for a tree with a given
commit id to contain all of the entries, irrespective of their
meaning to Git.
This commit addresses the issue as described.
For the just-mr.py script we modified the staging command to
'git add -f .'.
For the compiled just-mr, simply adding the force flag to
'git_index_add_all' did not work as intended for files found in
ignored subdirectories. This is a known libgit2 issue which has
been fixed in v1.6.3. Until we can upgrade our libgit2 version,
a workaround was implemented: we recursively read the directory
entries ourselves and add each of them iteratively using
'git_index_add_bypath', making sure to ignore the root '.git'
subtree (which cannot be staged).
At the moment the handling of Git submodules remains an open issue,
as Git does not allow '.git' subtrees to be forcefully added to the
index, and thus such directory entries will currently not be
considered as part of a git tree. This however is consistent
behavior between Git and libgit2.
|
|
...allowing the skipping of certain subtrees if needed. This is
useful, e.g., in simulating what a 'git add' call would do,
which ignores all '.git' subdirectories.
Also adds a corresponding test for the new method.
|
|
|
|
|
|
|
|
...and update tests accordingly.
|
|
...as early as possible. This ensures that callers always receive
only the tree entries for the supported object types.
For the symlinks non-upwardness check we pass a lambda capturing
the real backend of the tree entries, such that the symlinks can
be read.
Updates git_tree tests accordingly.
|
|
...but make sure it is still considered a special type.
The only non-special entry types remain file, executable, and tree.
|
|
|
|
A git CAS ist just a fall back, so it is OK if it is absent (e.g.,
the specificed directory does not exist). Therefore only log at
debug level, not at error level if we cannot open it.
|
|
... which became obsolete with the new fdless write/copy
implementations.
|
|
... to remove the risk of deadlocks on certain combinations
of C++ standard library and libc when performing the
copy/write in a child process. For 'fdless' copy/write, a
child process is used to prevent the parent from getting
polluted with open writable file descriptors (which might
get inherited by other children that keep them open and can
cause EBUSY errors).
|
|
This feature has been introduced with C++20.
|
|
... and in this way, continue to work correctly in the absence
of a current working directory.
|
|
|
|
|
