| Age | Commit message (Collapse) | Author |
|---|
|
|
|
...that validates hashes and stores some additional information about them.
|
|
...from ObjectInfo and ArtifactDigest
|
|
|
|
...to simplify further refactoring.
|
|
...bypassing ArtifactDigest functionality.
|
|
...with ArtifactDigest.
|
|
...with ArtifactDigest.
|
|
|
|
...with ArtifactDigest.
|
|
...with ArtifactDigest.
|
|
...with ArtifactDigest.
|
|
...with ArtifactDigest
|
|
...with ArtifactDigest
|
|
...with ArtifactDigest.
|
|
...with ArtifactDigest.
|
|
|
|
...with ArtifactDigest.
|
|
...with ArtifactDigest.
|
|
...with ArtifactDigest.
|
|
|
|
...and replace obvious redundant conversions to bazel_re::Digest, which were done to ensure that the digest represents a tree.
|
|
Remote execution of actions is handled via long-running operations.
Here we have to be careful with the involved status codes: there
is the status code of the operation and the response contains a
faild that also happens to be a status code. The protocol states
Errors discovered during creation of the `Operation` will be
reported as gRPC Status errors, while errors that occurred while
running the action will be reported in the `status` field of
the `ExecuteResponse`
So we have to distinguish between two kinds of DEADLINE_EXCEEDED.
- If reported by the rpc, it means, we failed to obtain the status
of the ongoing action in a reasonable amount of time; here we
can do nothing but retry.
- If we obtain an answer and that answer has state DEADLINE_EXCEEDED
this means "The execution timed out."; hence we must not retry
and report the result properly to the user.
|
|
The default options of std::filesystem::copy include following
symlinks, resulting in file repositories creating wrong trees if
containing unresolved symlinks, or failing unexpectedly early if
symlink cycles existed.
This is fixed by ensuring the copy_symlinks option is always used.
|
|
The root async map in a chain of calls should always be checked
for missing value, which can happen if, e.g., a cycle happens or
a thread gets killed by the system.
Properly handle this by checking explicitly if a value has been
posted. If not, check for cycles where it makes sense (for
example, in the resolving of symlinks), otherwise report any
pending map keys not yet processed.
This is done for all just-mr commands working with async maps.
|
|
...to be used when reporting pending keys on failure to post value.
|
|
...in async map instances, same as for reporting cycles.
This removes the restriction that the key object has to posses the
ToString method, allowing it to be used, e.g., with just-mr maps.
The now obsolete HasToString concept is removed.
|
|
|
|
Fixes a false assumption that the result of resolving the tree will
always be set if the map doesn't log fatal, when in fact the map
might fail to set a value if, e.g., a thread is killed by the
system or there is a symlinks cycle.
|
|
The separation of cache-key handling and CAS lockup in
e6a91bb733b0738cee0b3ae06ee640f70c1e787f unified the log-level of
two messages to warning: the absence of a cache entry (originally
debug) and a report on a malformed entry in the cache (originally
warning). As we routinely expect non-cached actions in a build,
demote those messages to debug level in order to keep the log
readable and not confuse the user with warnings about expected
behaviour.
|
|
... as those typically are of transient nature as well.
|
|
|
|
We already accept short writes in batch uploads, but when no
progress is made, we cannot simply retry, as this might lead to
an infinite loop. Instead, we give up on batching and upload each
blob one by one.
|
|
... and correctly report the error.
- If we cannot store the bytes we received, this is an internal error.
- If the bytes received have a different hash than announced, report
this user error as INVLID_ARGUMENT.
|
|
... while keeping our .clang-format file.
|
|
|
|
...into writing/reading to CAS and writing/reading entries (to remove code duplications in uplinking and obtaining the result).
The read functionality cannot be replaced by a single function since the CAS key of the entry is required during uplinking. Although the write functionality can, it is also separated into two parts to keep the reverse logic clear.
|
|
Removed named keys of entries (hash, size). These prefixes were added for every chunk creating an additional overhead without any benefit.
Removed prefixes of hashes (*62*hash) appearing in native mode and creating an additional overhead as well: it is known that all of them are blobs.
|
|
Also add empty directory in test script to ensure we don't regress
in the future. While there, fix some typos.
|
|
...by writing its tree directly in the object database instead of
working with the index. This allows the creation of trees that
contain also entries with 'magic' names, such as the .git folder
or .gitignore files.
Callers must ensure the given directory only contains the needed
entries. In particular, just-mr maps and serve service are updated
to separate the import-to-Git repository path from the temporary
path containing the content to be committed, to avoid polluting the
content path with entries generated on repository initialization.
|
|
The tree is traversed bottom-up and each entry is explicitly added
to the object database: non-directory entries have their content
read and are added as blobs with corresponding permissions based on
file type (regular, executable, symlink), while subdirectories are
added as trees via CreateTree based on the already added blobs.
|
|
While there, ensure optional Git operation parameters are checked
before use for the operations that require them.
|
|
In preparation for subsequent changes, specify the directory path
containing the tree content to be committed explicitly. This change
will allow eventually to be able to specify paths that are
different from the root path of the repository in which the commit
is created.
This commit renames and refactors StageAndCommitAllAnnonymous to
allow a directory path to be passed. The just-mr and serve service
logic is updated such that current behaviour is otherwise
unchanged.
|
|
The libgit2 treebuilder has unnecessary validity checks for tree
entries, including for Git-specific magic names (such as '.git'),
which cannot be disabled. However, in our tool any filesystem
entry should be allowed to be part of a tree.
Therefore, the treebuilder-based implementation for CreateTree is
replaced by a direct writing of trees, by content, into the
underlying repository object database.
Additionally, as direct insertion into the object dabase does not
check the validity of the tree entries, as was done implicitly by
the treebuilder before, add a check for existence of the tree
entries into the debug-level generic validity check, with the
option to not perform it for ReadTree (as it is unnecessary there).
|
|
|
|
The 'branch' field is deprecated, not being used by any of the
critical Git operations, thus it can be removed.
|
|
|
|
The remote execution protocol is a bit unclear about how to
deal with blob updates for which we got no response. While
some clients consider a blob update failed only if a failed
response is received, we are going extra defensive here and
also consider missing responses to be a failed blob update.
Issue a retry for the missing blobs.
|
|
|
|
... and also perform conflict check on the normalized paths. Still,
the the output of the "ACTION" funtion be keyed by the representation
of the paths as originally described, to allow the author of a rule
to use non-normalized paths as well.
|
