| Age | Commit message (Collapse) | Author |
|---|
|
|
|
While there, also check for availability first, to avoid duplicated
error messages (from git_cas and the caller).
(cherry-picked from 8a390462c84f656bdceceb168dc55759256a015e)
|
|
When garbage collecting the oldest generation, we expect it to be
a non-empty directory. Therefore, remove it recursively.
(cherry-picked from 48c2382218a5af10177e68ba24091e77f1e045e6)
|
|
The "generic" rules deliberately resolves conflicts on identical
paths in a latest-wins fashion (seeing all artifacts as later than
all runfiles) to allow an easy way to define actions. However, the
inputs stage obtained by this resolution can still contain conflicts
and those are an error. Properly detect those. Also clarify in the
documentation, that only conflicts on identical paths are resolved
in the described priority, not semantic overlap.
(cherry-picked from 5e104a526cf76fe75312d2fd288a3c88f506fb0a)
|
|
|
|
|
|
...which implicitly also enforces non-absolute.
(cherry-picked from 311c88641e3b4be067ed6575369b401288e8984c)
This ensures that there is no access outside the root directory of
a Git checkout or the temporary unpack directory of an archive.
|
|
(cherry-picked from a8b50ad395b72c6933c164d064d86d60cd6b594c)
|
|
This is an amendment to the changes in commit 8234079, as the
underlying issue was only partially solved there. While the call to
GitRepo::InitAndOpen is in itself properly guarded, it does not
share a lock with the call to create the path to the Git cache if
it is missing.
Fix this by moving the call to the method ensuring the Git cache
initialization to after acquiring the Git cache root garbage
collector shared lock.
(cherry-picked from 7d0a4df3822ca194d93abb6b65d0ebb264cc1974)
|
|
Similarly to just-mr, each SourceTree RPC must ensure that the Git
cache folder exists and the Git cache repository is initialized
before using it.
While there, fix missing shared locks on the Git cache root.
(cherry-picked from 82340791f4362a3a92ea1dfa9ff111c1258be19f)
|
|
If fetching via the primary API failed and there is no fallback,
we should fail rather than tacitly continuing with the next object
to fetch.
(cherry-picked from 74096dce2ee85ea5df940155fc2717d249e14d80)
|
|
...and access internal state via getters.
(cherry-picked from ca952159e778f0ed927082832a195842f6229a94)
|
|
...since there are no unique_locks any more.
(cherry-picked from 9a164558010af84d834dee86f1929bd6582a1ccb)
|
|
...and remove GuardedRepo.
(cherry-picked from d9f39250d302152d19a0aacd76eabae7a013f1a8)
|
|
(cherry-picked from 7b50ad08180edb160d023ed61518cd9256f65f70)
|
|
...and use it in GitRepo to set custom backends.
(cherry-picked 99860b78304817c4d5b27ec4c661b733e30a430a)
|
|
(cherry-picked from f40780393d68e2ebb866d8b941b5a30f0ea5f0de)
|
|
...and fix a potential memory leak in the try-catch for std::filesystem::absolute.
(cherry-picked from 44ec2679c68cbe6141b76d9758b5986725a62d91)
|
|
(cherry-picked from cdd3a6a777a36ff98b60cf47c91281a69c39c4e2)
|
|
(cherry-picked from d9f39250d302152d19a0aacd76eabae7a013f1a8)
|
|
|
|
|
|
(cherry-picked from 59a485598d1a57b78fb60fe7df7dfe08a1cadd83)
|
|
|
|
(cherry-picked from af759f79b9a04ea914edf35037951c1fe993d824)
|
|
(cherry-picked from 9dc61e9faca5e8b05a1a2a2eed83a5468aeb6202)
|
|
(cherry-picked from a9899714c61b998f408e36d870ad8408ec780590)
|
|
... by using an exclusive lock. A lock of which only ever shared instances
are requested has no synchronisation effect. Fix this.
(cherry-picked from dfb7ad5c7d5dcca13d9728534434079a2b60bdea)
|
|
... regardless of success. If traversing fails, we should just
return failure. In this way, we can also avoid an unnecessary
else-branch. While there, always return normally for tarverse,
avoiding direct exits.
(cherry-picked from 3583ed73a269d7467f2b485bf345a0b70cc1b279)
|
|
|
|
|
|
|
|
As we write the first message about the actual upload of a blob,
we should use future rather than past tense. Also, again at trace
level, add a message of successful completion, if we succeeded.
|
|
|
|
The specification for this status code is as follows.
One or more errors occurred in setting up the action requested,
such as a missing input or command or no worker being available.
The client may be able to fix the errors and retry.
We routinely ensure all inputs are available to the remote execution
before we start an action, so all prerequisites will be there on a
compliant server, however might not actually be on a server where
the CAS only has eventual consistency or is incorrect (due to old
cache entries on CAS purge) in its answer to FindMissingBlobs.
While we have no guarantee that a retry will help, we still retry;
at least in the case of an unavailable worker or CAS entries not
yet available due to eventual consistency, this will help. Also,
we log at debug lvel the full response, including the repeated Any
message. In this way, we can find out what useful information (if
any) is sent by popular remote-execution services and implement
more specific mitigations in the future.
|
|
In BatchUploadBlobs we accept short writes and, in case of no progress,
fall back to single blob upload. Therefore, failure to upload blobs
is not fatal and therefore should not be reported at error level.
Decrease the log level accordingly: a protocol failure to upload is
a performance-related event (as the retry needs additional time),
catching an internal exception is something that shouldn't really
happen, so we warn the user.
|
|
In some situations, we use a retry strategy, but in case of complete
failure have another way of attempting that task. In this case, we
should not emmit an error message right away. Add support for this.
|
|
... instead of relying on those dependencies being pulled in
indirectly.
|
|
...by using the new local api that can handle any remote endpoint,
irrespective of protocol.
Also ensure all tests for the serve service are now being run both
in native and compatible modes.
|
|
...irespective of the used protocol.
This api is useful in enabling just-mr and the SourceTree service
of just serve to interact seamlessly with any remote-execution
endpoint.
|
|
When returning digests in responses, ensure they are of the type
the remote would know. Compatible digests can be found vis the
stored mappings from Git digests to bazel objects.
|
|
...instead of passing just the Git hash, which imposes the remote
to always be native.
The serve service proto file is updated accordingly.
|
|
...to be able to interrogate remotes irrespective of protocol.
When serve endpoint is active, it will provide the correct digest
with which to interrogate the remote endpoint. Otherwise, for a
compatible remote check the file mappings for the correct digest.
The serve service proto file is updated accordingly.
|
|
...besides the simple Git hash, if syncing was done. This way one
can know what digest to ask for from the remote. The serve client
also needs to now know what hash function the remote expects.
The serve service proto file is updated accordingly.
|
|
|
|
In just-mr: to instantiate the new Git api instance, both storage
configs, as well as the compatible storage, need to be passed to
the maps. While there, use more explicit naming schemes for the
storage and CAS instances used.
In serve: also acquire gc locks for the local storages when needed
to instantiate the new Git api, which now has access to the CAS.
In all these instances we also pass, as needed, the local api, which
currently still operates only in native mode. This makes no
difference currently, but will ensure less changes needed when the
future compatible-aware local api will be used instead.
|
|
...irrespective of the used protocol.
This api is useful in enabling just-mr and the SourceTree service
of just serve to interact seamlessly with any remote-execution
endpoint.
|
|
...native and compatible, even if currently only native is active.
While there, be more explicit in which storage instance is being
used.
|
|
...to be later passed to maps that might need them.
This is a preparatory commit. Currently only the native storage is
actively being used.
|
|
As just-mr will always create Git roots, be explicit in the names
of variables to state that the default storage created is native.
|
