| Age | Commit message (Collapse) | Author |
|---|
|
|
|
Also adds a section in the curl_url test suite.
|
|
|
|
|
|
|
|
|
|
Also extends 'distdir' repositories logic accordingly.
|
|
In order to allow non-fatal retries of fetches, be it from same
remote or not (e.g., mirrors), the handle now reports with a
caller-defined LogLevel.
|
|
|
|
|
|
|
|
While we don't want to fail if the 'checkouts' map values are not
strings, we shouldn't just accept non-string values either,
instead we should warn the user and continue without them.
|
|
|
|
Make sure that all CopyFile, WriteFile, and CreateSymlink
functions properly unlink the target file (if it exists and
overwrite requested) to avoid interferences of the install
command. With this change, the clean up step for install-cas
and the within GraphTraverser can new be omitted.
|
|
In the specification of the action generating a fixed git tree,
also honor "inherit env", i.e., inherit the environment variables
specified in this field from the environment just-mr is invoked in.
As the expected output is fixed ahead of time anyway, this lack of
isolation does not affect correctness.
|
|
... using thread-based parallelism for the blobs of each tree.
|
|
Use parallelism if provided by the build API when synchronizing
artifacts of export targets and when synchronizing artifacts due
to the --remember option. Do so at build parallelism as this the
parallelism suitable for the build API.
|
|
Allow implementations to use a given number of threads to carry out
the synchronisation. In this way parallelism can be achieved even
in situations where batch reading degrades as objects of unknown
size have to be fetched; this situation typically occurs if a tree
object has a large number of direct children that are blobs.
If no implementation is provided, the default implementation is to
fall back to the normal (sequential) CAS synchronisation.
|
|
Typically, the number of export targets is small compared to the
size of the respective export targets. Moreover, export targets are
often chained, resulting in overlapping artifacts that we want to
fetch only once. Therefore, fetch export targets sequentially, giving
room to introduce parallelism in the individual fetch steps later.
While there, also log the beginning of the artifact synchronisation
for the export targets.
|
|
|
|
|
|
map itself
|
|
|
|
|
|
Also adds missing TARGETS file in serve_api folder and ensures
code comments are consistent with the proto file.
|
|
- add missing serve_api TARGETS file
- rename service client to align with server naming scheme
- fix inconsistencies in comments between implementation and protocol
|
|
This is required in order to make them available to 'just serve'
in a minimal just installation.
|
|
This way they can be used by 'just serve'.
|
|
This avoids using the more geenric GitRepoRemote method which
has libcurl as a dependency, something that is not needed for this
Git operation.
|
|
Also fixes a small typo in tree existence checker log messages.
|
|
Co-authored-by: Paul Cristian Sarbu <paul.cristian.sarbu@huawei.com>
|
|
|
|
The cache key for an export target should contain as target name
that of the export target (and its effective configuration) rather
than the exported target. As we computed the repository part of
the cache key for the target included in the key, this was still a
correct cache key except in the case an explicit file reference was
exported (as here, the information that the file was to be taken
rather than the target of the same name got lost). We still fix
this issue by making the implementation match our design (rather
than by including the file-reference bit in the cache key), as the
original design gives the cleaner protocol for target-level caching
as a service.
|
|
...of the internal cache used for keeping track of running operations.
|
|
...which signal either cient- or server-side failures.
This ensures we exit with a failed network fetch early in cases
where it is clear we won't receive useful data.
|
|
After successfully fetching data over the network for an archive,
if the optional checksums are not provided, the code will throw if
there is a mismatch between the hash of the fetched data stream
and the provided content hash.
This commit fixes the issue by adding an additional check which
properly handles the possible mismatch.
|
|
|
|
In order to not pollute builds, one should only store in the id
file the association between a commit and its *root* tree. As the
need for such a file only happens when asked to fetch absent
roots, which should be a quite rare use case, we can request the
whole tree of a commit to be served (incl. synced with remote
execution) and perform the subdir tree extraction locally instead.
|
|
|
|
|
|
This was causing the remote serve address to overwrite the one set
for remote execution.
Also, to keep things clean, some common remote server-related
methods and definitions were moved into their own library.
|
|
Using absent repositories depends on a just-serve endpoint. As
such, it is not a datum of the project, but one of the available
infrastructure. As the latter can change independently of the
project, it is desriable to have the option to specify those pieces
of information in separate files.
|
|
Just like the remote-execution protocol has several services (Execution,
ActionCache, ContentAddressableStorage, etc), so will the serve
protocol: the actual target-level caching, as well auxilliary
services, like the service to obtain the tree for a given root. Already
follow that scheme, before the protocol gets part of any release.
Also, move the status enum into the respective answer messages. In this
way, we can have different enums for different requests without causing
conflicts on the named enum constants.
|
|
server is started
|
|
... by also keeping the map of commit to tree locally.
|
|
The serve service will communicate with this endpoint when needed,
as well as ensure artifacts it provides are synced with the remote
execution CAS, if requested by the client.
If just-mr is given the --fetch-absent option, it Always produce
present roots irrespective of the 'absent' pragma. For Git repositories
marked with the 'absent' pragma, first try to fetch any commit
trees provided by the serve endpoint from the execution endpoint
CAS, before reverting to a network fetch.
Co-authored-by: Klaus Aehlig <klaus.aehlig@huawei.com>
Co-authored-by: Alberto Sartori <alberto.sartori@huawei.com>
|
|
This functionality will be needed to upload git trees
to a remote-execution end point by `just serve.
|
|
...via an 'absent' pragma in repository descriptions.
For 'git'-type repositories, first interrogates a 'just serve'
remote, if given, before reverting to fetching from the network.
|
|
|
|
|
