| Age | Commit message (Collapse) | Author |
|---|
|
...with regular instances that have controlled life-times.
This avoids race conditions in tracking and reporting the results
of analysis and build, as the serve endpoint can orchestrate
multiple builds at the same time asynchronously. As a bonus
side-effect this also ensures the correctness of the progress
reporting per orchestrated build.
|
|
The serve endpoint always has to access the correctly sharded
target cache, including during analysis. For this purpose, the
target cache instance interrogated during analysis has to be
explicitly provided.
|
|
This is needed in order to pass the correctly instantiated
TargetCache to AnalyseTarget even when bootstrapping 'just'.
|
|
... to clean up what can be done without losing cache information.
|
|
... and make rotation of generations optional, as with the removal
of ephemeral data there is now a useful collection even without
rotating generations.
|
|
... under a common root in the youngest generation. This will allow
a simple way of cleaning them up during garbage collection.
|
|
|
|
... instead of replacing the host name of each mirror, it
should only reorder the list of given mirrors.
|
|
|
|
|
|
... and ensure that paths starting with .// remain relative.
|
|
|
|
|
|
... by avoiding reusing temp dirs for execute. While we are
at it, also refactor LocalFetchViaTmpRepo() to create its
own empty temp dirs, that cannot be reused by the caller.
|
|
|
|
... and prohibit moves and move assignments.
|
|
Asking the remote-exeuction endpoint for the tree means, that,
upon success, we fetch every archive from the remote execution
endpoint, even those that we could get from a provided distdir. As
our preference is to only use the network if necessary, simply use
the standard way of creating a distdir: create it locally and for
each file, try to get it from local resources first, then from the
remote-execution endpoint, and finally from other network resources.
|
|
We deliberately have many functions that do not abort the process
on failure and instead simply return a corresponding value. It
is then up to the caller to decide how to handle this failure;
in particular, such a failure can be expected, e.g., if we try to
fetch a file from remote execution first, before fetching it from
the upstream location.
To have a consistent user experience, nothing that can occur in
a successfull build should be reported at error level; moreover,
messages that routinely occur during successfull builds should not
be reported at progress or above, except for the (stage) result
messages and the progress reporter.
|
|
|
|
|
|
It is better to avoid empty catches when we have a reasonable way
to log an (even unlikely) exception, even more so if memory
allocations are involved.
|
|
|
|
|
|
...also for 'analyse' and 'describe'.
|
|
As these subcommands can access the serve endpoint, it is not
enough anymore to just know the remote-execution endpoint, but one
must also try to connect to it. Thus, the client-side
authentication arguments need to be supported and read.
The just.1 man page is also updated accordingly.
|
|
When just serve has to create a distdir from files known to it,
there are two kind of hashes involved.
- The requests, as well as the resulting tree always refer to git
identifiers.
- Additionally, files can be stored in the local CAS of the serve
instance. This CAS might use a different hash, e.g., plain sha256.
Therefore, we cannot simply forward the hash from the request to
a lookup in our local CAS; fortunately, when we add a file to the
local CAS, we are told the value of the applicable hash, represented
as bazel digest, hence we can simply remember essentially this
value (converting it to unprefixed form when running in native
mode, as in the following we need the hash as it is part of an
artifact digest).
|
|
... after handling absent roots. Otherwise, we violate the condition of
calling the setter precisely once.
|
|
|
|
|
|
|
|
|
|
As the internal distdir data structure now supports the
executable bit, it is also expressive enough to support
foreign-file repositories. Hence we can use this cache,
getting potentially more cache hits.
|
|
|
|
As serve and just-mr share their caching of description-tree
association, we also have to change this cache. Thanks to json
encoding before hashing, we know that the old and new hash keys
do not overlap, so the change is save. As distdirs also keep the
respective files in the git root, no new downlaoding will happen
either, hence no warning in the CHANGELOG is needed.
|
|
|
|
... as this is the only part this map depends on, and not the
information on what to do with the archive, like style of unpacking,
etc. In this way, this map will also be reusable for fetching
foreign files.
|
|
The just serve client-side and API methods, as well as just-mr
utilities should use the noexcept specifier.
|
|
|
|
|
|
Firstly, in ServeDistdirTree the tree of a distdir should be
synced from the corresponding witnessing Git repository (as is the
case with all root trees), but was wrongly trying to sync from the
local CAS.
Secondly, a status of SYNC_ERROR, according to the protocol, must
always ensure the tree field is set, but some setter calls were
missing.
Thirdly, ServeContent and ServeTree failed to report on
successfully syncing to remote if the blob or tree, respetively,
were uploaded from the local CAS.
Lastly, ServeDistdirTree contained some legacy code sections and
comments that were superfluous.
This commit fixes these issues.
|
|
|
|
In order to set up roots, just-mr is able to interrogate, if given,
serve and/or remote-execution endpoints. However, just-mr operates
only with Git hashes, i.e., with a native mode CAS.
This commit ensures the correct interactions occur between just-mr
and the provided endpoints not only in native mode, but also in
comaptible mode, where a serve endpoint might be present even if
one cannot make use of its associated remote-exection endpoint.
The user always gets informed if any incompatibilities are
detected.
|
|
This request is needed by just-mr in order to verify that it can
use a provided serve endpoint when setting up the repository roots.
|
|
just-mr should always operate with the CAS location corresponding
to the native protocol, i.e., using Git hashes. This way all the
checks and transactions between local CAS and the Git cache are
correct.
The commit fixes the issue by ensuring we don't set globally the
compatibility mode or hashing function if being passed the
--compatible flag, as this flag should only be used to check
comaptibility with any given remote endpoint and not affect the
local CAS location used by just-mr.
|
|
This commit fixes the invariant that a file association between
a Git commit and the root tree should only be set if that tree is
found in our own Git cache. This ensures consistency between
present and absent roots and in the interaction with the serve
endpoint.
|
|
For archive repositories we need to ensure that a non-absent root
is backed by an archive content blob in the local CAS, in order to
also keep the proper root tree file associations. This change also
simplifies the content_cas_map logic by removing the previous
separation of implementation logic between fetching and setting up
the workspace root.
|
|
|
|
|
|
|
|
|
