| Age | Commit message (Collapse) | Author |
|---|
|
When just serve has to create a distdir from files known to it,
there are two kind of hashes involved.
- The requests, as well as the resulting tree always refer to git
identifiers.
- Additionally, files can be stored in the local CAS of the serve
instance. This CAS might use a different hash, e.g., plain sha256.
Therefore, we cannot simply forward the hash from the request to
a lookup in our local CAS; fortunately, when we add a file to the
local CAS, we are told the value of the applicable hash, represented
as bazel digest, hence we can simply remember essentially this
value (converting it to unprefixed form when running in native
mode, as in the following we need the hash as it is part of an
artifact digest).
|
|
... after handling absent roots. Otherwise, we violate the condition of
calling the setter precisely once.
|
|
|
|
|
|
|
|
|
|
As the internal distdir data structure now supports the
executable bit, it is also expressive enough to support
foreign-file repositories. Hence we can use this cache,
getting potentially more cache hits.
|
|
|
|
As serve and just-mr share their caching of description-tree
association, we also have to change this cache. Thanks to json
encoding before hashing, we know that the old and new hash keys
do not overlap, so the change is save. As distdirs also keep the
respective files in the git root, no new downlaoding will happen
either, hence no warning in the CHANGELOG is needed.
|
|
|
|
... as this is the only part this map depends on, and not the
information on what to do with the archive, like style of unpacking,
etc. In this way, this map will also be reusable for fetching
foreign files.
|
|
The just serve client-side and API methods, as well as just-mr
utilities should use the noexcept specifier.
|
|
|
|
|
|
Firstly, in ServeDistdirTree the tree of a distdir should be
synced from the corresponding witnessing Git repository (as is the
case with all root trees), but was wrongly trying to sync from the
local CAS.
Secondly, a status of SYNC_ERROR, according to the protocol, must
always ensure the tree field is set, but some setter calls were
missing.
Thirdly, ServeContent and ServeTree failed to report on
successfully syncing to remote if the blob or tree, respetively,
were uploaded from the local CAS.
Lastly, ServeDistdirTree contained some legacy code sections and
comments that were superfluous.
This commit fixes these issues.
|
|
|
|
In order to set up roots, just-mr is able to interrogate, if given,
serve and/or remote-execution endpoints. However, just-mr operates
only with Git hashes, i.e., with a native mode CAS.
This commit ensures the correct interactions occur between just-mr
and the provided endpoints not only in native mode, but also in
comaptible mode, where a serve endpoint might be present even if
one cannot make use of its associated remote-exection endpoint.
The user always gets informed if any incompatibilities are
detected.
|
|
This request is needed by just-mr in order to verify that it can
use a provided serve endpoint when setting up the repository roots.
|
|
just-mr should always operate with the CAS location corresponding
to the native protocol, i.e., using Git hashes. This way all the
checks and transactions between local CAS and the Git cache are
correct.
The commit fixes the issue by ensuring we don't set globally the
compatibility mode or hashing function if being passed the
--compatible flag, as this flag should only be used to check
comaptibility with any given remote endpoint and not affect the
local CAS location used by just-mr.
|
|
This commit fixes the invariant that a file association between
a Git commit and the root tree should only be set if that tree is
found in our own Git cache. This ensures consistency between
present and absent roots and in the interaction with the serve
endpoint.
|
|
For archive repositories we need to ensure that a non-absent root
is backed by an archive content blob in the local CAS, in order to
also keep the proper root tree file associations. This change also
simplifies the content_cas_map logic by removing the previous
separation of implementation logic between fetching and setting up
the workspace root.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Currently, the implementations of the split and splice operation are both
hidden behind the Bazel API implementation. This was sufficient to implement
splitting at the server and splicing at the client. In order to support the
other direction of splitting at the client and splicing at the server while
reusing their implementations, the code needs to be refactored. First, the
functionality of split and splice are explicitly exposed at the general
execution API interface and implemented in the sub APIs. Second, the
implementations of split and splice are factored into a separate utils class.
|
|
|
|
|
|
|
|
... if provided. This might help users to find the correct place
in their code base causing the action to fail.
|
|
When fetching git repositories, just-mr routinely shells out to
git. In this case, allow the user to specify via "inherit env",
which environment variables from the host environment should be
made available in this action. Typical variables to inherit are
ones providing credentials, like SSH_AUTH_SOCK. As the repository
description specifies the commit that will be taken, and hence the
resulting tree, correctness is not affected by the environement
leaking in here.
|
|
At some point we have to decide if a given git repository URL is
a path. So far, we assumed that anything not starting with ssh://,
http://, or https:// is a path. This ignores the facts that
- the file:// scheme, while referring to a file, does not denote
a relative path starting file://,
- the [user@]host:path scheme is not a path on the local machine,
- there exist the URL schemes git://, ftp://, and ftps://, and
- future extension might add additional schemes.
To also correctly handle new schemes that git might add (which we
indeed can handle, as we simply shell out to the git binary), we
reverse the approach: we give the user the means to unambigiously
specify that they refer to a path on the local machine, by either
- using the file:// scheme,
- providing an absolute path starting with /, or
- providing a relative path starting with ./
All other schemes will not be modified. The file scheme, as well
as the git://, http://, and https:// scheme, are handled interally
using libgit2; all others are passed on to git in an unmodified form.
|
|
We ensure that for each export target to be written to the target
cache all its implied export targets are written to the target
cache first. This ensures that the target cache maintains its
consistency at all times with respect to export target
dependencies.
|
|
|
|
As it is common to use ArtifactDigest objects with a default value
of 0, the std::hash implementation should not take the size into
account. The is_tree member however should be kept in as it always
needs to have a value and has no sensible default.
|
|
|
|
... of all produced binaries, including the intermediate
ones: protoc and grpc_cpp_plugin.
|
|
... glibc provides synchronization stubs for single-threaded
environments as weak symobls. When linking pthreads, these
weak symbols must be replaced by the strong symbols provided
by the pthread library. For dynamically linking pthreads,
this is done automatically. However, to support this for
static linking, we must ensure to link the whole archive.
|
|
... as any unguarded access to non-const members of the same
shared_ptr instance require the use of `atomic_load` and
`atomic_store`.
|
|
... to provide an informative error message on how a rule is related
to a particular import and, in particularly, at which expression
a problem with the import occurred.
While there, also improve the message in the other error case to
follow our standard line-breaking scheme.
|
|
|
|
By showing the full entity name and also adding the usual
newline character after every "While ..." clause.
|
|
|
|
... instead of erroring on missing file. In this way, whenever a
rule or expression from an absent root would have to be read, we
get a meaningful error message and not a complaint about a file
not being there.
|
|
|
|
... as, for absent repositories, we need to get the description from
the serve end point. As a consequence, also support -r and --compatible,
as the remote-execution endpoint needs to fit with the one for the
serve endpoint.
|
|
... as even this command now adds entries to CAS, e.g., the shard
for the target-level cache. Also, this command block gc by keeping
a lock (also in the local build root).
|
|
Also cleans up the logging when parsing the serve service
configuration file.
|
