| Age | Commit message (Collapse) | Author |
|---|
|
Main culprits:
- std::size_t, std::nullptr_t, and NULL require <cstddef>
- std::move and std::forward require <utility>
- unordered maps and sets require respective includes
- std::for_each and std::all_of require <algorithm>
|
|
|
|
Some of the more specific issues addressed:
- missing log_level target/include
- header-only libs wrongly marking deps as private
- missing/misplaced gsl includes
|
|
Commit 2ebf355989eb92ac9967eceee0af14d39477afe0 moved the tmpdir
creation for various tasks into the task itself. In doing so, TmpDir
was called with a relative path; that was, however, is interpreted
relative to the working directory, violating the property that
our tool never write anything outside the local build root unless
explicitly asked to do so (by specifying the output path in an
install or install-cas invocation). Fix this, by calling the the
tmp-dir function that is storage-layout aware.
|
|
... as the fs_utils have a lot more dependencies making them usable
in less places. Moreover, this function also serves to shape the
layout of the local build root and hence is more appropriately
placed in the config anyway.
|
|
|
|
... instead of replacing the host name of each mirror, it
should only reorder the list of given mirrors.
|
|
|
|
|
|
... and ensure that paths starting with .// remain relative.
|
|
|
|
|
|
... by avoiding reusing temp dirs for execute. While we are
at it, also refactor LocalFetchViaTmpRepo() to create its
own empty temp dirs, that cannot be reused by the caller.
|
|
|
|
Asking the remote-exeuction endpoint for the tree means, that,
upon success, we fetch every archive from the remote execution
endpoint, even those that we could get from a provided distdir. As
our preference is to only use the network if necessary, simply use
the standard way of creating a distdir: create it locally and for
each file, try to get it from local resources first, then from the
remote-execution endpoint, and finally from other network resources.
|
|
It is better to avoid empty catches when we have a reasonable way
to log an (even unlikely) exception, even more so if memory
allocations are involved.
|
|
|
|
...also for 'analyse' and 'describe'.
|
|
... after handling absent roots. Otherwise, we violate the condition of
calling the setter precisely once.
|
|
|
|
|
|
|
|
|
|
As serve and just-mr share their caching of description-tree
association, we also have to change this cache. Thanks to json
encoding before hashing, we know that the old and new hash keys
do not overlap, so the change is save. As distdirs also keep the
respective files in the git root, no new downlaoding will happen
either, hence no warning in the CHANGELOG is needed.
|
|
... as this is the only part this map depends on, and not the
information on what to do with the archive, like style of unpacking,
etc. In this way, this map will also be reusable for fetching
foreign files.
|
|
The just serve client-side and API methods, as well as just-mr
utilities should use the noexcept specifier.
|
|
|
|
In order to set up roots, just-mr is able to interrogate, if given,
serve and/or remote-execution endpoints. However, just-mr operates
only with Git hashes, i.e., with a native mode CAS.
This commit ensures the correct interactions occur between just-mr
and the provided endpoints not only in native mode, but also in
comaptible mode, where a serve endpoint might be present even if
one cannot make use of its associated remote-exection endpoint.
The user always gets informed if any incompatibilities are
detected.
|
|
just-mr should always operate with the CAS location corresponding
to the native protocol, i.e., using Git hashes. This way all the
checks and transactions between local CAS and the Git cache are
correct.
The commit fixes the issue by ensuring we don't set globally the
compatibility mode or hashing function if being passed the
--compatible flag, as this flag should only be used to check
comaptibility with any given remote endpoint and not affect the
local CAS location used by just-mr.
|
|
This commit fixes the invariant that a file association between
a Git commit and the root tree should only be set if that tree is
found in our own Git cache. This ensures consistency between
present and absent roots and in the interaction with the serve
endpoint.
|
|
For archive repositories we need to ensure that a non-absent root
is backed by an archive content blob in the local CAS, in order to
also keep the proper root tree file associations. This change also
simplifies the content_cas_map logic by removing the previous
separation of implementation logic between fetching and setting up
the workspace root.
|
|
When fetching git repositories, just-mr routinely shells out to
git. In this case, allow the user to specify via "inherit env",
which environment variables from the host environment should be
made available in this action. Typical variables to inherit are
ones providing credentials, like SSH_AUTH_SOCK. As the repository
description specifies the commit that will be taken, and hence the
resulting tree, correctness is not affected by the environement
leaking in here.
|
|
At some point we have to decide if a given git repository URL is
a path. So far, we assumed that anything not starting with ssh://,
http://, or https:// is a path. This ignores the facts that
- the file:// scheme, while referring to a file, does not denote
a relative path starting file://,
- the [user@]host:path scheme is not a path on the local machine,
- there exist the URL schemes git://, ftp://, and ftps://, and
- future extension might add additional schemes.
To also correctly handle new schemes that git might add (which we
indeed can handle, as we simply shell out to the git binary), we
reverse the approach: we give the user the means to unambigiously
specify that they refer to a path on the local machine, by either
- using the file:// scheme,
- providing an absolute path starting with /, or
- providing a relative path starting with ./
All other schemes will not be modified. The file scheme, as well
as the git://, http://, and https:// scheme, are handled interally
using libgit2; all others are passed on to git in an unmodified form.
|
|
|
|
... of all produced binaries, including the intermediate
ones: protoc and grpc_cpp_plugin.
|
|
... as, for absent repositories, we need to get the description from
the serve end point. As a consequence, also support -r and --compatible,
as the remote-execution endpoint needs to fit with the one for the
serve endpoint.
|
|
... as even this command now adds entries to CAS, e.g., the shard
for the target-level cache. Also, this command block gc by keeping
a lock (also in the local build root).
|
|
As the serve service always has an associated remote-execution
endpoint, when just-mr gets passed only a serve endpoint address it
will now default to implying a remote-execution endpoint exists too
at that address. Additionally, now the implementation of the
--backup-to-remote option more clearly shows that it can only be
done in native mode.
|
|
The requests to retrieve the tree of a commit, archive, or distdir
also set up those trees in a way that the serve endpoint can later
build against them, besides allowing just-mr to set up roots
locally. Therefore, if the witnessing entity (Git commit, content
blob, or distdir, respectively) is known to the serve endpoint,
then failing to set up the root tree there should result in a
failure also of the just-mr setup on the client side.
|
|
For archives and Git repositories we should ensure that not finding
the witnessing entity (archive content blob or Git commit,
respectively) results in a distinct status in the response to a
request that sets up roots on the serve endpoint. This will allow
just-mr to better handle its interaction with the serve endpoint.
|
|
While just-mr does not use remote-execution properties, it is
still useful to have those as a separate entry in the rc file. With
rc-file delegation, this gives committed rc files an easy way to
specify the image to be used without having to set all the remaining
arguments for the various just subcommands in "just args".
|
|
|
|
|
|
... to pull in rc files from different locations, given by
location objects.
|
|
|
|
... to allow, in a clean way, add computing the effective rc
by overlaying delegated rc files.
|
|
To take advantage of absent roots, we need to ensure that a given
serve endpoint can build against the tree of this generated root.
For a 'distdir' repository we can know the resulting tree
identifier directly without actually needing to fetch anything.
Therefore, we only set the root as absent if the serve endpoint
knows already this tree, if it can set it up itself, or if
we can provide this tree to the serve endpoint from one of our
CAS locations (local or remote), based on our tree invariant
guarantee. A network fetch of the archives never gets performed
for an absent root.
If a serve endpoint is not provided, an absent root can still be
generated, but only if no network fetches are required. In this
case a warning is emitted.
|
|
To take advantage of absent roots, we need to ensure that a given
serve endpoint can build against the tree of this generated root.
To this end, for an 'archive' repository we only set the root as
absent if the serve endpoint knows already the root, it can set it
up itself, or we can create the root locally without a network
fetch and then upload it to the serve endpoint via the remote CAS.
A network fetch never gets performed for an absent root.
If a serve endpoint is not provided, an absent root can still be
generated, but only if no network fetches are required. In this
case a warning is emitted.
|
|
...by passing around the AsyncMap key struct instead of individual
members. This will also make future code changes more easy to
implement and improve code readability.
|
|
To take advantage of absent roots, we need to ensure that a given
serve endpoint can build against the tree of this generated root.
To this end, for a 'git' repository we only set the root as absent
if the serve endpoint knows already the root, it can set it up
itself, or we can create the root locally without a network fetch
and then upload it to the serve endpoint via the remote CAS.
A network fetch never gets performed for an absent root.
If a serve endpoint is not provided, an absent root can still be
generated, but only if no network fetches are required. In this
case a warning is emitted.
|
