| Age | Commit message (Collapse) | Author |
|---|
|
|
|
... by avoiding reusing temp dirs for execute. While we are
at it, also refactor LocalFetchViaTmpRepo() to create its
own empty temp dirs, that cannot be reused by the caller.
|
|
|
|
... as this is the only part this map depends on, and not the
information on what to do with the archive, like style of unpacking,
etc. In this way, this map will also be reusable for fetching
foreign files.
|
|
In order to set up roots, just-mr is able to interrogate, if given,
serve and/or remote-execution endpoints. However, just-mr operates
only with Git hashes, i.e., with a native mode CAS.
This commit ensures the correct interactions occur between just-mr
and the provided endpoints not only in native mode, but also in
comaptible mode, where a serve endpoint might be present even if
one cannot make use of its associated remote-exection endpoint.
The user always gets informed if any incompatibilities are
detected.
|
|
just-mr should always operate with the CAS location corresponding
to the native protocol, i.e., using Git hashes. This way all the
checks and transactions between local CAS and the Git cache are
correct.
The commit fixes the issue by ensuring we don't set globally the
compatibility mode or hashing function if being passed the
--compatible flag, as this flag should only be used to check
comaptibility with any given remote endpoint and not affect the
local CAS location used by just-mr.
|
|
For archive repositories we need to ensure that a non-absent root
is backed by an archive content blob in the local CAS, in order to
also keep the proper root tree file associations. This change also
simplifies the content_cas_map logic by removing the previous
separation of implementation logic between fetching and setting up
the workspace root.
|
|
When fetching git repositories, just-mr routinely shells out to
git. In this case, allow the user to specify via "inherit env",
which environment variables from the host environment should be
made available in this action. Typical variables to inherit are
ones providing credentials, like SSH_AUTH_SOCK. As the repository
description specifies the commit that will be taken, and hence the
resulting tree, correctness is not affected by the environement
leaking in here.
|
|
As the serve service always has an associated remote-execution
endpoint, when just-mr gets passed only a serve endpoint address it
will now default to implying a remote-execution endpoint exists too
at that address. Additionally, now the implementation of the
--backup-to-remote option more clearly shows that it can only be
done in native mode.
|
|
For archives and Git repositories we should ensure that not finding
the witnessing entity (archive content blob or Git commit,
respectively) results in a distinct status in the response to a
request that sets up roots on the serve endpoint. This will allow
just-mr to better handle its interaction with the serve endpoint.
|
|
To take advantage of absent roots, we need to ensure that a given
serve endpoint can build against the tree of this generated root.
To this end, for an 'archive' repository we only set the root as
absent if the serve endpoint knows already the root, it can set it
up itself, or we can create the root locally without a network
fetch and then upload it to the serve endpoint via the remote CAS.
A network fetch never gets performed for an absent root.
If a serve endpoint is not provided, an absent root can still be
generated, but only if no network fetches are required. In this
case a warning is emitted.
|
|
To take advantage of absent roots, we need to ensure that a given
serve endpoint can build against the tree of this generated root.
To this end, for a 'git tree' repository we only set the root as
absent only if the given serve endpoint has this root, or the tree
is known locally and can be provided via the remote CAS. While
generating an absent root the fetch command will never be called.
Generating an absent root without being provided a serve endpoint
is still allowed, but results in a warning.
|
|
This is to uphold the coding style guide we employ.
|
|
The change added to interrogate also the local CAS for the tree of
a 'git tree' repository failed to add support for backing up such
tree found into the remote CAS. This commit fixes the issue.
|
|
Now we look for the tree also in the local CAS, not just in the
local Git cache. If found, we import the tree from local CAS to the
Git cache and continue as usual.
|
|
Now we look for the content blob also in the local Git cache, not
just in local CAS. If found, we store the blob read from Git cache
into local CAS and continue as usual.
|
|
This fixes the multiple instances of (mainly) missing or wrong
includes of standard, third-party, and own headers in the
other_tools source folder.
|
|
This includes implementing the logic for the --backup-to-remote
and --fetch-absent options.
|
|
|
|
|
|
As preparatory step for extending the types of repositories that
are targeted by a just-mr fetch subcommand, names of certain
variables and of the currently only fetch operation async map were
changed to clarify that they only target archives.
|
|
As this map is supposed to work as an all-or-nothing guarantee of
content being in CAS, we can replace the bool return type with the
null type instead.
|
|
|
|
Before trying to fetch an archive content from other non-local
sources, check first the just serve endpoint.
|
|
...before going to the network, if 'just serve' endpoint given.
ContentCASMap was reworked to provide a cleaner logic for handling
pure fetches, which also reduced code duplication in ContentGitMap.
|
|
|
|
Also extends 'distdir' repositories logic accordingly.
|
|
|
|
This is required in order to make them available to 'just serve'
in a minimal just installation.
|
|
|
|
After successfully fetching data over the network for an archive,
if the optional checksums are not provided, the code will throw if
there is a mismatch between the hash of the fetched data stream
and the provided content hash.
This commit fixes the issue by adding an additional check which
properly handles the possible mismatch.
|
|
...via an 'absent' pragma in repository descriptions.
For 'git'-type repositories, first interrogates a 'just serve'
remote, if given, before reverting to fetching from the network.
|
|
The IsAvailable method with the correct signature is now called.
|
|
|
|
...for archives not already in local CAS.
|
|
For 'file' and 'archive' checkouts we will handle more than just
the 'ignore' value.
|
|
|
|
This feature has been introduced with C++20.
|
|
|
|
...and make the use of std::hash consistent.
This will make it easier to remove the fix once the libc
implementation we use catches up with the C++ standard.
|
|
When deciding if two archive-based repositories create the same root
it is not enough to compare the contents of the archive; it also
matters which subdirectory we will take as root. This is especially
important, if we take the same archive on the one hand for the code
and also the example subdir for testing.
Additionally, we cannot ignore the archive type; there are tar
archives that are also zip archives, but with different content. (Tar
only cares about the initial segment till it finds the two empty
blocks; zip has the index at the end and ignores any initial segment
that might have been prepended to a valid zip archive.)
|
|
In this way, we have the whole layout of the local build root
consolidated in one place. Moreover, in this way, the location
of the git root is also available to the build tool itself and
can, e.g., be used as fallback CAS.
|
|
In the rare cases that we need to shell out to git, let the user
configure what binary to use. Option resolves in the same way as
the just executable, including allowing it to be set via just-mrrc.
Updates all cases of shelling out to git (fetch and commit update).
Update just-mr and just-mrrc docs accordingly.
|
|
...to ensure the temporary directories exist before they are needed,
as expected by those methods. This way, the TmpDir class takes care
to also clean up after itself.
Also, pass the local launcher to the methods that now shell out.
|
|
...due to limitations in SSH support in libgit2. In this case, we
simply execute 'git ls-remote <repo> [<branch>]' and then parse
the output. Remote interogation requires no local repository, so
it is an asynchronious operation by default.
|
|
...due to limited SSH support in libgit2. In order to allow the
fetches to still be parallel, we execute:
git fetch --no-auto-gc --no-write-fetch-head <repo> [<branch>]
This only fetches the packs without updating any refs, at the slight
cost of sometimes fetching some redundant information, which for our
purposes is practically a non-issue.
(If really needed, a 'git gc' call can be done eventually to try to
compact the fetched packs, although a save in disk space is not
actually guaranteed.)
|
|
|
|
To avoid unnecessary work, just-mr uses on-disk caches, including
for the mapping of a distdir to the corresponding git tree. This,
however, implies that the number of repositories that are actually
considered varies: in order to fetch a distdir repository, all
involved archives have to be fetched, but if we have a cache hit
none of them is even looked at.
So, in order to have a consistent reporting only count top-level
targets (i.e., the reachable repositories) in the statistics,
not the archives implicitly contained in a distdir, nor low-level
sub tasks. The actual fetch acitvity is shown separately by the
task tracker.
|
|
The improved GC implementation uses refactored storage
classes instead of directly accessing "unknown" file paths.
The required storage class refactoring is quite substantial
and outlined in the following paragraphs.
The module `buildtool/file_system` was extended by:
- `ObjectCAS`: a plain CAS implementation for
reading/writing blobs and computing digests for a given
`ObjectType`. Depending on that type, files written to the
file system may have different properties (e.g., the x-bit
set) or the digest may be computed differently (e.g., tree
digests in non-compatible mode).
A new module `buildtool/storage` was introduced containing:
- `LocalCAS`: provides a common interface for the "logical
CAS", which internally combines three `ObjectCAS`s, one
for each `ObjectType` (file, executable, tree).
- `LocalAC`: implements the action cache, which needs the
`LocalCAS` for storing cache values.
- `TargetCache`: implements the high-level target cache,
which also needs the `LocalCAS` for storing cache values.
- `LocalStorage`: combines the storage classes `LocalCAS`,
`LocalAC`, and `TargetCache`. Those are initialized with
settings from `StorageConfig`, such as the build root base
path or number of generations for the garbage collector.
`LocalStorage` is templated with a Boolean parameter
`kDoGlobalUplink`, which indicates that, on every
read/write access, the garbage collector should be used
for uplinking across all generations (global).
- `GarbageCollector`: responsible for garbage collection and
the global uplinking across all generations. To do so, it
employs instances of `LocalStorage` with `kDoGlobalUplink`
set to false, in order to avoid endless recursion. The
actual (local) uplinking within two single generations is
performed by the corresponding storage class (e.g.,
`TargetCache` implements uplinking of target cache entries
between two target cache generations etc.). Thereby, the
actual knowledge how data should be uplinked is
implemented by the instance that is responsible for
creating the data in the first place.
|
|
|
