| Age | Commit message (Collapse) | Author |
|---|
|
|
|
Commit f5f9be5bc07b16807aceac86fba9212e3889762a changed from
tracking progress by cache key to tracking by configured target;
however, the absent-target map was forgotten. Fix this while also
switching to the shortend export-target representation introduced
in the previous commit.
|
|
As we always analyse export targets in their canonical configuration (and
do count in the progress the mapping from obtained configuraiton
to canonical one), the shortend name (i.e., the representation
with null values dropped form the configuration) is still a unique
representation of the target. Use this in progress reporting to
simplify reading the progress sample.
|
|
|
|
... using the same mechanism that just uses. As just-mr now also
does quite some grpc calls, e.g., fetching distribution files from
the remote-execution endpoint and negociating roots with the serve
endpoint, it may now also be affected by a bad network or a service
temporarily be unavailable.
|
|
... to simplify reuse.
|
|
... to the same value as TEST_TMPDIR. While TEST_TMPDIR is the
conventional name that tests should use for their temporary file,
setting TMPDIR as well has the advantage that more library functions
can be used to create temporary files and directories, as quite a
few of them inspect the variable TMPDIR.
While there, also fix some typos in the documentation strings.
|
|
... instead of inlining the code. In this way, we also have all
the checks that the test runner consists of precisely one artifact.
|
|
... and nothing reconstructed for simple (i.e., non-export) targets.
|
|
During garbage collection split and remove from the storage every entry that is larger than a threshold.
|
|
During garbage collection remove from the storage every entry that has the large entry.
|
|
|
|
|
|
|
|
|
|
|
|
... so that they are available in machine-readable form. In this
way, all logs can automatically be collected without the need of
parsing human-targeted error messages.
|
|
... in a structured way to eventually support machine-readable access to
the identifiers of the log files.
|
|
... in logger extension.
|
|
This means that on the serve endpoint, where a logger for the
GraphTraverser is explicitly set, the log messages during action
execution is also made available to the client via the CAS-stored
analysis and build log blob.
|
|
|
|
|
|
and trees.
|
|
executable files during splitting.
|
|
|
|
Configured targets, by design, cannot distinguish between a value
not occuring in the configuration and occuring there with value
null. Therefore, to understand the conflict, we can as well drop
all the null values of the target configuration when reporting it.
|
|
- update boringssl to commit ae72a45
- update protobuf to v25.1
- update abseil to v20240116.0 (including patches)
Also add mirrors for third-party dependencies where known and
hashes correspond.
|
|
|
|
As we use chunking also for reducing storage, we have to consider
the overhead of block devices which is in the order of kB per file.
So our target chunk size should be at least 2 orders of magnitude
above this. This suggests to minimally aim for a chunk size of
128kB, a target size that also has the advantage the that maximal
chunk size associated with this size is 1MB which is still well
below the maximal transmission size of grpc allowing us to avoid
the streaming API.
As we're scaling everything up by a factor of 16, we also have
to increase the number of bits in the involved masks by 4. We use
this to also extend the window size by using the 2 most significant
octets. Following the advice of the paper proposing FastCDC to
spread out the ones roughly equally suggests 0x4444 as a suitable
value for the two most significant octets.
We also change the suggested extension of the remote-execution API
accordingly. As the precise parameters for FastCDC when announced
over the remote-execution APIs are still under discussion upstream,
we simplify the name to not mention the target size.
|
|
... as the typical chunk size is mainly determined by the masks
used internally. So, as long as we hard code them, we should be
honest to ourselves and accept that the chunking parameters are
hard-coded as well.
|
|
... as this is the only thing the user cares about when trying
to investigate why that action failed.
|
|
... with only the non-null entries of the configuration. This
information is enough for the user to build this target, e.g.,
when searching for the cause of a build failure.
|
|
... by removing from an object the outer keys where the value is null.
|
|
As `just serve` is used like a daemon it can be desirable to restrict
stderr, e.g., to only errors, while keeping a detailled log of the
activity in a file.
|
|
|
|
|
|
|
|
Messages on the command line can be more disturbing than, e.g.,
in a log file. In particular, for debugging it often is useful
to have very verbose logs. In order to have the command-line
experience manageable also in this cases, support restricting the
command-line logging further. In this way, while interacting with
concise command-line messages, verbose logs are still written for
later analysis.
|
|
... to simplify set ups where configuration files are
provided as symbolic links to some central store.
|
|
|
|
The association map file for a resolved tree was supposed to
guarantee that the respective tree is kept alive in a Git
repository as part of a tagged commit.
This commit fixes this issue by tagging the tree (found in the Git
cache after resolution) before writing its association file.
|
|
Also improves and extends accordingly the Git operations tests.
|
|
Also adds an appropriate test for this method.
|
|
In certain cases, e.g., on the serve endpoint, an unresolved tree
might lie in a repository other than the Git cache, therefore we
cannot create any new entries there, as it would violate our
guarantee that we only write under our local build root.
Therefore, the resolve_symlinks_map now receives pointers to both
the source and target Git databases and ensures that:
1. any tree created on-the-fly is stored exclusively in the target
repository, and
2. any other entry required for those trees is made available in
the target repository by copying it from the source repository.
Note that in our use case the target repository is always our Git
cache and passing a pointer to that object database is done to
avoid the overhead of otherwise opening the database very often.
|
|
|
|
This bug went under the radar because the returned pointer is never
explicitly used, just tested if set. As such, the correctness of
just-mr was never actually afected by it.
This commit fixes the issue and also cleans up small
inconsistencies.
|
|
Also extends the tests accordingly.
|
|
If we set up the root for an archive repository as absent, we
should first check if the serve endpoint can set it up for us, and
only then try to provide it from locally available means.
|
|
|
|
|
