| Age | Commit message (Collapse) | Author |
|---|
|
As we use chunking also for reducing storage, we have to consider
the overhead of block devices which is in the order of kB per file.
So our target chunk size should be at least 2 orders of magnitude
above this. This suggests to minimally aim for a chunk size of
128kB, a target size that also has the advantage the that maximal
chunk size associated with this size is 1MB which is still well
below the maximal transmission size of grpc allowing us to avoid
the streaming API.
As we're scaling everything up by a factor of 16, we also have
to increase the number of bits in the involved masks by 4. We use
this to also extend the window size by using the 2 most significant
octets. Following the advice of the paper proposing FastCDC to
spread out the ones roughly equally suggests 0x4444 as a suitable
value for the two most significant octets.
We also change the suggested extension of the remote-execution API
accordingly. As the precise parameters for FastCDC when announced
over the remote-execution APIs are still under discussion upstream,
we simplify the name to not mention the target size.
|
|
|
|
Also adds an appropriate test for this method.
|
|
Also extends the tests accordingly.
|
|
|
|
|
|
|
|
For splicing of large objects from external sources additional checks are performed:
* The digest of the spliced result must be equal to the expected digest;
* The parts of a spliced tree must be in the storage.
Tested:
* Regular splicing of large objects;
* If the result is unexpected, splicing fails;
* If some parts of a tree are missing, splicing fails.
|
|
* Uplink parts of the large entry before entry itself;
* Uplink large entries in LargeObjectCAS::GetEntryPath to not split things two times;
* Promote spliced tree during uplinking of a large tree entry to properly promote parts of the tree;
* Uplink large entries in LocalUplink{Blob, Tree} to support proper uplinking in Action Cache and Target Cache;
Tested:
* Uplink large blobs and trees;
* Uplink a large object that depends on other large objects.
|
|
Implicitly reconstruct objects during regular uplinking of Blobs/Trees.
|
|
* Add LargeObjectCAS fields for files and trees to LocalCAS;
* Add logic for splitting objects located in the main storage.
Tested:
Splitting of large, small and empty objects.
|
|
Numerical values are used at some places in justbuild: as value for
timeout scaling, as well as by the "range" expression that is used,
e.g., to define repreated test runs. Therefore, improve support
for numerical values by adding basic operations.
|
|
For an absent export target, the first step of analysis is to ask
serve for the flexible variables. The answer to this request is,
however, independent of the configuration for this target. So we
can avoid calls by caching the answer in an additional map.
|
|
This allows to be explicit and thus have better control on where
messages get logged.
|
|
Some of the more specific issues addressed:
- missing log_level target/include
- header-only libs wrongly marking deps as private
- missing/misplaced gsl includes
|
|
During analysis it is useful to track and report the progress for
all export targets. This is not exclusively linked to a serve
endpoint being present, despite most of the time being expected to
be spent in export targets being served from the remote endpoint.
This commit refactors the current implementation to give proper
feedback to the user on the progress of the analysis phase.
|
|
...with regular instances that have controlled life-times.
This avoids race conditions in tracking and reporting the results
of analysis and build, as the serve endpoint can orchestrate
multiple builds at the same time asynchronously. As a bonus
side-effect this also ensures the correctness of the progress
reporting per orchestrated build.
|
|
The serve endpoint always has to access the correctly sharded
target cache, including during analysis. For this purpose, the
target cache instance interrogated during analysis has to be
explicitly provided.
|
|
|
|
... by avoiding reusing temp dirs for execute. While we are
at it, also refactor LocalFetchViaTmpRepo() to create its
own empty temp dirs, that cannot be reused by the caller.
|
|
... in order to not rely on the default launcher to pull it in.
|
|
... to not rely on env implicitly pulling in echo to PATH.
|
|
... in order to not assume echo to be on the standard search
path pulled in by env.
|
|
... instead of hard-coding it in the example graphs. In this way,
we can honor a provided PATH.
|
|
|
|
... to make tests pass also on systems where /bin/sh does not,
by default, pull in a path to "the standard tools".
|
|
The requests to retrieve the tree of a commit, archive, or distdir
also set up those trees in a way that the serve endpoint can later
build against them, besides allowing just-mr to set up roots
locally. Therefore, if the witnessing entity (Git commit, content
blob, or distdir, respectively) is known to the serve endpoint,
then failing to set up the root tree there should result in a
failure also of the just-mr setup on the client side.
|
|
For archives and Git repositories we should ensure that not finding
the witnessing entity (archive content blob or Git commit,
respectively) results in a distinct status in the response to a
request that sets up roots on the serve endpoint. This will allow
just-mr to better handle its interaction with the serve endpoint.
|
|
...in accordance to our coding style.
|
|
|
|
... that are eligible for caching. In this way, we can accurately keep
track of the dependencies between target-level cache entries. Note
that it is enough to track the export targets eligible for caching,
as no target depending on an ineligible export target can be eligible.
|
|
In order for the serve endpoint to correctly dispatch a build to
the correct remote-execution endpoint, the platform properties and
dispatch list for a build need to be passed explicitly to the
executor (via the graph traverser instance) instead of always being
taken from the RemoteExecutionConfig struct.
This commit implements these changes, including updating existing
tests accordingly.
|
|
...instead of the vector containing the digests of the uploaded
blobs. The returned vector was never inspected by the callers, except
for its size.
The tests have been accordingly amended.
|
|
With the introduction of 'just serve', export targets can now be
built also independently from one another based on their
corresponding minimal repository configuration, as stored in the
target cache key.
In this context, this commit changes the RepositoryConfig usage
from one global (static) instance to pointers passed as necessary
throughout the code.
|
|
|
|
|
|
Absent roots are characterised only by a Git tree hash, so a new
variant of the underlying stored information was added in the form
of a plain string.
In order to avoid unwanted implicit conversions when instantiating
via literal strings, we force callers of the constructors to
explicitly differentiate between plain strings and filesystem
paths. Existing tests were updated to reflect this.
Co-authored-by: Alberto Sartori <alberto.sartori@huawei.com>
|
|
The function ToJson used `file_type` key to express the type of the
artifact, on the other hand, FromJson was expecting `type`.
This patch makes the two functions consistent, prefering `file_type`
for historical reasons.
|
|
- add missing serve_api TARGETS file
- rename service client to align with server naming scheme
- fix inconsistencies in comments between implementation and protocol
|
|
This avoids using the more geenric GitRepoRemote method which
has libcurl as a dependency, something that is not needed for this
Git operation.
|
|
The serve service will communicate with this endpoint when needed,
as well as ensure artifacts it provides are synced with the remote
execution CAS, if requested by the client.
If just-mr is given the --fetch-absent option, it Always produce
present roots irrespective of the 'absent' pragma. For Git repositories
marked with the 'absent' pragma, first try to fetch any commit
trees provided by the serve endpoint from the execution endpoint
CAS, before reverting to a network fetch.
Co-authored-by: Klaus Aehlig <klaus.aehlig@huawei.com>
Co-authored-by: Alberto Sartori <alberto.sartori@huawei.com>
|
|
|
|
|
|
... causing an error with gcc 13.2.0.
|
|
While a foldl is enough to implement a reverse functionality,
adding it as a built in allows doing so in linear time.
|
|
... to obtain from a list of strings a map with those entries
as keys and true as value. In this way, repeated membership tests
in lists can be implemented more efficiently.
|
|
|
|
|
|
|
|
|
