summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2024-03-18graph_traverser: Remove unnecessary get callPaul Cristian Sarbu
...as it is redundant here and exception prone.
2024-03-18parallelism: Do not spawn more threads than asked toPaul Cristian Sarbu
...in the tool proper, i.e., excepting tests.
2024-03-15Clean up more includes and targetsPaul Cristian Sarbu
Some of the more specific issues addressed: - missing log_level target/include - header-only libs wrongly marking deps as private - missing/misplaced gsl includes
2024-03-14Add end-to-end test with serve building several targetsKlaus Aehlig
... simultaneously, in order to find race conditions during the build.
2024-03-14export of gsl: add missing flexible_config "ENV"Klaus Aehlig
As a library, that target depends implicitly on the toolchain, hence the already present export of "TOOLCHAIN_CONFIG", but also the environment for using that toolchain. So, export "ENV" as well.
2024-03-13Export more third-party top-level entry pointsKlaus Aehlig
... so that they can be served and hence the corresponding dependency can be absent.
2024-03-13BytestreamServer: also use the correct interface function for tmp dirsKlaus Aehlig
... so that any updates of the local-build-root layout are correctly taken into account. In particular, this change also moves the temporary directory under the emphemeral root, allowing more quick clean up. Co-authored-by: Paul Cristian Sarbu <paul.cristian.sarbu@huawei.com>
2024-03-13Fix tmpdir locationKlaus Aehlig
Commit 2ebf355989eb92ac9967eceee0af14d39477afe0 moved the tmpdir creation for various tasks into the task itself. In doing so, TmpDir was called with a relative path; that was, however, is interpreted relative to the working directory, violating the property that our tool never write anything outside the local build root unless explicitly asked to do so (by specifying the output path in an install or install-cas invocation). Fix this, by calling the the tmp-dir function that is storage-layout aware.
2024-03-13Move storage-aware tmpdir creation to configKlaus Aehlig
... as the fs_utils have a lot more dependencies making them usable in less places. Moreover, this function also serves to shape the layout of the local build root and hence is more appropriately placed in the config anyway.
2024-03-12catch2 main: ensure proper build rootKlaus Aehlig
Some of our C-tests under catch2 may rely on the local build root to get a proper tmp directory; ensure this is set properly to avoid falling back to the user's home directory.
2024-03-12storage config: remove unneeded private dependencyKlaus Aehlig
2024-03-12Exports-progress reporter: do not encode sampleKlaus Aehlig
... as it is already encoded to be meaningfully printable.
2024-03-12Export-target progress reporting: report the target nameKlaus Aehlig
... with effective config instead of the actual cache key, which is simply a blob identifier that is probably not so meaningful for the user watching the build.
2024-03-12target evalution: handle error in dependency evaluationKlaus Aehlig
... instead of blindly assuming the evaluation succeeds. Co-authord-by: Paul Cristian Sarbu <paul.cristian.sarbu@huawei.com>
2024-03-12just serve: Improve progress reportingPaul Cristian Sarbu
During analysis it is useful to track and report the progress for all export targets. This is not exclusively linked to a serve endpoint being present, despite most of the time being expected to be spent in export targets being served from the remote endpoint. This commit refactors the current implementation to give proper feedback to the user on the progress of the analysis phase.
2024-03-12serve target: Report the analysed targetPaul Cristian Sarbu
...when orchestrating a build.
2024-03-11target_map: Typo fix in log messagePaul Cristian Sarbu
2024-03-11just: Replace singletons for progress tracking and statistics...Paul Cristian Sarbu
...with regular instances that have controlled life-times. This avoids race conditions in tracking and reporting the results of analysis and build, as the serve endpoint can orchestrate multiple builds at the same time asynchronously. As a bonus side-effect this also ensures the correctness of the progress reporting per orchestrated build.
2024-03-11just analyse: Use explicit target cache instancePaul Cristian Sarbu
The serve endpoint always has to access the correctly sharded target cache, including during analysis. For this purpose, the target cache instance interrogated during analysis has to be explicitly provided.
2024-03-11target cache: Add type aware of bootstrappingPaul Cristian Sarbu
This is needed in order to pass the correctly instantiated TargetCache to AnalyseTarget even when bootstrapping 'just'.
2024-03-11Extend gc end-to-end test to cover --no-rotateKlaus Aehlig
... verifying that - emphemeral directories are removed, - left-over remove-me directories are removed, and - the available content of CAS and cache are not affected.
2024-03-11gc: add option --no-rotationKlaus Aehlig
... to clean up what can be done without losing cache information.
2024-03-11garbage collection: remove ephemeral dataKlaus Aehlig
... and make rotation of generations optional, as with the removal of ephemeral data there is now a useful collection even without rotating generations.
2024-03-11local build root layout: collect all ephemeral directoriesKlaus Aehlig
... under a common root in the youngest generation. This will allow a simple way of cleaning them up during garbage collection.
2024-03-08Remove dead code for host replacementOliver Reiche
2024-03-08doc: Improve description of "preferred hostnames"Oliver Reiche
2024-03-08just-mr: Add test for "preferred hostnames"Oliver Reiche
2024-03-08just-mr: Fix "preferred hostnames"Oliver Reiche
... instead of replacing the host name of each mirror, it should only reorder the list of given mirrors.
2024-03-08mirrors: Add test for SortByHostnameOliver Reiche
2024-03-08mirrors: Implement SortByHostname routineOliver Reiche
2024-03-08CurlURLHandle: Add GetHostname routineOliver Reiche
2024-03-08doc: Update man page on Git file URLsOliver Reiche
2024-03-08just-mr: Normalize paths of Git file URLsOliver Reiche
... and ensure that paths starting with .// remain relative.
2024-03-08just-mr: Report fetch URLs on debug levelOliver Reiche
2024-03-08SystemCommand: Transfer outfile responsibility to callerOliver Reiche
2024-03-08just-mr: Fix shell out executionOliver Reiche
... by avoiding reusing temp dirs for execute. While we are at it, also refactor LocalFetchViaTmpRepo() to create its own empty temp dirs, that cannot be reused by the caller.
2024-03-07just-mr: Fix unguarded optional accessOliver Reiche
2024-03-07TmpDir: Getter should return const refOliver Reiche
... and prohibit moves and move assignments.
2024-03-07Improve documentation for the (internal) test rules with infrastructureKlaus Aehlig
... explaining, in particular, which environment variables are provided by the test-runner framework.
2024-03-07Add test verifying that just-mr only uses the network if necessaryKlaus Aehlig
2024-03-07distdir setup: do not ask remote-excution endpoint for treeKlaus Aehlig
Asking the remote-exeuction endpoint for the tree means, that, upon success, we fetch every archive from the remote execution endpoint, even those that we could get from a provided distdir. As our preference is to only use the network if necessary, simply use the standard way of creating a distdir: create it locally and for each file, try to get it from local resources first, then from the remote-execution endpoint, and finally from other network resources.
2024-03-07Add test verifying reporting levelsKlaus Aehlig
... in particular, verify that no error is reported on successfull command completion.
2024-03-07Down-level log message for errors that can be handledKlaus Aehlig
We deliberately have many functions that do not abort the process on failure and instead simply return a corresponding value. It is then up to the caller to decide how to handle this failure; in particular, such a failure can be expected, e.g., if we try to fetch a file from remote execution first, before fetching it from the upstream location. To have a consistent user experience, nothing that can occur in a successfull build should be reported at error level; moreover, messages that routinely occur during successfull builds should not be reported at progress or above, except for the (stage) result messages and the progress reporter.
2024-03-07serve target: Improved log messagesPaul Cristian Sarbu
2024-03-07serve target: Use CopyFrom to populate Digest message fieldsPaul Cristian Sarbu
2024-03-07exceptions handling: small improvementsPaul Cristian Sarbu
It is better to avoid empty catches when we have a reasonable way to log an (even unlikely) exception, even more so if memory allocations are involved.
2024-03-07just-mr fetch: Correctly report if no fetch requiredPaul Cristian Sarbu
2024-03-07absent_target_map: Improve log messagesPaul Cristian Sarbu
2024-03-07just-mr: Correctly pass client-side auth args to 'just'...Paul Cristian Sarbu
...also for 'analyse' and 'describe'.
2024-03-07just: 'analyse' and 'describe' require client-side authenticationPaul Cristian Sarbu
As these subcommands can access the serve endpoint, it is not enough anymore to just know the remote-execution endpoint, but one must also try to connect to it. Thus, the client-side authentication arguments need to be supported and read. The just.1 man page is also updated accordingly.