summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2024-08-26Add test to check that we ignore Git magic names in treesPaul Cristian Sarbu
2024-08-26install-cas --archive: Fix empty directories not added to archivePaul Cristian Sarbu
Also add empty directory in test script to ensure we don't regress in the future. While there, fix some typos.
2024-08-26GitRepo: Create commit from a directory explicitly...Paul Cristian Sarbu
...by writing its tree directly in the object database instead of working with the index. This allows the creation of trees that contain also entries with 'magic' names, such as the .git folder or .gitignore files. Callers must ensure the given directory only contains the needed entries. In particular, just-mr maps and serve service are updated to separate the import-to-Git repository path from the temporary path containing the content to be committed, to avoid polluting the content path with entries generated on repository initialization.
2024-08-26GitRepo: Add method to create tree from directoryPaul Cristian Sarbu
The tree is traversed bottom-up and each entry is explicitly added to the object database: non-directory entries have their content read and are added as blobs with corresponding permissions based on file type (regular, executable, symlink), while subdirectories are added as trees via CreateTree based on the already added blobs.
2024-08-26GitOps: Pass source directory to GitInitialCommit operationPaul Cristian Sarbu
While there, ensure optional Git operation parameters are checked before use for the operations that require them.
2024-08-26GitRepo: Change logic that creates commits to explicitly give directoryPaul Cristian Sarbu
In preparation for subsequent changes, specify the directory path containing the tree content to be committed explicitly. This change will allow eventually to be able to specify paths that are different from the root path of the repository in which the commit is created. This commit renames and refactors StageAndCommitAllAnnonymous to allow a directory path to be passed. The just-mr and serve service logic is updated such that current behaviour is otherwise unchanged.
2024-08-26GitRepo: Create trees by directly writing to object databasePaul Cristian Sarbu
The libgit2 treebuilder has unnecessary validity checks for tree entries, including for Git-specific magic names (such as '.git'), which cannot be disabled. However, in our tool any filesystem entry should be allowed to be part of a tree. Therefore, the treebuilder-based implementation for CreateTree is replaced by a direct writing of trees, by content, into the underlying repository object database. Additionally, as direct insertion into the object dabase does not check the validity of the tree entries, as was done implicitly by the treebuilder before, add a check for existence of the tree entries into the debug-level generic validity check, with the option to not perform it for ReadTree (as it is unnecessary there).
2024-08-26GitRepo: Update to use specific getter for repository root pathPaul Cristian Sarbu
2024-08-26GitOpParams: Remove unneeded branch fieldPaul Cristian Sarbu
The 'branch' field is deprecated, not being used by any of the critical Git operations, thus it can be removed.
2024-08-26just-mr.py: Accept all tree entries for bootstrapped justPaul Cristian Sarbu
The Python script used for the first stage of bootstrapping just uses the Git index to create trees from directories (be it file repositories, unpacked archives, or distfiles), therefore it has the limitations of Git itself in committing trees that contain entries ignored by Git, such as empty directories, the .git folder, .gitignore files and files referenced there, or other entries with Git-specific magic names. This commit updates the Python script to replace the use of the Git index for importing directories to directly writing the needed blobs and trees to the object database, then commit the resulting top tree explicitly. While there, fix a typing issue from our relaxed approach in using os.environ to set the subprocess env when running commands. As on POSIX the type is _Environ, not simple Dict[str, str], use implicit dictionary merging (Python v3.5+) to set the Git envars. The issue was initially flagged by pyright.
2024-08-23Add test verifying that conflicts in the artifacts stage are recognizedKlaus Aehlig
2024-08-23Rule "generic": verify that the output is a properly formed artifact stageKlaus Aehlig
2024-08-23CasClient: Retry batch update for missing responseOliver Reiche
The remote execution protocol is a bit unclear about how to deal with blob updates for which we got no response. While some clients consider a blob update failed only if a failed response is received, we are going extra defensive here and also consider missing responses to be a failed blob update. Issue a retry for the missing blobs.
2024-08-23Executor: Also report missing output dirsOliver Reiche
2024-08-23Add a test verifying that out_dirs are normalizedKlaus Aehlig
2024-08-23When creating actions, normalize paths in the action definitionKlaus Aehlig
... and also perform conflict check on the normalized paths. Still, the the output of the "ACTION" funtion be keyed by the representation of the paths as originally described, to allow the author of a rule to use non-normalized paths as well.
2024-08-23test gc-repo: forward PATH in launcherKlaus Aehlig
... so that we can run with whatever ambient path is present rather than relying on standard paths pulled in by env.
2024-08-22Use lint defaults as in rulesKlaus Aehlig
... inheriting path from CC and shell defaults.
2024-08-22Add ["lint", "defaults"] to provide paths for lint actionsKlaus Aehlig
... and honor those in ["lint", "targets"].
2024-08-22Remote-execution properties: restore the latest-wins semanticsKlaus Aehlig
... that was accidentially replaced by a first-wins semantics in 62d204ff4cc94c12c1635f189255710901682825 which fortunately did not make it to any release.
2024-08-21["test", "suite"]: document fieldsKlaus Aehlig
2024-08-21tests: use newly defined test suiteKlaus Aehlig
... so that linting information gets propagated properly.
2024-08-20["utils/serve_service", "CC test"]: honor LINTKlaus Aehlig
2024-08-20["utils/remote-execution", "CC test"]: honor LINTKlaus Aehlig
2024-08-20Add ["test", "suite"]Klaus Aehlig
... collecting test results similar to the way "install" rules are typically used; however in such a way that the provider "lint" is forwarded.
2024-08-20["shell/test", "script"] forward lint information of depsKlaus Aehlig
2024-08-20["CC", "test"]: support providing lint informationKlaus Aehlig
While there, also properly transition "srcs" and "private-hdrs" to the host version.
2024-08-20["lint", "targets"]: deduplicate tasksKlaus Aehlig
In case many "targets" are given, the field "lint" will contain all the concatenation of the provider "lint" of the given targets. There is, however, not need to lint the same file in the same context twice, so deduplicate the targets first. While this does not change the amount of lint actions carried out (as equal actions are handled only once anyway), it keeps the summary clean by not having dulicate entries.
2024-08-14expression language: add nub_leftKlaus Aehlig
Originally, the expression lanuage only contained a function to deduplicate a list, keeping only the right-most occurence. The reason was that this is the order needed for linking: a library providing an open symbol has to come on the command line after the library using that symbol (and hence making it an open symbol). However, by now use cases have emerged that require a topological sorting where definition comes before use; also, when composing the value of PATH from fragments, we usually want to keep the first occurrence in order for it to take precedence. Therefore, also add "nub_left" as built-in function, allowing a more condense (and slightly more efficient) description in rules instead of the revserse-nub_right-reverse pattern.
2024-08-14absent_target_map: fix private hdrsKlaus Aehlig
... and, in particular, do not include headers of other libraries that are not even needed.
2024-08-14lint: use standard launchersKlaus Aehlig
2024-08-14new rule ["lint", "targets"]Klaus Aehlig
2024-08-14Make CC binary and library support providing information for lintingKlaus Aehlig
If the configuration variable "LINT" is set, also provide information on compile actions and header files (with preprocessing as described command, in particular also providing the correct flags) in correct dependency context. In this way, lint rules can request the needed information for performing their checks.
2024-08-07Remove code duplication from SourceTreeServiceMaksim Denisov
2024-08-07Remove code duplication from CasServerMaksim Denisov
2024-08-07Pass a longrunning operation to ExecutionServer::WriteResult by rvalueMaksim Denisov
2024-08-07ExecutionServer: use one method for conversion to IExecutionActionMaksim Denisov
2024-08-07ExecutionServer: use one method for conversion to bazel ExecuteResponseMaksim Denisov
2024-08-07ExecutionServer: Move creation of ActionResult to a static functionMaksim Denisov
2024-08-07ExecutionServer: move creation of bazel instances to functionsMaksim Denisov
2024-08-07Remove ExecutionServiceImpl::StoreActionResult methodMaksim Denisov
...and move the storing logic to Execute directly. There is no need to pass additional parameters to the method to just perform a check inside, and after this removal there is no need to preserve a one-line method.
2024-08-07Use expected to return errors from ExecutionServiceImpl's methodsMaksim Denisov
2024-08-07ExecutionServer: remove redundant scopesMaksim Denisov
2024-08-07Refactor read/write logic in BytestreamServerMaksim Denisov
1. In reading remove additional buffer field and reuse the response's buffer; 2. In writing preserve the file descriptor alive.
2024-08-07Extend BazelNetworkReader to avoid redundant conversionsMaksim Denisov
2024-08-07Generate bazel trees in LocalCasReaderMaksim Denisov
...and use this functionality in ExecutionServer
2024-08-07Remove HashFunction field from LocalACMaksim Denisov
...and get it from LocalCAS.
2024-08-07Avoid deep copies of containers in responses.Maksim Denisov
2024-08-07Remove code duplication in Populate method in responses.Maksim Denisov
2024-08-07Remove unused code from Artifact::ObjectInfoMaksim Denisov
Deserialization from json is used in a single test only