summaryrefslogtreecommitdiff
path: root/src
AgeCommit message (Collapse)Author
2024-08-27Bytestream server: verify hash of uploaded objectKlaus Aehlig
... and correctly report the error. - If we cannot store the bytes we received, this is an internal error. - If the bytes received have a different hash than announced, report this user error as INVLID_ARGUMENT.
2024-08-27Reformat code to comply with clang-format 18Klaus Aehlig
... while keeping our .clang-format file.
2024-08-26Store LocalAC keys as ArtifactDigestsMaksim Denisov
2024-08-26Split serialization/deserialization logic in LocalACMaksim Denisov
...into writing/reading to CAS and writing/reading entries (to remove code duplications in uplinking and obtaining the result). The read functionality cannot be replaced by a single function since the CAS key of the entry is required during uplinking. Although the write functionality can, it is also separated into two parts to keep the reverse logic clear.
2024-08-26Reduce the size of large object entries.Maksim Denisov
Removed named keys of entries (hash, size). These prefixes were added for every chunk creating an additional overhead without any benefit. Removed prefixes of hashes (*62*hash) appearing in native mode and creating an additional overhead as well: it is known that all of them are blobs.
2024-08-26install-cas --archive: Fix empty directories not added to archivePaul Cristian Sarbu
Also add empty directory in test script to ensure we don't regress in the future. While there, fix some typos.
2024-08-26GitRepo: Create commit from a directory explicitly...Paul Cristian Sarbu
...by writing its tree directly in the object database instead of working with the index. This allows the creation of trees that contain also entries with 'magic' names, such as the .git folder or .gitignore files. Callers must ensure the given directory only contains the needed entries. In particular, just-mr maps and serve service are updated to separate the import-to-Git repository path from the temporary path containing the content to be committed, to avoid polluting the content path with entries generated on repository initialization.
2024-08-26GitRepo: Add method to create tree from directoryPaul Cristian Sarbu
The tree is traversed bottom-up and each entry is explicitly added to the object database: non-directory entries have their content read and are added as blobs with corresponding permissions based on file type (regular, executable, symlink), while subdirectories are added as trees via CreateTree based on the already added blobs.
2024-08-26GitOps: Pass source directory to GitInitialCommit operationPaul Cristian Sarbu
While there, ensure optional Git operation parameters are checked before use for the operations that require them.
2024-08-26GitRepo: Change logic that creates commits to explicitly give directoryPaul Cristian Sarbu
In preparation for subsequent changes, specify the directory path containing the tree content to be committed explicitly. This change will allow eventually to be able to specify paths that are different from the root path of the repository in which the commit is created. This commit renames and refactors StageAndCommitAllAnnonymous to allow a directory path to be passed. The just-mr and serve service logic is updated such that current behaviour is otherwise unchanged.
2024-08-26GitRepo: Create trees by directly writing to object databasePaul Cristian Sarbu
The libgit2 treebuilder has unnecessary validity checks for tree entries, including for Git-specific magic names (such as '.git'), which cannot be disabled. However, in our tool any filesystem entry should be allowed to be part of a tree. Therefore, the treebuilder-based implementation for CreateTree is replaced by a direct writing of trees, by content, into the underlying repository object database. Additionally, as direct insertion into the object dabase does not check the validity of the tree entries, as was done implicitly by the treebuilder before, add a check for existence of the tree entries into the debug-level generic validity check, with the option to not perform it for ReadTree (as it is unnecessary there).
2024-08-26GitRepo: Update to use specific getter for repository root pathPaul Cristian Sarbu
2024-08-26GitOpParams: Remove unneeded branch fieldPaul Cristian Sarbu
The 'branch' field is deprecated, not being used by any of the critical Git operations, thus it can be removed.
2024-08-23Rule "generic": verify that the output is a properly formed artifact stageKlaus Aehlig
2024-08-23CasClient: Retry batch update for missing responseOliver Reiche
The remote execution protocol is a bit unclear about how to deal with blob updates for which we got no response. While some clients consider a blob update failed only if a failed response is received, we are going extra defensive here and also consider missing responses to be a failed blob update. Issue a retry for the missing blobs.
2024-08-23Executor: Also report missing output dirsOliver Reiche
2024-08-23When creating actions, normalize paths in the action definitionKlaus Aehlig
... and also perform conflict check on the normalized paths. Still, the the output of the "ACTION" funtion be keyed by the representation of the paths as originally described, to allow the author of a rule to use non-normalized paths as well.
2024-08-22Remote-execution properties: restore the latest-wins semanticsKlaus Aehlig
... that was accidentially replaced by a first-wins semantics in 62d204ff4cc94c12c1635f189255710901682825 which fortunately did not make it to any release.
2024-08-14expression language: add nub_leftKlaus Aehlig
Originally, the expression lanuage only contained a function to deduplicate a list, keeping only the right-most occurence. The reason was that this is the order needed for linking: a library providing an open symbol has to come on the command line after the library using that symbol (and hence making it an open symbol). However, by now use cases have emerged that require a topological sorting where definition comes before use; also, when composing the value of PATH from fragments, we usually want to keep the first occurrence in order for it to take precedence. Therefore, also add "nub_left" as built-in function, allowing a more condense (and slightly more efficient) description in rules instead of the revserse-nub_right-reverse pattern.
2024-08-14absent_target_map: fix private hdrsKlaus Aehlig
... and, in particular, do not include headers of other libraries that are not even needed.
2024-08-07Remove code duplication from SourceTreeServiceMaksim Denisov
2024-08-07Remove code duplication from CasServerMaksim Denisov
2024-08-07Pass a longrunning operation to ExecutionServer::WriteResult by rvalueMaksim Denisov
2024-08-07ExecutionServer: use one method for conversion to IExecutionActionMaksim Denisov
2024-08-07ExecutionServer: use one method for conversion to bazel ExecuteResponseMaksim Denisov
2024-08-07ExecutionServer: Move creation of ActionResult to a static functionMaksim Denisov
2024-08-07ExecutionServer: move creation of bazel instances to functionsMaksim Denisov
2024-08-07Remove ExecutionServiceImpl::StoreActionResult methodMaksim Denisov
...and move the storing logic to Execute directly. There is no need to pass additional parameters to the method to just perform a check inside, and after this removal there is no need to preserve a one-line method.
2024-08-07Use expected to return errors from ExecutionServiceImpl's methodsMaksim Denisov
2024-08-07ExecutionServer: remove redundant scopesMaksim Denisov
2024-08-07Refactor read/write logic in BytestreamServerMaksim Denisov
1. In reading remove additional buffer field and reuse the response's buffer; 2. In writing preserve the file descriptor alive.
2024-08-07Extend BazelNetworkReader to avoid redundant conversionsMaksim Denisov
2024-08-07Generate bazel trees in LocalCasReaderMaksim Denisov
...and use this functionality in ExecutionServer
2024-08-07Remove HashFunction field from LocalACMaksim Denisov
...and get it from LocalCAS.
2024-08-07Avoid deep copies of containers in responses.Maksim Denisov
2024-08-07Remove code duplication in Populate method in responses.Maksim Denisov
2024-08-07Remove unused code from Artifact::ObjectInfoMaksim Denisov
Deserialization from json is used in a single test only
2024-08-07Remove unused code from BazelMasgFactoryMaksim Denisov
1. Remove NodeProperties from CreateDirectory; 2. Set digest in CreateFileNode, CreateDirectoryNode.
2024-08-07Pass SymlinksCheckFunc to GitRepo as not_nullMaksim Denisov
+ invoke it only if there are symlinks to check + remove the corresponding runtime check since it is replaced by a compile-time check
2024-08-07Pass to ObjectCAS an optional ExistsFunc using std::optionalMaksim Denisov
...since this is more readable. And adjust the LocalCAS's uplinker creator.
2024-08-07Replace classic C boolean operators with keywordsMaksim Denisov
! => not; && => and, || => or
2024-08-06analyse: extend provides map when switching to action inputKlaus Aehlig
When switching from a target to the artifacts that are the inputs of a particular action, the provides map is also switched to provide additional (besides the inputs) information about the action, in particular the command. Extend this provides map with the remaining information, in particular the working directory.
2024-08-05Expression language: add expression from_subdirKlaus Aehlig
... allowing to select only the keys in a specific subdir, and move the them to top-level.
2024-08-05When using errno, log the actual error messagePaul Cristian Sarbu
For the user it is more useful to see the actual error message, provided by strerror(), than the pure error code.
2024-08-05Add missing cerrno includesPaul Cristian Sarbu
2024-08-05authentication: Properly log failures to open certification filesPaul Cristian Sarbu
2024-08-02just: report completion of analyse earlierKlaus Aehlig
... right after having successfully finished analysing the target. As a side effect, we also get the statistics of the export targets reported on analysis requests. As cached export targets project artifacts to known ones, that information is highly relevant there.
2024-08-02Actions with non-trivial cwd: add empty input tree, if requiredKlaus Aehlig
While our local action execution implicitly creates the specified cwd with the first output file or directory, this behaviour is not mandated by the remote-execution protocol. There, an action definition has to ensure that cwd is a directory implied by the input files. Achieve this, by adding an empty input directory at cwd if this can be done without creating tree conflicts.
2024-08-02generic rule: add support for cwdKlaus Aehlig
2024-08-02rules: make ACTION expression support "cwd"Klaus Aehlig