summaryrefslogtreecommitdiff
path: root/src/buildtool/file_system
AgeCommit message (Collapse)Author
2024-08-30Replace bazel_re::Digest in ObjectCASMaksim Denisov
...with ArtifactDigest.
2024-08-30Replace bazel_re::Digest in GitRepo::SymlinksCheckFunc callbackMaksim Denisov
...with ArtifactDigest.
2024-08-30Unify symlink checks in git treeMaksim Denisov
2024-08-29FileSystemManager: Implicit copy should not follow symlinksPaul Cristian Sarbu
The default options of std::filesystem::copy include following symlinks, resulting in file repositories creating wrong trees if containing unresolved symlinks, or failing unexpectedly early if symlink cycles existed. This is fixed by ensuring the copy_symlinks option is always used.
2024-08-27Reformat code to comply with clang-format 18Klaus Aehlig
... while keeping our .clang-format file.
2024-08-26GitRepo: Create commit from a directory explicitly...Paul Cristian Sarbu
...by writing its tree directly in the object database instead of working with the index. This allows the creation of trees that contain also entries with 'magic' names, such as the .git folder or .gitignore files. Callers must ensure the given directory only contains the needed entries. In particular, just-mr maps and serve service are updated to separate the import-to-Git repository path from the temporary path containing the content to be committed, to avoid polluting the content path with entries generated on repository initialization.
2024-08-26GitRepo: Add method to create tree from directoryPaul Cristian Sarbu
The tree is traversed bottom-up and each entry is explicitly added to the object database: non-directory entries have their content read and are added as blobs with corresponding permissions based on file type (regular, executable, symlink), while subdirectories are added as trees via CreateTree based on the already added blobs.
2024-08-26GitRepo: Change logic that creates commits to explicitly give directoryPaul Cristian Sarbu
In preparation for subsequent changes, specify the directory path containing the tree content to be committed explicitly. This change will allow eventually to be able to specify paths that are different from the root path of the repository in which the commit is created. This commit renames and refactors StageAndCommitAllAnnonymous to allow a directory path to be passed. The just-mr and serve service logic is updated such that current behaviour is otherwise unchanged.
2024-08-26GitRepo: Create trees by directly writing to object databasePaul Cristian Sarbu
The libgit2 treebuilder has unnecessary validity checks for tree entries, including for Git-specific magic names (such as '.git'), which cannot be disabled. However, in our tool any filesystem entry should be allowed to be part of a tree. Therefore, the treebuilder-based implementation for CreateTree is replaced by a direct writing of trees, by content, into the underlying repository object database. Additionally, as direct insertion into the object dabase does not check the validity of the tree entries, as was done implicitly by the treebuilder before, add a check for existence of the tree entries into the debug-level generic validity check, with the option to not perform it for ReadTree (as it is unnecessary there).
2024-08-26GitRepo: Update to use specific getter for repository root pathPaul Cristian Sarbu
2024-08-07Pass SymlinksCheckFunc to GitRepo as not_nullMaksim Denisov
+ invoke it only if there are symlinks to check + remove the corresponding runtime check since it is replaced by a compile-time check
2024-08-07Pass to ObjectCAS an optional ExistsFunc using std::optionalMaksim Denisov
...since this is more readable. And adjust the LocalCAS's uplinker creator.
2024-08-07Replace classic C boolean operators with keywordsMaksim Denisov
! => not; && => and, || => or
2024-08-05Add missing cerrno includesPaul Cristian Sarbu
2024-07-22Pass HashFunction from StorageConfig to StorageMaksim Denisov
2024-07-22Pass HashFunction to ArtifactDigest::CreateMaksim Denisov
2024-07-22Move ObjectCAS::CreateDigest to the private spaceMaksim Denisov
2024-07-19git_repo: return keep tags on successKlaus Aehlig
... instead of the plain boolean value.
2024-07-12Use static Create functions to construct ArtifactDescriptionMaksim Denisov
...instead of unobvious ctors relying on overload resolution.
2024-07-12Move implementation details of ArtifactDescription to the cpp fileMaksim Denisov
2024-07-05FileSystemManager::CreateFileHardlink: return error code on failureKlaus Aehlig
Instead of returning a plain boolean, return an expected with the same boolean value that in case of an error indicates the error code. In this way, an error-specific handling is possible by consumers. While there, also add proper quoting of the involved file names.
2024-07-05Pass StorageConfig to git_repo and git_repo_remoteMaksim Denisov
2024-07-05Use StorageConfig functionality via Instance()Maksim Denisov
...to track changes during refactoring easier.
2024-07-05Move functionality from StorageConfig to related classes.Maksim Denisov
2024-07-04Mark more constructors 'explicit'Paul Cristian Sarbu
Since c++17 the 'explicit' keyword has use also for constructors with more than one argument and it is recommended to use it by default whereever implicit conversions are not expected bahaviour.
2024-06-28Use (un)expected for Git repoOliver Reiche
2024-05-17FileRoot: Fix parsing of ignore-special Git tree rootsPaul Cristian Sarbu
When populating the GitTree instance stored in a Git tree-type FileRoot with the ignore-special flag set, the GitTree instance would be created with an empty raw_id_ field, signaling that some of the entries might have been skipped and thus the root tree id is not anymore in a one-to-one correspondence with the stored list of entries. This however caused FileRoot instances with missing tree id information. This commit fixes the issue by always storing the raw_id_ field as the root id of the Git tree, as well as clarifying the relationship between this field and the ignore_special_ flag, including refactoring the tree id getters.
2024-05-02Use a named constant length for FileStorage directory namesMaksim Denisov
2024-04-17Compactification: Obtain storage roots from LocalCAS.Maksim Denisov
2024-04-10GitRepo: Add method to keep tree alive by taggingPaul Cristian Sarbu
Also adds an appropriate test for this method.
2024-04-10resolve_symlinks_map: Allow separate source and target repositoriesPaul Cristian Sarbu
In certain cases, e.g., on the serve endpoint, an unresolved tree might lie in a repository other than the Git cache, therefore we cannot create any new entries there, as it would violate our guarantee that we only write under our local build root. Therefore, the resolve_symlinks_map now receives pointers to both the source and target Git databases and ensures that: 1. any tree created on-the-fly is stored exclusively in the target repository, and 2. any other entry required for those trees is made available in the target repository by copying it from the source repository. Note that in our use case the target repository is always our Git cache and passing a pointer to that object database is done to avoid the overhead of otherwise opening the database very often.
2024-04-10git_repo: Add blob writer methodPaul Cristian Sarbu
Also extends the tests accordingly.
2024-04-10git_repo: Improve error message for CreateTreePaul Cristian Sarbu
2024-04-08Use properly included standard library types by defaultPaul Cristian Sarbu
2024-04-08Consistently guard all POSIX C includesPaul Cristian Sarbu
2024-04-02ObjectCAS: Move the method for calculating file digests to the public space.Maksim Denisov
This is needed for LocalCAS's splice routines.
2024-03-26Add missing system includesPaul Cristian Sarbu
Main culprits: - std::size_t, std::nullptr_t, and NULL require <cstddef> - std::move and std::forward require <utility> - unordered maps and sets require respective includes - std::for_each and std::all_of require <algorithm>
2024-03-15Clean up more includes and targetsPaul Cristian Sarbu
Some of the more specific issues addressed: - missing log_level target/include - header-only libs wrongly marking deps as private - missing/misplaced gsl includes
2024-03-13Fix tmpdir locationKlaus Aehlig
Commit 2ebf355989eb92ac9967eceee0af14d39477afe0 moved the tmpdir creation for various tasks into the task itself. In doing so, TmpDir was called with a relative path; that was, however, is interpreted relative to the working directory, violating the property that our tool never write anything outside the local build root unless explicitly asked to do so (by specifying the output path in an install or install-cas invocation). Fix this, by calling the the tmp-dir function that is storage-layout aware.
2024-03-08just-mr: Fix shell out executionOliver Reiche
... by avoiding reusing temp dirs for execute. While we are at it, also refactor LocalFetchViaTmpRepo() to create its own empty temp dirs, that cannot be reused by the caller.
2024-02-16async maps: Create utility library to handle cycle detectionPaul Cristian Sarbu
2024-01-31serve source tree: Increase server-side granularity in response statusesPaul Cristian Sarbu
For archives and Git repositories we should ensure that not finding the witnessing entity (archive content blob or Git commit, respectively) results in a distinct status in the response to a request that sets up roots on the serve endpoint. This will allow just-mr to better handle its interaction with the serve endpoint.
2024-01-08GitRepo: Add blob lookup methodPaul Cristian Sarbu
2023-12-12Filesystem: Fix copy overwrite of symlink with fileOliver Reiche
... and improve log messages in case of failure.
2023-12-05FileRoot: Add ignore-special flag getter and remove unneeded inliningPaul Cristian Sarbu
2023-12-05just main: Move root parser into static FileRoot method...Paul Cristian Sarbu
...to make it available also for setting up 'just serve' builds.
2023-11-30Resolve inconsistencies in third-party headers include formatPaul Cristian Sarbu
2023-11-30Add files to CAS without fully keeping them in memoryKlaus Aehlig
2023-11-30ObjectType: Add noexcept specifierPaul Cristian Sarbu
2023-11-30FileRoot: Fix content description for ignore-special rootsPaul Cristian Sarbu
Ignore-special git-tree-based roots are still content defined, so one should use the correct marker in the JSON description of the root that will be stored in the repository description of target cache keys. This commit fixes the issue and improves documentation.