| Age | Commit message (Collapse) | Author | |
|---|---|---|---|
| 2024-07-22 | Pass HashFunction from StorageConfig to Storage | Maksim Denisov | |
| 2024-07-22 | Store HashFunction in StorageConfig | Maksim Denisov | |
| 2024-07-22 | Pass HashFunction to ArtifactDigest::Create | Maksim Denisov | |
| 2024-07-22 | Use HashFunction functionality via Instance() | Maksim Denisov | |
| ...to track changes during refactoring easier. | |||
| 2024-07-22 | Move ObjectCAS::CreateDigest to the private space | Maksim Denisov | |
| 2024-07-19 | Add garbage collection routines for repository roots | Klaus Aehlig | |
| 2024-07-19 | Storage config: change layout to support several storage generations | Klaus Aehlig | |
| 2024-07-16 | TargetCache: Use StorageConfig instance for sharding | Paul Cristian Sarbu | |
| Instead of computing the shard based on the RemoteExecutionConfig singleton, use the already computed hash stored in the passed StorageConfig instance, which now needs to be set up separately if bootstrapping in order to avoid unwanted includes. Storing the backend description to CAS and corresponding audit checks now take place in main. | |||
| 2024-07-16 | StorageConfig: Store also the execution backend description id | Paul Cristian Sarbu | |
| ...such that it will be available to the TargetCache for sharing. Also, GC does not require remote execution information, so the logic for this subcommand is moved earlier in main. | |||
| 2024-07-16 | Create separate library for DescribeBackend | Paul Cristian Sarbu | |
| 2024-07-12 | Store newly spliced executables as copies. | Maksim Denisov | |
| During multithreaded splicing, the main process can be forked (inheriting open file descriptors). In this case, the executable file saved using hardlinking becomes inaccessible. To prevent this, executables must be stored as copies made in a child process. | |||
| 2024-07-05 | Convert StorageConfig to a general class | Maksim Denisov | |
| 2024-07-05 | Convert Storage to a general class | Maksim Denisov | |
| 2024-07-05 | Call uplinking via Uplinker | Maksim Denisov | |
| ... instead of static calls to GarbageCollector | |||
| 2024-07-05 | Implement uplinking logic in a separate class. | Maksim Denisov | |
| 2024-07-05 | Use StorageConfig with generation for initialization of Storage's generations | Maksim Denisov | |
| ...instead of std::filesystem::path. StorageConfig is extended to return paths of Storage's parts. | |||
| 2024-07-05 | Pass Storage and StorageConfig to StorageUtils by reference | Maksim Denisov | |
| 2024-07-05 | Pass StorageConfig to GarbageCollection | Maksim Denisov | |
| 2024-07-05 | Pass StorageConfig to GarbageCollector::SharedLock | Maksim Denisov | |
| 2024-07-05 | Use StorageConfig functionality via Instance() | Maksim Denisov | |
| ...to track changes during refactoring easier. | |||
| 2024-07-05 | Move functionality from StorageConfig to related classes. | Maksim Denisov | |
| 2024-07-05 | Remove ConfigData from StorageConfig | Maksim Denisov | |
| ...since it is an extra class that overcomplicates the config class. | |||
| 2024-07-05 | Use a separate constructor for sharded TargetCache | Maksim Denisov | |
| 2024-07-01 | Reorder compatibility commands in Compactification | Maksim Denisov | |
| 2024-07-01 | Ensure unique paths are used in compactification | Maksim Denisov | |
| 2024-06-28 | Use (un)expected for CAS access | Oliver Reiche | |
| 2024-06-11 | Move creation of TargetCacheKey description to TargetCache | Maksim Denisov | |
| ...to use corresponding Storage for storing auxiliary information. | |||
| 2024-05-27 | Fix headers in local_cas and bazel_network. | Maksim Denisov | |
| 2024-05-27 | Use common interface for stream dumping | Maksim Denisov | |
| ...in LocalApi and BazelApi. | |||
| 2024-05-27 | Use common interface for reading tree entries and leafs | Maksim Denisov | |
| ...in LocalApi and BazelApi. | |||
| 2024-05-02 | Use a named constant length for FileStorage directory names | Maksim Denisov | |
| 2024-04-30 | Compactification: Split files concurrently. | Maksim Denisov | |
| 2024-04-30 | Compactification: Remove spliced files concurrently. | Maksim Denisov | |
| 2024-04-30 | Compactification: Remove invalid files concurrently. | Maksim Denisov | |
| 2024-04-30 | Compactification: Implement a generic concurrent logic. | Maksim Denisov | |
| 2024-04-22 | Compactification: Remove invalid entries from the storage. | Maksim Denisov | |
| During compactification, invalid entries must be deleted. | |||
| 2024-04-17 | Compactification: Split large entries. | Maksim Denisov | |
| During garbage collection split and remove from the storage every entry that is larger than a threshold. | |||
| 2024-04-17 | Compactification: Remove spliced entries. | Maksim Denisov | |
| During garbage collection remove from the storage every entry that has the large entry. | |||
| 2024-04-17 | Compactification: Obtain generation cache directories for compatibility modes. | Maksim Denisov | |
| 2024-04-17 | Compactification: Obtain storage roots from LocalCAS. | Maksim Denisov | |
| 2024-04-15 | LargeBlobs: Skip splicing of dependent objects during uplinking of AC, TC ↵ | Maksim Denisov | |
| and trees. | |||
| 2024-04-15 | LargeBlobs: Make LocalCAS::BlobPathNoSync public to fix synchronization of ↵ | Maksim Denisov | |
| executable files during splitting. | |||
| 2024-04-12 | file chunker: increase chunk sizes | Klaus Aehlig | |
| As we use chunking also for reducing storage, we have to consider the overhead of block devices which is in the order of kB per file. So our target chunk size should be at least 2 orders of magnitude above this. This suggests to minimally aim for a chunk size of 128kB, a target size that also has the advantage the that maximal chunk size associated with this size is 1MB which is still well below the maximal transmission size of grpc allowing us to avoid the streaming API. As we're scaling everything up by a factor of 16, we also have to increase the number of bits in the involved masks by 4. We use this to also extend the window size by using the 2 most significant octets. Following the advice of the paper proposing FastCDC to spread out the ones roughly equally suggests 0x4444 as a suitable value for the two most significant octets. We also change the suggested extension of the remote-execution API accordingly. As the precise parameters for FastCDC when announced over the remote-execution APIs are still under discussion upstream, we simplify the name to not mention the target size. | |||
| 2024-04-12 | file chunker: remove average chunk size from interface | Klaus Aehlig | |
| ... as the typical chunk size is mainly determined by the masks used internally. So, as long as we hard code them, we should be honest to ourselves and accept that the chunking parameters are hard-coded as well. | |||
| 2024-04-08 | Use properly included standard library types by default | Paul Cristian Sarbu | |
| 2024-04-02 | LargeBlobs: Splice large objects from external sources. | Maksim Denisov | |
| For splicing of large objects from external sources additional checks are performed: * The digest of the spliced result must be equal to the expected digest; * The parts of a spliced tree must be in the storage. Tested: * Regular splicing of large objects; * If the result is unexpected, splicing fails; * If some parts of a tree are missing, splicing fails. | |||
| 2024-04-02 | LargeBlobs: Uplink large objects. | Maksim Denisov | |
| * Uplink parts of the large entry before entry itself; * Uplink large entries in LargeObjectCAS::GetEntryPath to not split things two times; * Promote spliced tree during uplinking of a large tree entry to properly promote parts of the tree; * Uplink large entries in LocalUplink{Blob, Tree} to support proper uplinking in Action Cache and Target Cache; Tested: * Uplink large blobs and trees; * Uplink a large object that depends on other large objects. | |||
| 2024-04-02 | LargeBlobs: Splice large objects implicitly. | Maksim Denisov | |
| Implicitly reconstruct objects during regular uplinking of Blobs/Trees. | |||
| 2024-04-02 | LargeBlobs: Split large objects. | Maksim Denisov | |
| * Add LargeObjectCAS fields for files and trees to LocalCAS; * Add logic for splitting objects located in the main storage. Tested: Splitting of large, small and empty objects. | |||
| 2024-04-02 | LargeObjectCAS: Implement auxiliary class for storing error information. | Maksim Denisov | |
 |
index : ~aehlig/just-buildsystem/justbuild.git | |
| Main repository of just-buildsystem, containing the buildtool (just) in bootstrappable form, the multi-repository setup and launch tool (just-mr) and some auxilliary tools. | Klaus Aehlig |
| summaryrefslogtreecommitdiff |