| Age | Commit message (Collapse) | Author |
|---|
|
|
|
The default options of std::filesystem::copy include following
symlinks, resulting in file repositories creating wrong trees if
containing unresolved symlinks, or failing unexpectedly early if
symlink cycles existed.
This is fixed by ensuring the copy_symlinks option is always used.
|
|
The root async map in a chain of calls should always be checked
for missing value, which can happen if, e.g., a cycle happens or
a thread gets killed by the system.
Properly handle this by checking explicitly if a value has been
posted. If not, check for cycles where it makes sense (for
example, in the resolving of symlinks), otherwise report any
pending map keys not yet processed.
This is done for all just-mr commands working with async maps.
|
|
...to be used when reporting pending keys on failure to post value.
|
|
...in async map instances, same as for reporting cycles.
This removes the restriction that the key object has to posses the
ToString method, allowing it to be used, e.g., with just-mr maps.
The now obsolete HasToString concept is removed.
|
|
|
|
Fixes a false assumption that the result of resolving the tree will
always be set if the map doesn't log fatal, when in fact the map
might fail to set a value if, e.g., a thread is killed by the
system or there is a symlinks cycle.
|
|
|
|
The separation of cache-key handling and CAS lockup in
e6a91bb733b0738cee0b3ae06ee640f70c1e787f unified the log-level of
two messages to warning: the absence of a cache entry (originally
debug) and a report on a malformed entry in the cache (originally
warning). As we routinely expect non-cached actions in a build,
demote those messages to debug level in order to keep the log
readable and not confuse the user with warnings about expected
behaviour.
|
|
... as those typically are of transient nature as well.
|
|
|
|
We already accept short writes in batch uploads, but when no
progress is made, we cannot simply retry, as this might lead to
an infinite loop. Instead, we give up on batching and upload each
blob one by one.
|
|
|
|
... and correctly report the error.
- If we cannot store the bytes we received, this is an internal error.
- If the bytes received have a different hash than announced, report
this user error as INVLID_ARGUMENT.
|
|
|
|
... while keeping our .clang-format file.
|
|
This is an essential step in the debugging process that was missing
in the tutorial.
|
|
|
|
...into writing/reading to CAS and writing/reading entries (to remove code duplications in uplinking and obtaining the result).
The read functionality cannot be replaced by a single function since the CAS key of the entry is required during uplinking. Although the write functionality can, it is also separated into two parts to keep the reverse logic clear.
|
|
Removed named keys of entries (hash, size). These prefixes were added for every chunk creating an additional overhead without any benefit.
Removed prefixes of hashes (*62*hash) appearing in native mode and creating an additional overhead as well: it is known that all of them are blobs.
|
|
|
|
Also add empty directory in test script to ensure we don't regress
in the future. While there, fix some typos.
|
|
...by writing its tree directly in the object database instead of
working with the index. This allows the creation of trees that
contain also entries with 'magic' names, such as the .git folder
or .gitignore files.
Callers must ensure the given directory only contains the needed
entries. In particular, just-mr maps and serve service are updated
to separate the import-to-Git repository path from the temporary
path containing the content to be committed, to avoid polluting the
content path with entries generated on repository initialization.
|
|
The tree is traversed bottom-up and each entry is explicitly added
to the object database: non-directory entries have their content
read and are added as blobs with corresponding permissions based on
file type (regular, executable, symlink), while subdirectories are
added as trees via CreateTree based on the already added blobs.
|
|
While there, ensure optional Git operation parameters are checked
before use for the operations that require them.
|
|
In preparation for subsequent changes, specify the directory path
containing the tree content to be committed explicitly. This change
will allow eventually to be able to specify paths that are
different from the root path of the repository in which the commit
is created.
This commit renames and refactors StageAndCommitAllAnnonymous to
allow a directory path to be passed. The just-mr and serve service
logic is updated such that current behaviour is otherwise
unchanged.
|
|
The libgit2 treebuilder has unnecessary validity checks for tree
entries, including for Git-specific magic names (such as '.git'),
which cannot be disabled. However, in our tool any filesystem
entry should be allowed to be part of a tree.
Therefore, the treebuilder-based implementation for CreateTree is
replaced by a direct writing of trees, by content, into the
underlying repository object database.
Additionally, as direct insertion into the object dabase does not
check the validity of the tree entries, as was done implicitly by
the treebuilder before, add a check for existence of the tree
entries into the debug-level generic validity check, with the
option to not perform it for ReadTree (as it is unnecessary there).
|
|
|
|
The 'branch' field is deprecated, not being used by any of the
critical Git operations, thus it can be removed.
|
|
The Python script used for the first stage of bootstrapping just
uses the Git index to create trees from directories (be it file
repositories, unpacked archives, or distfiles), therefore it has
the limitations of Git itself in committing trees that contain
entries ignored by Git, such as empty directories, the .git folder,
.gitignore files and files referenced there, or other entries with
Git-specific magic names.
This commit updates the Python script to replace the use of the
Git index for importing directories to directly writing the needed
blobs and trees to the object database, then commit the resulting
top tree explicitly.
While there, fix a typing issue from our relaxed approach in using
os.environ to set the subprocess env when running commands. As on
POSIX the type is _Environ, not simple Dict[str, str], use implicit
dictionary merging (Python v3.5+) to set the Git envars. The issue
was initially flagged by pyright.
|
|
|
|
|
|
The remote execution protocol is a bit unclear about how to
deal with blob updates for which we got no response. While
some clients consider a blob update failed only if a failed
response is received, we are going extra defensive here and
also consider missing responses to be a failed blob update.
Issue a retry for the missing blobs.
|
|
|
|
|
|
... and also perform conflict check on the normalized paths. Still,
the the output of the "ACTION" funtion be keyed by the representation
of the paths as originally described, to allow the author of a rule
to use non-normalized paths as well.
|
|
... so that we can run with whatever ambient path is present
rather than relying on standard paths pulled in by env.
|
|
... inheriting path from CC and shell defaults.
|
|
... and honor those in ["lint", "targets"].
|
|
... that was accidentially replaced by a first-wins semantics in
62d204ff4cc94c12c1635f189255710901682825 which fortunately did not
make it to any release.
|
|
|
|
... so that linting information gets propagated properly.
|
|
|
|
|
|
... collecting test results similar to the way "install" rules
are typically used; however in such a way that the provider "lint"
is forwarded.
|
|
|
|
While there, also properly transition "srcs" and "private-hdrs"
to the host version.
|
|
In case many "targets" are given, the field "lint" will contain
all the concatenation of the provider "lint" of the given targets.
There is, however, not need to lint the same file in the same
context twice, so deduplicate the targets first. While this does
not change the amount of lint actions carried out (as equal actions
are handled only once anyway), it keeps the summary clean by not
having dulicate entries.
|
|
Originally, the expression lanuage only contained a function to
deduplicate a list, keeping only the right-most occurence. The
reason was that this is the order needed for linking: a library
providing an open symbol has to come on the command line after the
library using that symbol (and hence making it an open symbol).
However, by now use cases have emerged that require a topological
sorting where definition comes before use; also, when composing
the value of PATH from fragments, we usually want to keep the first
occurrence in order for it to take precedence. Therefore, also
add "nub_left" as built-in function, allowing a more condense (and
slightly more efficient) description in rules instead of the
revserse-nub_right-reverse pattern.
|
|
... and, in particular, do not include headers of other libraries
that are not even needed.
|
