| Age | Commit message (Collapse) | Author |
|---|
|
|
|
|
|
|
|
|
|
... so that, e.g., we can set the logging from an expression value.
|
|
While for unit tests it is best practice to replicate the layout
of the source tree, for end-to-end tests, this is less so. This
is particularly true, as the distinction between the tests for two
tools is a bit blurry; some tests for just still use just-mr as a
launcher, and, e.g., the fetch test for just-mr uses just to carry
out the garbage collection. Therefore, move all end-to-end tests
together so that we also have a joinded target for precisely the
end-to-end tests.
In this reorganisation also indicate more explicitly which tests are
also available for the bootstrap multi-repo tool. That information
was so far hidden by the fact that in the other directory the tool
dependency would not dispatch on TEST_BOOTSTRAP_JUST_MR.
|
|
The distdir should only contain the direct dependencies of just.
Co-authored-by: Klaus Aehlig <klaus.aehlig@huawei.com>
|
|
|
|
|
|
|
|
|
|
To avoid unnecessary work, just-mr uses on-disk caches, including
for the mapping of a distdir to the corresponding git tree. This,
however, implies that the number of repositories that are actually
considered varies: in order to fetch a distdir repository, all
involved archives have to be fetched, but if we have a cache hit
none of them is even looked at.
So, in order to have a consistent reporting only count top-level
targets (i.e., the reachable repositories) in the statistics,
not the archives implicitly contained in a distdir, nor low-level
sub tasks. The actual fetch acitvity is shown separately by the
task tracker.
|
|
|
|
... by only taking the additional roots from the reachable
repositories, not also from the additional roots discovered
that way.
Co-authored-by: Paul Cristian Sarbu <paul.cristian.sarbu@huawei.com>
|
|
By default, messages at level "progress" and above are shown. So,
messages at that level should be useful for the user in the typical
case. For the count of export targets, this information is only useful
if the build involves export targets at all; the absence of export
targets is typically only used when debugging the build structure.
|
|
|
|
|
|
...bazel_types.hpp just requires the messages, not the grpc related
services and classes
|
|
tests have been updated accordingly
|
|
...both buildbarn and just execute do not allow a direct upload to the
action cache. Moreover, our tool does not support it, anyway, meaning
it was dead code.
|
|
|
|
|
|
...starting from a template (aka configuration file), and using the
variables defined via a ["CC/auto", "config"] target.
For example, to use a CMake configuration file, the targets could be
defined as follows
...
, "foo-header-blueprint":
{ "type": ["@", "rules", "CC/auto", "config_file"]
, "input": ["config.hpp.in"]
, "output": ["config.hpp"]
, "stage": ["foo"]
, "magic_string": ["cmakedefine"]
, "@only": ["true"]
}
, "foo-header":
{ "type": "configure"
, "target": "foo-header-blueprint"
, "config":
{ "type": "let*"
, "bindings":
[ [ "defines"
, [ ["var", "\"string value\""]
, ["FOO_MAJOR_VERSION", "3"]
, ["use_this_feature", true]
]
]
]
, "body": {"type": "env", "vars": ["defines"]}
}
}
...
The file config.hpp.in may look as follows
#ifndef config_cmake
#define config_cmake
#cmakedefine var
#cmakedefine use_this_feature
#cmakedefine01 use_this_feature
#cmakedefine unused
#define FOO_VERSION @FOO_MAJOR_VERSION@
#define DONT_TOUCH_THIS ${FOO_MAJOR_VERSION}
#endif
and the generated configuration file foo/config.hpp is
#ifndef config_cmake
#define config_cmake
#define var "string value"
#define use_this_feature
#define use_this_feature 1
/* #undef unused */
#define FOO_VERSION 3
#define DONT_TOUCH_THIS ${FOO_MAJOR_VERSION}
#endif
|
|
|
|
|
|
|
|
The improved GC implementation uses refactored storage
classes instead of directly accessing "unknown" file paths.
The required storage class refactoring is quite substantial
and outlined in the following paragraphs.
The module `buildtool/file_system` was extended by:
- `ObjectCAS`: a plain CAS implementation for
reading/writing blobs and computing digests for a given
`ObjectType`. Depending on that type, files written to the
file system may have different properties (e.g., the x-bit
set) or the digest may be computed differently (e.g., tree
digests in non-compatible mode).
A new module `buildtool/storage` was introduced containing:
- `LocalCAS`: provides a common interface for the "logical
CAS", which internally combines three `ObjectCAS`s, one
for each `ObjectType` (file, executable, tree).
- `LocalAC`: implements the action cache, which needs the
`LocalCAS` for storing cache values.
- `TargetCache`: implements the high-level target cache,
which also needs the `LocalCAS` for storing cache values.
- `LocalStorage`: combines the storage classes `LocalCAS`,
`LocalAC`, and `TargetCache`. Those are initialized with
settings from `StorageConfig`, such as the build root base
path or number of generations for the garbage collector.
`LocalStorage` is templated with a Boolean parameter
`kDoGlobalUplink`, which indicates that, on every
read/write access, the garbage collector should be used
for uplinking across all generations (global).
- `GarbageCollector`: responsible for garbage collection and
the global uplinking across all generations. To do so, it
employs instances of `LocalStorage` with `kDoGlobalUplink`
set to false, in order to avoid endless recursion. The
actual (local) uplinking within two single generations is
performed by the corresponding storage class (e.g.,
`TargetCache` implements uplinking of target cache entries
between two target cache generations etc.). Thereby, the
actual knowledge how data should be uplinked is
implemented by the instance that is responsible for
creating the data in the first place.
|
|
|
|
|
|
|
|
... as internally we do not do the Tree-dance. Instead, we
directly store the root Directory digest for ActionResult's
OutputDirectories.
|
|
|
|
|
|
|
|
... instead of using pipes that are prone to get full. Also
increase log level to Trace.
|
|
|
|
... on reporting the exec command; json encoding is already
quoting enough.
|
|
|
|
The forwarding of the log-limit argument is only useful, if
the value is different from the default.
|
|
A local launcher should be forwarded, if it is different to
the default launcher (which is ["env", "--"]); in particular,
an explictly empy launcher should be forwarded.
|
|
... and, in this way, avoid duplication of these constants that
might lead to inconsistencies between those two tools later.
|
|
|
|
remote endpoint
|
|
... now that they no longer depend on an external remote
execution.
|
|
... to bring their own remote-execution endpoint.
|
|
In order to keep our tests self-contained, do not rely on
an external remote-execution service to be present; instead,
use `just execute` to provide the remote execution service.
|
|
in the tree CAS.
Before this patch, after the checking in the tree cas, also the blob
cas was checked leading to a NOT-FOUND error.
|
|
google::longrunning::Operations::GetOperation
For each action that is executed, an entry is added to a shared thread
safe cache. Once the number of operations stored exceeds twice 2^n,
where n is given by the option --log-operations-threshold, at most 2^n
operations will be removed, in a FIFO scheme.
|
|
|
|
... by reporting unsupported config values explicitly and fixing
inconsistencies.
|
