From b66a7359fbbff35af630c88c56598bbc06b393e1 Mon Sep 17 00:00:00 2001 From: Oliver Reiche Date: Thu, 1 Jun 2023 13:36:32 +0200 Subject: doc: Convert orgmode files to markdown --- doc/future-designs/execution-properties.md | 125 +++++++++++++++++++++++++++++ 1 file changed, 125 insertions(+) create mode 100644 doc/future-designs/execution-properties.md (limited to 'doc/future-designs/execution-properties.md') diff --git a/doc/future-designs/execution-properties.md b/doc/future-designs/execution-properties.md new file mode 100644 index 00000000..d6fc53e8 --- /dev/null +++ b/doc/future-designs/execution-properties.md @@ -0,0 +1,125 @@ +Action-controlled execution properties +====================================== + +Motivation +---------- + +### Varying execution platforms + +It is a common situation that software is developed for one platform, +but it is desirable to build on a different one. For example, the other +platform could be faster (common theme when developing for embedded +devices), cheaper, or simply available in larger quantities. The +standard solution for these kind of situations is cross compiling: the +binary is completely built on one platform, while being intended to run +on a different one. This can be achieved by constructing the compiler +invocations accordingly and is already built into our rules (at least +for `C` and `C++`). + +The situation changes, however, once testing (especially end-to-end +testing) comes into play. Here, we actually have to run the built +binary---and do so on the target architecture. Nevertheless, we still +want to offload as much as possible of the work to the other platform +and perform only the actual test execution on the target platform. This +requires a single build executing actions on two (or more) platforms. + +### Varying execution times + +#### Calls to foreign build systems + +Often, third-party dependencies that natively build with a different +build system and don't change to often (yet often enough to not +have them part of the build image) are simply put in a single +action, so that they get built only once, and then stay in cache for +everyone. This is precisely, what our `rules-cc` rules like +`["CC/foreign/make", +"library"]` and `["CC/foreign/cmake", "library"]` do. + +For those compound actions, we of course expect them to run longer +than normal actions that only consist of a single compiler or linker +invocation. Giving an absolute amount of time needed for such an +action is not reasonable, as that very much depends on the +underlying hardware. However, it is reasonable to give a number +"typical" actions this compound action corresponds to. + +#### Long-running end-to-end tests + +A similar situation where a significantly longer action is needed in +a build otherwise consisting of short actions are end-to-end tests. +Test using the final binary might have a complex set up, potentially +involving several instances running to test communication, and +require a lengthy sequence of interactions to get into the situation +that is to be tested, or to verify the absence of degrading of the +service under high load or extended usage. + +Status Quo +---------- + +Action can at the moment specify + + - the actual action, i.e., inputs, outputs, and the command vector, + - the environment variables, + - a property that the action can fail (e.g., for test actions), and + - a property that the action is not to be taken from cache (e.g., + testing for flakiness). + +No other properties can be set by the action itself. In particular, +remote-execution properties and timeout are equal for all actions of a +build. + +Proposed changes +---------------- + +### Extension of the `"ACTION"` function + +We propose to extend the `"ACTION"` function available in the rule +definition by the following attributes. All of the new attributes are +optional, and the default is taken to reflect the status quo. Hence, the +proposed changes are backwards compatible. + +#### `"execution properties"` + +This value has to evaluate to a map of strings; if not given, the +empty map is taken as default. This map is taken as a union with any +remote-execution properties specified at the invocation of the build +(if keys are defined both, for the entire build and in +`"execution properties"` of a specific action, the latter takes +precedence). + +Local execution continues to any execution properties specified. +However, with the auxiliary change to `just` described later, such +execution properties can also influence a build that is local by +default. + +#### `"timeout scaling"` + +If given, the value has to be a number greater or equal than `1.0`, +with `1.0` taken as default. The action timeout specified for this +build (the default value, or whatever is specified on the command +line) is multiplied by the given factor and taken as timeout for +this action. This applies for both, local and remote builds. + +### `just` to support dispatching based on remote-execution properties + +In simple setups, like using `just execute`, the remote execution is not +capable of dispatching to different workers based on remote-execution +properties. To nevertheless have the benefits of using different +execution environments, `just` will allow an optional configuration file +to be passed on the command line via a new option +`--endpoint-configuration`. This configuration file will contain a list +of pairs of remote-execution properties and remote-execution endpoints. +The first matching entry (i.e., the first entry where the +remote-execution property map coincides with the given map when +restricted to its domain) determines the remote-execution endpoint to be +used; if no entry matches, the default remote-execution endpoint is +used. In any case, the remote-execution properties are forwarded to the +chosen remote-execution endpoint without modification. + +When connecting a non-standard remote-execution endpoint, `just` will +ensure that the applicable CAS of that endpoint will have all the needed +artifacts for that action. It will also transfer all result artifacts +back to the CAS of the default remote-execution endpoint. + +`just serve` (once implemented) will also support this new option. As +with the default execution endpoint, there is the understanding that the +client uses the same configuration as the `just serve` endpoint. -- cgit v1.2.3