Retry Execution on FAILED_PRECONDITION

The specification for this status code is as follows.

  One or more errors occurred in setting up the action requested,
  such as a missing input or command or no worker being available.
  The client may be able to fix the errors and retry.

We routinely ensure all inputs are available to the remote execution
before we start an action, so all prerequisites will be there on a
compliant server, however might not actually be on a server where
the CAS only has eventual consistency or is incorrect (due to old
cache entries on CAS purge) in its answer to FindMissingBlobs.
While we have no guarantee that a retry will help, we still retry;
at least in the case of an unavailable worker or CAS entries not
yet available due to eventual consistency, this will help. Also,
we log at debug lvel the full response, including the repeated Any
message. In this way, we can find out what useful information (if
any) is sent by popular remote-execution services and implement
more specific mitigations in the future.

0 files changed, 0 insertions, 0 deletions