1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
|
# Invocation Logging and Profiling
For large projects, it can be helpful to find out which actions
make, e.g., the runs of the continuous integration slow. `just` has
an option `--profile` that instructs it to write a profile file to
the specified location on disk. That profile file contains (among
other things, see `just-profile(5)`) for each action that was
considered during the build whether it was cached, and, if not,
its run time. These actions can be associated to targets via the
`"origins"` filed in the action graph that can be obtained via the
`--dump-graph` option.
However, not only the run time of actions in a particular build
matters, but also how often an action has to be rerun because some
input changed. Therefore, we need to collect profiling data over many
invocations. To do so, `just-mr` honors an option `"invocation log"`
in its rc file. For that key, several subkeys can be specified with
the most important being `"directory"`; the location object defined
there specifies the directory where the logging happens (always on
the local machine, even if remote-execution is used). Inside this
directory, a subdirectory is used for each project; the project can
be specified by the subkey `"project id"`. Typically, setting the
project identifier is delegated to an rc file in the workspace.
So, if we have `~/.just-mrrc` with contents
``` json
{ "rc files": [{"root": "workspace", "path": "etc/rc.json"}]
, "invocation log": {"directory": {"root": "home", "path": "log/justbuild"}}
}
```
and the project contains an `etc/rc.json` with
``` json
{ "invocation log": {"project id": "hot-new-product"}}
```
then for each build triggered through `just-mr` a new subdirectory
of `~/log/justbuild/hot-new-product` will be created. The name
of that subdirectory consists of date and time, followed by a
universally-unique identifier. In particular, parallel invocations
are no problem, even if invocation logging is activated. The
prefixing with date and time allows simple aggregation or cleanup
over fixed periods of time (like monthly, daily, hourly, etc).
Of course, just creating an empty directory for each invocation
of `just-mr` is not very useful. Therefore, more subkeys can
be specified.
- `"--profile"` specifies the file name within the invocation-log
directory to be used when generating the `--profile` option in
the command line for the `just` invocation.
- `"--dump-graph"` does the same for the `--dump-graph` option.
- `"metadata"` specifies a file name inside the invocation directory
a metadata file should be written by `just-mr`; that file
contains, in particular, the full command line that is executed
and the blob identifier of the repository configuration file
used in that invocation of `just` (if any).
So, if invocation logging is desired, a typical `~/.just-mrrc` file
would look as follows.
``` json
{ "rc files": [{"root": "workspace", "path": "etc/rc.json"}]
, "invocation log":
{ "directory": {"root": "home", "path": "log/justbuild"}
, "metadata": "meta.json"
, "--profile": "profile.json"
, "--dump-graph": "graph.json"
}
}
```
Of course, the main motivation for invocation logging is doing
statistical analysis later. However, it can also be useful to browse
through the most recent invocations, looking, e.g., at failed actions
and their output, or actions that took particularly long. In the
`justbuild` source tree, under `doc/invocations-http-server` there
is a simple application serving one directory of invocation logs,
i.e., the logs for one particular project id.
|
