diff options
Diffstat (limited to 'doc/tutorial/proto.md')
| -rw-r--r-- | doc/tutorial/proto.md | 474 |
1 files changed, 474 insertions, 0 deletions
diff --git a/doc/tutorial/proto.md b/doc/tutorial/proto.md new file mode 100644 index 00000000..8a04e373 --- /dev/null +++ b/doc/tutorial/proto.md @@ -0,0 +1,474 @@ +Using protocol buffers +====================== + +The rules *justbuild* uses for itself also support protocol buffers. +This tutorial shows how to use those rules and the targets associated +with them. It is not a tutorial on protocol buffers itself; rather, it +is assumed that the reader has some knowledge on [protocol +buffers](https://developers.google.com/protocol-buffers/). + +Setting up the repository configuration +--------------------------------------- + +Before we begin, we first need to declare where the root of our +workspace is located by creating the empty file `ROOT`: + +``` sh +$ touch ROOT +``` + +The `protobuf` repository conveniently contains an +[example](https://github.com/protocolbuffers/protobuf/tree/v3.12.4/examples), +so we can use this and just add our own target files. We create file +`repos.template.json` as follows. + +``` {.jsonc srcname="repos.template.json"} +{ "repositories": + { "": + { "repository": + { "type": "zip" + , "content": "7af7165b585e4aed714555a747b6822376176ef4" + , "fetch": "https://github.com/protocolbuffers/protobuf/archive/refs/tags/v3.12.4.zip" + , "subdir": "protobuf-3.12.4/examples" + } + , "target_root": "tutorial" + , "bindings": {"rules": "rules-cc"} + } + , "tutorial": {"repository": {"type": "file", "path": "."}} + } +} +``` + +The missing entry `"rules-cc"` refers to our C/C++ build rules provided +[online](https://github.com/just-buildsystem/rules-cc). These rules +support protobuf if the dependency `"protoc"` is provided. To import +this rule repository including the required transitive dependencies for +protobuf, the `bin/just-import-git` script with option `--as rules-cc` +can be used to generate the actual `repos.json`: + +``` sh +$ just-import-git -C repos.template.json -b master --as rules-cc https://github.com/just-buildsystem/rules-cc > repos.json +``` + +To build the example with `just`, the only task is to write targets +files. As that contains a couple of new concepts, we will do this step +by step. + +The proto library +----------------- + +First, we have to declare the proto library. In this case, it only +contains the file `addressbook.proto` and has no dependencies. To +declare the library, create a `TARGETS` file with the following content: + +``` {.jsonc srcname="TARGETS"} +{ "address": + { "type": ["@", "rules", "proto", "library"] + , "name": ["addressbook"] + , "srcs": ["addressbook.proto"] + } +} +``` + +In general, proto libraries could also depend on other proto libraries; +those would be added to the `"deps"` field. + +When building the library, there's very little to do. + +``` sh +$ just-mr build address +INFO: Requested target is [["@","","","address"],{}] +INFO: Analysed target [["@","","","address"],{}] +INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 0 actions, 0 trees, 0 blobs +INFO: Building [["@","","","address"],{}]. +INFO: Processed 0 actions, 0 cache hits. +INFO: Artifacts built, logical paths are: +$ +``` + +On the other hand, what did we expect? A proto library is an abstract +description of a protocol, so, as long as we don't specify for which +language we want to have bindings, there is nothing to generate. + +Nevertheless, a proto library target is not empty. In fact, it can't be +empty, as other targets can only access the values of a target and have +no insights into its definitions. We already relied on this design +principle implicitly, when we exploited target-level caching for our +external dependencies and did not even construct the dependency graph +for that target. A proto library simply provides the dependency +structure of the `.proto` files. + +``` sh +$ just-mr analyse --dump-nodes - address +INFO: Requested target is [["@","","","address"],{}] +INFO: Result of target [["@","","","address"],{}]: { + "artifacts": { + }, + "provides": { + "proto": [ + {"id":"2a483a2de7f25c1bc066e47245f55ec9a2d4a719","type":"NODE"} + ] + }, + "runfiles": { + } + } +INFO: Target nodes of target [["@","","","address"],{}]: +{ + "089f6cae7ca77bb786578d3e0138b6ff445c5c92": { + "result": { + "artifact_stage": { + "addressbook.proto": { + "data": { + "file_type": "f", + "id": "b4b33b4c658924f0321ab4e7a9dc9cf8da1acec3", + "size": 1234 + }, + "type": "KNOWN" + } + }, + "provides": { + }, + "runfiles": { + } + }, + "type": "VALUE_NODE" + }, + "2a483a2de7f25c1bc066e47245f55ec9a2d4a719": { + "node_type": "library", + "string_fields": { + "name": ["addressbook"], + "stage": [""] + }, + "target_fields": { + "deps": [], + "srcs": [{"id":"089f6cae7ca77bb786578d3e0138b6ff445c5c92","type":"NODE"}] + }, + "type": "ABSTRACT_NODE" + } +} +$ +``` + +The target has one provider `"proto"`, which is a node. Nodes are an +abstract representation of a target graph. More precisely, there are two +kind of nodes, and our example contains one of each. + +The simple kind of nodes are the value nodes; they represent a target +that has a fixed value, and hence are given by artifacts, runfiles, and +provided data. In our case, we have one value node, the one for the +`.proto` file. + +The other kind of nodes are the abstract nodes. They describe the +arguments for a target, but only have an abstract name (i.e., a string) +for the rule. Combining such an abstract target with a binding for the +abstract rule names gives a concrete "anonymous" target that, in our +case, will generate the library with the bindings for the concrete +language. In this example, the abstract name is `"library"`. The +alternative in our proto rules would have been `"service library"`, for +proto libraries that also contain `rpc` definitions (which is used by +[gRPC](https://grpc.io/)). + +Using proto libraries +--------------------- + +Using proto libraries requires, as discussed, bindings for the abstract +names. Fortunately, our `CC` rules are aware of proto libraries, so we +can simply use them. Our target file hence continues as follows. + +``` {.jsonc srcname="TARGETS"} +... +, "add_person": + { "type": ["@", "rules", "CC", "binary"] + , "name": ["add_person"] + , "srcs": ["add_person.cc"] + , "private-proto": ["address"] + } +, "list_people": + { "type": ["@", "rules", "CC", "binary"] + , "name": ["list_people"] + , "srcs": ["list_people.cc"] + , "private-proto": ["address"] + } +... +``` + +The first time, we build a target that requires the proto compiler (in +that particular version, built in that particular way), it takes a bit +of time, as the proto compiler has to be built. But in follow-up builds, +also in different projects, the target-level cache is filled already. + +``` sh +$ just-mr build add_person +... +$ just-mr build add_person +INFO: Requested target is [["@","","","add_person"],{}] +INFO: Analysed target [["@","","","add_person"],{}] +INFO: Export targets found: 3 cached, 0 uncached, 0 not eligible for caching +INFO: Discovered 5 actions, 2 trees, 0 blobs +INFO: Building [["@","","","add_person"],{}]. +INFO: Processed 5 actions, 5 cache hits. +INFO: Artifacts built, logical paths are: + add_person [bcbb3deabfe0d77e6d3ea35615336a2f59a1b0aa:2285928:x] +$ +``` + +If we look at the actions associated with the binary, we find that those +are still the two actions we expect: a compile action and a link action. + +``` sh +$ just-mr analyse add_person --dump-actions - +INFO: Requested target is [["@","","","add_person"],{}] +INFO: Result of target [["@","","","add_person"],{}]: { + "artifacts": { + "add_person": {"data":{"id":"fcf211e2291b2867375e915538ce04cb4dfae86d","path":"add_person"},"type":"ACTION"} + }, + "provides": { + }, + "runfiles": { + } + } +INFO: Actions for target [["@","","","add_person"],{}]: +[ + { + "command": ["c++","-I","work","-isystem","include","-c","work/add_person.cc","-o","work/add_person.o"], + "env": { + "PATH": "/bin:/usr/bin" + }, + "input": { + ... + }, + "output": ["work/add_person.o"] + }, + { + "command": ["c++","-o","add_person","add_person.o","libaddressbook.a","libprotobuf.a","libprotobuf_lite.a","libzlib.a"], + "env": { + "PATH": "/bin:/usr/bin" + }, + "input": { + ... + }, + "output": ["add_person"] + } +] +$ +``` + +As discussed, the `libaddressbook.a` that is conveniently available +during the linking of the binary (as well as the `addressbook.pb.h` +available in the `include` tree for the compile action) are generated by +an anonymous target. Using that during the build we already filled the +target-level cache, we can have a look at all targets still analysed. In +the one anonymous target, we find again the abstract node we discussed +earlier. + +``` sh +$ just-mr analyse add_person --dump-targets - +INFO: Requested target is [["@","","","add_person"],{}] +INFO: Result of target [["@","","","add_person"],{}]: { + "artifacts": { + "add_person": {"data":{"id":"fcf211e2291b2867375e915538ce04cb4dfae86d","path":"add_person"},"type":"ACTION"} + }, + "provides": { + }, + "runfiles": { + } + } +INFO: List of analysed targets: +{ + "#": { + "eda46ea21de25033ff7250e6a4cdc0b2c24be0c7": { + "2a483a2de7f25c1bc066e47245f55ec9a2d4a719": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}] + } + }, + "@": { + "": { + "": { + "add_person": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}], + "address": [{}] + } + }, + "rules-cc": { + "CC": { + "defaults": [{}] + } + }, + "rules-cc/just/protobuf": { + "": { + "C++ runtime": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}], + "protoc": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}], + "well_known_protos": [{}] + } + } + } +} +$ +``` + +It should be noted, however, that this tight integration of proto into +our `C++` rules is just convenience of our code base. If we had to +cooperate with rules not aware of proto, we could have created a +separate rule delegating the library creation to the anonymous target +and then simply reflecting the values of that target. In fact, we could +simply use an empty library with a public `proto` dependency for this +purpose. + +``` {.jsonc srcname="TARGETS"} +... +, "address proto library": + {"type": ["@", "rules", "CC", "library"], "proto": ["address"]} +... +``` + +``` sh +$ just-mr analyse 'address proto library' +... +INFO: Requested target is [["@","","","address proto library"],{}] +INFO: Result of target [["@","","","address proto library"],{}]: { + "artifacts": { + }, + "provides": { + ... + "compile-deps": { + "addressbook.pb.h": {"data":{"id":"6d70cd10fabcbc7591cd82aae2f100cca39d3879","path":"work/addressbook.pb.h"},"type":"ACTION"}, + ... + }, + "link-args": [ + "libaddressbook.a", + ... + ], + "link-deps": { + "libaddressbook.a": {"data":{"id":"753073bd026b6470138c47e004469dd1d3df08d4","path":"libaddressbook.a"},"type":"ACTION"}, + ... + }, + ... + }, + "runfiles": { + } + } +$ +``` + +Adding a test +------------- + +Finally, let's add a test. As we use the `protobuf` repository as +workspace root, we add the test script ad hoc into a targets file, using +the `"file_gen"` rule. For debugging a potentially failing test, we also +keep the intermediate files the test generates. Create a top-level +`TARGETS` file with the following content: + +``` {.jsonc srcname="TARGETS"} +... +, "test.sh": + { "type": "file_gen" + , "name": "test.sh" + , "data": + { "type": "join" + , "separator": "\n" + , "$1": + [ "set -e" + , "(echo 12345; echo 'John Doe'; echo 'jdoe@example.org'; echo) | ./add_person addressbook.data" + , "./list_people addressbook.data > out.txt" + , "grep Doe out.txt" + ] + } + } +, "test": + { "type": ["@", "rules", "shell/test", "script"] + , "name": ["read-write-test"] + , "test": ["test.sh"] + , "deps": ["add_person", "list_people"] + , "keep": ["addressbook.data", "out.txt"] + } +... +``` + +That example also shows why it is important that the generation of the +language bindings is delegated to an anonymous target: we want to +analyse only once how the `C++` bindings are generated. Nevertheless, +many targets can depend (directly or indirectly) on the same proto +library. And, indeed, analysing the test, we get the expected additional +targets and the one anonymous target is reused by both binaries. + +``` sh +$ just-mr analyse test --dump-targets - +INFO: Requested target is [["@","","","test"],{}] +INFO: Result of target [["@","","","test"],{}]: { + "artifacts": { + "result": {"data":{"id":"20967787c42a289f5598249e696f851dde50065c","path":"result"},"type":"ACTION"}, + "stderr": {"data":{"id":"20967787c42a289f5598249e696f851dde50065c","path":"stderr"},"type":"ACTION"}, + "stdout": {"data":{"id":"20967787c42a289f5598249e696f851dde50065c","path":"stdout"},"type":"ACTION"}, + "time-start": {"data":{"id":"20967787c42a289f5598249e696f851dde50065c","path":"time-start"},"type":"ACTION"}, + "time-stop": {"data":{"id":"20967787c42a289f5598249e696f851dde50065c","path":"time-stop"},"type":"ACTION"}, + "work/addressbook.data": {"data":{"id":"20967787c42a289f5598249e696f851dde50065c","path":"work/addressbook.data"},"type":"ACTION"}, + "work/out.txt": {"data":{"id":"20967787c42a289f5598249e696f851dde50065c","path":"work/out.txt"},"type":"ACTION"} + }, + "provides": { + }, + "runfiles": { + "read-write-test": {"data":{"id":"c9d7bfc5bc8448bfef25b4e73e4494560bf6c350"},"type":"TREE"} + } + } +INFO: List of analysed targets: +{ + "#": { + "eda46ea21de25033ff7250e6a4cdc0b2c24be0c7": { + "2a483a2de7f25c1bc066e47245f55ec9a2d4a719": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}] + } + }, + "@": { + "": { + "": { + "add_person": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}], + "address": [{}], + "list_people": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}], + "test": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"RUNS_PER_TEST":null,"TEST_ENV":null}], + "test.sh": [{}] + } + }, + "rules-cc": { + "CC": { + "defaults": [{}] + } + }, + "rules-cc/just/protobuf": { + "": { + "C++ runtime": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}], + "protoc": [{"ADD_CFLAGS":null,"ADD_CXXFLAGS":null,"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"COMPILER_FAMILY":null,"CXX":null,"CXXFLAGS":null,"DEBUG":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}], + "well_known_protos": [{}] + } + } + } +} +INFO: Target tainted ["test"]. +$ +``` + +Finally, the test passes and the output is as expected. + +``` sh +$ just-mr build test -Pwork/out.txt +INFO: Requested target is [["@","","","test"],{}] +INFO: Analysed target [["@","","","test"],{}] +INFO: Export targets found: 3 cached, 0 uncached, 0 not eligible for caching +INFO: Target tainted ["test"]. +INFO: Discovered 8 actions, 4 trees, 1 blobs +INFO: Building [["@","","","test"],{}]. +INFO: Processed 8 actions, 5 cache hits. +INFO: Artifacts built, logical paths are: + result [7ef22e9a431ad0272713b71fdc8794016c8ef12f:5:f] + stderr [e69de29bb2d1d6434b8b29ae775ad8c2e48c5391:0:f] + stdout [7fab9dd1ee66a1e76a3697a27524f905600afbd0:196:f] + time-start [7ac216a2a98b7739ae5304d96cdfa6f0b0ed87b6:11:f] + time-stop [7ac216a2a98b7739ae5304d96cdfa6f0b0ed87b6:11:f] + work/addressbook.data [baa6f28731ff6d93fbef9fcc5f7e8ae900da5ba5:41:f] + work/out.txt [7fb178dd66ecf24fdb786a0f96ae5969b55442da:101:f] + (1 runfiles omitted.) +Person ID: 12345 + Name: John Doe + E-mail address: jdoe@example.org + Updated: 2022-12-14T18:08:36Z +INFO: Target tainted ["test"]. +$ +``` |