path: root/doc/tutorial/proto.org

* Using protocol buffers

The rules /justbuild/ uses for itself also support protocol
buffers. This tutorial shows how to use those rules and the targets
associated with them. It is not a tutorial on protocol buffers
itself; rather, it is assumed that the reader has some knowledge on
[[https://developers.google.com/protocol-buffers/][protocol buffers]].

** Setting up the repository configuration

The ~protobuf~ repository conveniently contains an
[[https://github.com/protocolbuffers/protobuf/tree/v3.12.4/examples][example]],
so we can use this and just add our own target files. We create
file ~repos.json~ as follows.

#+BEGIN_SRC js
{ "repositories":
  { "":
    { "repository": "protobuf"
    , "target_root": "tutorial"
    , "bindings": {"rules": "rules"}
    }
  , "tutorial": {"repository": {"type": "file", "path": "."}}
  , ...
  }
}
#+END_SRC

The remaining entries are just the ones from ~etc/repos.json~ from
the /justbuild/ repository; here we keep in mind that we're now working
in a different location, and hence have to replace the relative paths
by appropriate absolute ones.

To build the example with ~just~, the only task is to write a targets
file at ~examples/TARGETS~. As that contains a couple of new concepts,
we will do this step by step.

** The proto library

First, we have to declare the proto library. In this case, it only
contains one file and has no dependencies.

#+BEGIN_SRC js
{ "address":
  { "type": ["@", "rules", "proto", "library"]
  , "name": ["addressbook"]
  , "srcs": ["addressbook.proto"]
  }
...
#+END_SRC

In general, proto libraries could also depend on other proto libraries;
those would be added to the ~"deps"~ field.

When building the library, there's very little to do.

#+BEGIN_SRC sh
$ just-mr build examples address
INFO: Requested target is [["@","","examples","address"],{}]
INFO: Analysed target [["@","","examples","address"],{}]
INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
INFO: Discovered 0 actions, 0 trees, 0 blobs
INFO: Building [["@","","examples","address"],{}].
INFO: Processed 0 actions, 0 cache hits.
INFO: Artifacts built, logical paths are:
$
#+END_SRC

On the other hand, what did we expect? A proto library is an abstract
description of a protocol, so, as long as we don't specify for which
language we want to have bindings, there is nothing to generate.

Nevertheless, a proto library target is not empty. In fact, it can't be empty,
as other targets can only access the values of a target and have no
insights into its definitions. We already relied on this design principle
implicitly, when we exploited target-level caching for our external dependencies
and did not even construct the dependency graph for that target. A proto
library simply provides the dependency structure of the ~.proto~ files.

#+BEGIN_SRC sh
$ just-mr analyse --dump-nodes - examples address
INFO: Requested target is [["@","","examples","address"],{}]
INFO: Result of target [["@","","examples","address"],{}]: {
        "artifacts": {
        },
        "provides": {
          "proto": [
            {"id":"2a483a2de7f25c1bc066e47245f55ec9a2d4a719","type":"NODE"}
          ]
        },
        "runfiles": {
        }
      }
INFO: Target nodes of target [["@","","examples","address"],{}]:
{
  "089f6cae7ca77bb786578d3e0138b6ff445c5c92": {
    "result": {
      "artifact_stage": {
        "addressbook.proto": {
          "data": {
            "file_type": "f",
            "id": "b4b33b4c658924f0321ab4e7a9dc9cf8da1acec3",
            "size": 1234
          },
          "type": "KNOWN"
        }
      },
      "provides": {
      },
      "runfiles": {
      }
    },
    "type": "VALUE_NODE"
  },
  "2a483a2de7f25c1bc066e47245f55ec9a2d4a719": {
    "node_type": "library",
    "string_fields": {
      "name": ["addressbook"],
      "stage": [""]
    },
    "target_fields": {
      "deps": [],
      "srcs": [{"id":"089f6cae7ca77bb786578d3e0138b6ff445c5c92","type":"NODE"}]
    },
    "type": "ABSTRACT_NODE"
  }
}
$
#+END_SRC

The target has one provider ~"proto"~, which is a node. Nodes are
an abstract representation of a target graph. More precisely, there
are two kind of nodes, and our example contains one of each.

The simple kind of nodes are the value nodes; they represent a
target that has a fixed value, and hence are given by artifacts,
runfiles, and provided data. In our case, we have one value node,
the one for the ~.proto~ file.

The other kind of nodes are the abstract nodes. They describe the
arguments for a target, but only have an abstract name (i.e., a
string) for the rule. Combining such an abstract target with a
binding for the abstract rule names gives a concrete "anonymous"
target that, in our case, will generate the library with the bindings
for the concrete language. In this example, the abstract name is
~"library"~. The alternative in our proto rules would have been
~"service library"~, for proto libraries that also contain ~rpc~
definitions (which is used by [[https://grpc.io/][gRPC]]).

** Using proto libraries

Using proto libraries requires, as discussed, bindings for the
abstract names. Fortunately, our ~CC~ rules are aware of proto
libraries, so we can simply use them. Our target file hence
continues as follows.

#+BEGIN_SRC js
...
, "add_person":
  { "type": ["@", "rules", "CC", "binary"]
  , "name": ["add_person"]
  , "srcs": ["add_person.cc"]
  , "proto": ["address"]
  }
, "list_people":
  { "type": ["@", "rules", "CC", "binary"]
  , "name": ["list_people"]
  , "srcs": ["list_people.cc"]
  , "proto": ["address"]
  }
...
#+END_SRC

The first time, we build a target that requires the proto compiler
(in that particular version, built in that particular way), it takes
a bit of time, as the proto compiler has to be built. But in follow-up
builds, also in different projects, the target-level cache is filled already.

#+BEGIN_SRC sh
$ just-mr build examples add_person
...
$ just-mr build examples add_person
INFO: Requested target is [["@","","examples","add_person"],{}]
INFO: Analysed target [["@","","examples","add_person"],{}]
INFO: Export targets found: 3 cached, 0 uncached, 0 not eligible for caching
INFO: Discovered 5 actions, 2 trees, 0 blobs
INFO: Building [["@","","examples","add_person"],{}].
INFO: Processed 5 actions, 5 cache hits.
INFO: Artifacts built, logical paths are:
        add_person [7210834b05139defe783811d77087aa7c256405c:1980320:x]
$
#+END_SRC

If we look at the actions associated with the binary, we find that those
are still the two actions we expect: a compile action and a link action.

#+BEGIN_SRC sh
$ just-mr analyse examples add_person --dump-actions -
INFO: Requested target is [["@","","examples","add_person"],{}]
INFO: Result of target [["@","","examples","add_person"],{}]: {
        "artifacts": {
          "add_person": {"data":{"id":"51f7e29f0669608f9e0a0d8c8f4946c239a3ed09","path":"add_person"},"type":"ACTION"}
        },
        "provides": {
        },
        "runfiles": {
        }
      }
INFO: Actions for target [["@","","examples","add_person"],{}]:
[
  {
    "command": ["clang++","-std=c++20","-O2","-Wall","-Wextra","-Wpedantic","-Wsign-conversion","-I","work","-isystem","include","-c","work/add_person.cc","-o","work/add_person.o"],
    "env": {
      "PATH": "/bin:/sbin:/usr/bin:/usr/sbin"
    },
    "input": {
      ...
      }
    },
    "output": ["work/add_person.o"]
  },
  {
    "command": ["clang++","-o","add_person","add_person.o","libaddressbook.a","libprotobuf.a","libprotobuf_lite.a","libzlib.a"],
    "env": {
      "PATH": "/bin:/sbin:/usr/bin:/usr/sbin"
    },
    "input": {
      ...
    },
    "output": ["add_person"]
  }
]
$
#+END_SRC

As discussed, the ~libaddressbook.a~ that is conveniently available
during the linking of the binary (as well as the ~addressbook.pb.h~
available in the ~include~ tree for the compile action) are generated
by an anonymous target. Using that during the build we already
filled the target-level cache, we can have a look at all targets
still analysed. In the one anonymous target, we find again the
abstract node we discussed earlier.

#+BEGIN_SRC sh
$ just-mr analyse examples add_person  --dump-targets -
INFO: Requested target is [["@","","examples","add_person"],{}]
INFO: Result of target [["@","","examples","add_person"],{}]: {
        "artifacts": {
          "add_person": {"data":{"id":"51f7e29f0669608f9e0a0d8c8f4946c239a3ed09","path":"add_person"},"type":"ACTION"}
        },
        "provides": {
        },
        "runfiles": {
        }
      }
INFO: List of analysed targets:
{
  "#": {
    "acde278315be59c6bdf436efa9dc9782a6c59f36": {
      "2a483a2de7f25c1bc066e47245f55ec9a2d4a719": [{"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"CXX":null,"CXXFLAGS":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}]
    }
  },
  "@": {
    "": {
      "examples": {
        "add_person": [{"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"CXX":null,"CXXFLAGS":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}],
        "address": [{}]
      }
    },
    "protobuf": {
      "": {
        "C++ runtime": [{"AR":null,"ARCH":null,"CXX":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}],
        "protoc": [{"AR":null,"ARCH":null,"CXX":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}],
        "well_known_protos": [{}]
      }
    },
    "rules": {
      "CC": {
        "defaults": [{}]
      }
    }
  }
}
$
#+END_SRC

It should be noted, however, that this tight integration of proto
into our ~C++~ rules is just convenience of our code base. If had
to cooperate with rules not aware of proto, we could have created
a separate rule delegating the library creation to the anonymous
target and then simply reflecting the values of that target.

** Adding a test

Finally, let's add a test. As we use the ~protobuf~ repository as
workspace root, we add the test script ad hoc into the targets file,
using the ~"file_gen"~ rule. For debugging a potentially failing
test, we also keep the intermediate files the test generates.

#+BEGIN_SRC js
...
, "test.sh":
  { "type": "file_gen"
  , "name": "test.sh"
  , "data":
    { "type": "join"
    , "separator": "\n"
    , "$1":
      [ "set -e"
      , "(echo 12345; echo 'John Doe'; echo 'jdoe@example.org'; echo) | ./add_person addressbook.data"
      , "./list_people addressbook.data > out.txt"
      , "grep Doe out.txt"
      ]
    }
  }
, "test":
  { "type": ["@", "rules", "shell/test", "script"]
  , "name": ["read-write-test"]
  , "test": ["test.sh"]
  , "deps": ["add_person", "list_people"]
  , "keep": ["addressbook.data", "out.txt"]
  }
}
#+END_SRC

That example also shows why it is important that the generation
of the language bindings is delegated to an anonymous target: we
want to analyse only once how the ~C++~ bindings are generated.
Nevertheless, many targets can depend (directly or indirectly) on
the same proto library. And, indeed, analysing the test, we get
the expected additional targets and the one anonymous target is
reused by both binaries.

#+BEGIN_SRC sh
$ just-mr analyse examples test  --dump-targets -
INFO: Requested target is [["@","","examples","test"],{}]
INFO: Result of target [["@","","examples","test"],{}]: {
        "artifacts": {
          "result": {"data":{"id":"dd5983ceb5ffbe6bee6da1664485d1948a5e952b","path":"result"},"type":"ACTION"},
          "stderr": {"data":{"id":"dd5983ceb5ffbe6bee6da1664485d1948a5e952b","path":"stderr"},"type":"ACTION"},
          "stdout": {"data":{"id":"dd5983ceb5ffbe6bee6da1664485d1948a5e952b","path":"stdout"},"type":"ACTION"},
          "time-start": {"data":{"id":"dd5983ceb5ffbe6bee6da1664485d1948a5e952b","path":"time-start"},"type":"ACTION"},
          "time-stop": {"data":{"id":"dd5983ceb5ffbe6bee6da1664485d1948a5e952b","path":"time-stop"},"type":"ACTION"},
          "work/addressbook.data": {"data":{"id":"dd5983ceb5ffbe6bee6da1664485d1948a5e952b","path":"work/addressbook.data"},"type":"ACTION"},
          "work/out.txt": {"data":{"id":"dd5983ceb5ffbe6bee6da1664485d1948a5e952b","path":"work/out.txt"},"type":"ACTION"}
        },
        "provides": {
        },
        "runfiles": {
          "read-write-test": {"data":{"id":"92a7e0fb13fbfea251760e81e66258782800b165"},"type":"TREE"}
        }
      }
INFO: List of analysed targets:
{
  "#": {
    "acde278315be59c6bdf436efa9dc9782a6c59f36": {
      "2a483a2de7f25c1bc066e47245f55ec9a2d4a719": [{"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"CXX":null,"CXXFLAGS":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}]
    }
  },
  "@": {
    "": {
      "examples": {
        "add_person": [{"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"CXX":null,"CXXFLAGS":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}],
        "address": [{}],
        "list_people": [{"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"CXX":null,"CXXFLAGS":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}],
        "test": [{"AR":null,"ARCH":null,"CC":null,"CFLAGS":null,"CXX":null,"CXXFLAGS":null,"ENV":null,"HOST_ARCH":null,"OS":null,"RUNS_PER_TEST":null,"TARGET_ARCH":null,"TEST_ENV":null}],
        "test.sh": [{}]
      }
    },
    "protobuf": {
      "": {
        "C++ runtime": [{"AR":null,"ARCH":null,"CXX":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}],
        "protoc": [{"AR":null,"ARCH":null,"CXX":null,"ENV":null,"HOST_ARCH":null,"OS":null,"TARGET_ARCH":null}],
        "well_known_protos": [{}]
      }
    },
    "rules": {
      "CC": {
        "defaults": [{}]
      }
    }
  }
}
INFO: Target tainted ["test"].
$
#+END_SRC

Finally, the test passes and the output is as expected.

#+BEGIN_SRC sh
$ just-mr build examples test -Pwork/out.txt
INFO: Requested target is [["@","","examples","test"],{}]
INFO: Analysed target [["@","","examples","test"],{}]
INFO: Export targets found: 3 cached, 0 uncached, 0 not eligible for caching
INFO: Target tainted ["test"].
INFO: Discovered 8 actions, 4 trees, 1 blobs
INFO: Building [["@","","examples","test"],{}].
INFO: Processed 8 actions, 5 cache hits.
INFO: Artifacts built, logical paths are:
        result [7ef22e9a431ad0272713b71fdc8794016c8ef12f:5:f]
        stderr [e69de29bb2d1d6434b8b29ae775ad8c2e48c5391:0:f]
        stdout [7fab9dd1ee66a1e76a3697a27524f905600afbd0:196:f]
        time-start [9488d8109ff186e7b9ffd7bdfe9f0cc11e99f781:11:f]
        time-stop [9488d8109ff186e7b9ffd7bdfe9f0cc11e99f781:11:f]
        work/addressbook.data [040e76802da97fab00070bb4dbca50d91f43ac7f:41:f]
        work/out.txt [a47b62aba8783b8f923218a6838972c77ac082f2:101:f]
      (1 runfiles omitted.)
Person ID: 12345
  Name: John Doe
  E-mail address: jdoe@example.org
  Updated: 2022-06-22T13:03:29Z
INFO: Target tainted ["test"].
$
#+END_SRC