path: root/doc/tutorial/target-file-glob-tree.org

* Target versus ~FILE~, ~GLOB~, and ~TREE~

So far, we referred to defined targets as well as source files
by their name and it just worked. When considering third-party
software we already saw the ~TREE~ reference. In this section, we
will highlight in more detail the ways to refer to sources, as well
as the difference between defined and source targets. The latter
is used, e.g., when third-party software has to be patched.

As example for this section we use gnu ~units~ where we want to
patch into the standard units defintion add two units of area
popular in German news.

** Repository Config for ~units~ with patches

The sources are an archive available on the web. As upstream uses a
different build system, we have to provide our own build description;
we take the top-level directory as layer for this. As we also want
to patch the definition file, we add the subdirectory ~files~ as
logical repository for the patches. Hence we create a file ~repos.json~
with the following content.

#+BEGIN_SRC js
{ "main": "units"
, "repositories":
  { "just-rules":{ /* ... unchanged ... */ }
  , "import targets": {"repository": {"type": "file", "path": "."}}
  , "patches": {"repository": {"type": "file", "path": "files"}}
  , "units":
    { "repository":
      { "type": "archive"
      , "content": "9781174d42bd593d3bab6c6decfdcae60e3ce328"
      , "fetch": "https://ftp.gnu.org/gnu/units/units-2.21.tar.gz"
      , "subdir": "units-2.21"
      }
    , "target_root": "import targets"
    , "target_file_name": "TARGETS.units"
    , "bindings": {"rules": "just-rules", "patches": "patches"}
    }
  }
}
#+END_SRC

The repository to set up is ~units~ and, as usual, we can use ~just-mr~ to
fetch the archive and obtain the resulting multi-repository configuration.

#+BEGIN_SRC sh
$ just-mr setup units
#+END_SRC

** Patching a file: targets versus ~FILE~

Let's start by patching the source file ~definitions.units~. While,
conceptionally, we want to patch a third-party source file, we do /not/
modify the sources. The workspace root is a git tree and stay like this.
Instead, we remember that we specify /targets/ and the definition of a
target is looked up in the targets file; only if not defined there, it
is implicitly considered a source target and taken from the target root.
So we will define a /target/ named ~definitions.units~ to replace the
original source file.

Let's first generate the patch. As we're already referring to source files
as targets, we have to provide a targets file already; we start with the
empty object and refine it later.

#+BEGIN_SRC sh
$ echo {} > TARGETS.units
$ just-mr install -o . definitions.units
INFO: Requested target is [["@","units","","definitions.units"],{}]
INFO: Analysed target [["@","units","","definitions.units"],{}]
INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
INFO: Discovered 0 actions, 0 trees, 0 blobs
INFO: Building [["@","units","","definitions.units"],{}].
INFO: Processed 0 actions, 0 cache hits.
INFO: Artifacts can be found in:
        /tmp/work-2022-08-22/definitions.units [0f24a321694aab5c1d3676e22d01fc73492bee42:342718:f]
$ cp definitions.units definitions.units.orig
$ ed definitions.units
342718
/German units
# A few German units as currently in use.
+2a
area_soccerfield        105 m * 68 m
area_saarland           2570 km^2
.
w
342789
q
$ mkdir files
$ echo {} > files/TARGETS
$ diff -u definitions.units.orig definitions.units > files/definitions.units.diff
$ rm definitions.units*
#+END_SRC

Our rules conveniently contain a rule ~["patch", "file"]~ to patch
a single file, and we already created the patch. The only other
input missing is the source file. So far, we could refer to it as
~"definitions.units"~ because there was no target of that name, but
now we're about to define a target with that very name. Fortunately,
in target files, we can use a special syntax to explictly refer to
a source file of the current module, even if there is a target with
the same name: ~["FILE", null, "definition.units"]~. The syntax
requires the explict ~null~ value for the current module, despite
the fact that explicit file references are only allowed for the
current module; in this way, the name is a list of length more than
two and cannot be confused with a top-level module called ~FILE~.
So we add this target and obtain as ~TARGETS.units~ the following.

#+BEGIN_SRC js
{ "definitions.units":
  { "type": ["@", "rules", "patch", "file"]
  , "src": [["FILE", ".", "definitions.units"]]
  , "patch": [["@", "patches", "", "definitions.units.diff"]]
  }
}
#+END_SRC

Analysing ~"definitions.units"~ we find our defined target which
contains an action output. Still, it looks like a patched source
file; the new artifact is staged to the original location. Staging
is also used in the action definition, to avoid magic names (like
file names starting with ~-~), in-place operations (all actions
must not modify their inputs) and, in fact, have a
fixed command line.

#+BEGIN_SRC sh
$ just-mr analyse definitions.units --dump-actions -
INFO: Requested target is [["@","units","","definitions.units"],{}]
INFO: Result of target [["@","units","","definitions.units"],{}]: {
        "artifacts": {
          "definitions.units": {"data":{"id":"98e3c7758f5dd433c6aa7b327040be676faf6f34","path":"patched"},"type":"ACTION"}
        },
        "provides": {
        },
        "runfiles": {
          "definitions.units": {"data":{"id":"98e3c7758f5dd433c6aa7b327040be676faf6f34","path":"patched"},"type":"ACTION"}
        }
      }
INFO: Actions for target [["@","units","","definitions.units"],{}]:
[
  {
    "command": ["patch","-s","--read-only=ignore","--follow-symlinks","-o","patched","orig","patch"],
    "input": {
      "orig": {
        "data": {
          "file_type": "f",
          "id": "0f24a321694aab5c1d3676e22d01fc73492bee42",
          "size": 342718
        },
        "type": "KNOWN"
      },
      "patch": {
        "data": {
          "path": "definitions.units.diff",
          "repository": "patches"
        },
        "type": "LOCAL"
      }
    },
    "output": ["patched"]
  }
]
$
#+END_SRC

Building ~"definitions.units"~ we find out patch applied correctly.

#+BEGIN_SRC sh
$ just-mr build definitions.units -P definitions.units | grep -A 5 'German units'
INFO: Requested target is [["@","units","","definitions.units"],{}]
INFO: Analysed target [["@","units","","definitions.units"],{}]
INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
INFO: Discovered 1 actions, 0 trees, 1 blobs
INFO: Building [["@","units","","definitions.units"],{}].
INFO: Processed 1 actions, 0 cache hits.
INFO: Artifacts built, logical paths are:
        definitions.units [bfc9cebfce398205e94a441aaffc0f8a750b111e:342789:f]
# A few German units as currently in use.
#

area_soccerfield        105 m * 68 m
area_saarland           2570 km^2
zentner                 50 kg
$
#+END_SRC

** Globbing source files: ~"GLOB"~

Next, we collect all ~.units~ files. We could simply do this by enumerating
them in a target.

#+BEGIN_SRC js
, "data": { "type": "install", "deps": ["definitions.units", "currency.units"]}
#+END_SRC

In this way, we get the desired collection of one unmodified source file and
the output of the patch action.

#+BEGIN_SRC sh
$ just-mr analyse data
INFO: Requested target is [["@","units","","data"],{}]
INFO: Result of target [["@","units","","data"],{}]: {
        "artifacts": {
          "currency.units": {"data":{"file_type":"f","id":"ac6da8afaac0f34e114e123e4ab3a41e59121b10","size":14707},"type":"KNOWN"},
          "definitions.units": {"data":{"id":"98e3c7758f5dd433c6aa7b327040be676faf6f34","path":"patched"},"type":"ACTION"}
        },
        "provides": {
        },
        "runfiles": {
          "currency.units": {"data":{"file_type":"f","id":"ac6da8afaac0f34e114e123e4ab3a41e59121b10","size":14707},"type":"KNOWN"},
          "definitions.units": {"data":{"id":"98e3c7758f5dd433c6aa7b327040be676faf6f34","path":"patched"},"type":"ACTION"}
        }
      }
$
#+END_SRC

The disadvantage, however, that we might miss newly added ~.units~
files if we update and upstream added new files. So we want all
source files that have the respective ending. The corresponding
source reference is ~"GLOB"~. A glob expands to the /collection/
of all /sources/ that are /files/ in the /top-level/ directory of
the current module and that match the given pattern. It is important
to understand this in detail and the rational behind it.
- First of all, the artifact (and runfiles) map has an entry for
  each file that matches. In particular, targets have the option to
  define individual actions for each file, like ~["CC", "binary"]~
  does for the source files. This is different from ~"TREE"~ where
  the artifact map contains a single artifact that happens to be a
  directory. The tree behaviour is preferrable when the internals
  of the directory only matter for the execution of actions and not
  for analysis; then there are less entries to carry around during
  analysis and action-key computation, and the whole directory
  is "reserved" for that tree avoid staging conflicts when latter
  adding entries there.
- As a source reference, a glob expands to explict source files;
  targets having the same name as a source file are not taken into
  account. In our example, ~["GLOB", null, "*.units"]~ therefore
  contains the unpatched source file ~definitions.units~. In this
  way, we avoid any surprises in the expansion of a glob when a new
  source file is added with a name equal to an already existing target.
- Only files are are considered for matching the glob. Directories
  are ignored.
- Matches are only cosidered at the top-level directory. In this
  way, only one directory has to be read during analysis; allowing
  deeper globs would require traversal of subdirectories requiring
  larger cost. While the explicit ~"TREE"~ reference allows recursive
  traversal, in the typical use case of the respective workspace root
  being a ~git~ root, it is actually cheap; we can look up the
  ~git~ tree identifier without traversing the tree. Such a quick
  look up would not be possible if matches had to be selected.

So, ~["GLOB", null, "*.units"]~ expands to all the relevant source
files; but we still want to keep the patching. Most rules, like ~"install"~,
disallow staging conflicts to avoid accidentally ignoring a file due
to conflicting name. In our case, however, the dropping of the source
file in favour of the patched one is deliberate. For this, there is
the rule ~["data", "overlay"]~ taking the union of the artifacts of
the specified targets, accepting conflicts and resolving them in a
latest-wins fashion. Keep in mind, that our target fields are list,
not sets. Looking at the definition of the rule, one finds that
it is simply a ~"map_union"~. Hence we refine our ~"data"~ target.

#+BEGIN_SRC js
, "data":
  { "type": ["@", "rules", "data", "overlay"]
  , "deps": [["GLOB", null, "*.units"], "definitions.units"]
  }
#+END_SRC

The result of the analysis, of course, still is the same.

** Finishing the example: binaries from globbed sources

The source-code organisation of units is pretty simple. All source
and header files are in the top-level directory. As the header files
are not in a directory of their own, we can't use a tree, so we use
a glob, which is fine for the private headers of a binary. For the
source files, we have to have them individually anyway. So our first
attempt of defining the binary is as follows.

#+BEGIN_SRC js
, "units":
  { "type": ["@", "rules", "CC", "binary"]
  , "name": ["units"]
  , "link external": ["-lm"]
  , "pure C": ["YES"]
  , "srcs": [["GLOB", null, "*.c"]]
  , "private-hdrs": [["GLOB", null, "*.h"]]
  }
#+END_SRC

The result basically work and shows that we have 5 source files in total,
giving 5 compile and one link action.

#+BEGIN_SRC sh
$ just-mr build units
INFO: Requested target is [["@","units","","units"],{}]
INFO: Analysed target [["@","units","","units"],{}]
INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
INFO: Discovered 6 actions, 1 trees, 0 blobs
INFO: Building [["@","units","","units"],{}].
INFO (action:12af248ce5737be492f7f5909284d4e3b6488807):
     Stderr of command: ["cc","-I","work","-isystem","include","-c","work/strfunc.c","-o","work/strfunc.o"]
     work/strfunc.c:109:8: warning: extra tokens at end of #endif directive [-Wendif-labels]
       109 | #endif NO_STRSPN
           |        ^~~~~~~~~
INFO: Processed 6 actions, 0 cache hits.
INFO: Artifacts built, logical paths are:
        units [718cb1489bd006082f966ea73e3fba3dd072d084:124488:x]
$
#+END_SRC

To keep the build clean, we want to get rid of the warning. Of course, we could
simply set an appropriate compiler flag, but let's do things properly and patch
away the underlying reason. To do so, we first create a patch.

#+BEGIN_SRC sh
$ just-mr install -o . strfunc.c
INFO: Requested target is [["@","units","","strfunc.c"],{}]
INFO: Analysed target [["@","units","","strfunc.c"],{}]
INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
INFO: Discovered 0 actions, 0 trees, 0 blobs
INFO: Building [["@","units","","strfunc.c"],{}].
INFO: Processed 0 actions, 0 cache hits.
INFO: Artifacts can be found in:
        /tmp/work-2022-08-22/strfunc.c [e2aab4b825fa2822ccf33746d467a4944212abb9:2201:f]
$ cp strfunc.c strfunc.c.orig
$ ed strfunc.c
2201
109
#endif NO_STRSPN
s|N|// N
#endif // NO_STRSPN
w
2204
q
$ diff strfunc.c.orig strfunc.c > files/strfunc.c.diff
$ rm strfunc.c*
$
#+END_SRC

Then we amend our ~"units"~ target.

#+BEGIN_SRC js
, "units":
  { "type": ["@", "rules", "CC", "binary"]
  , "name": ["units"]
  , "link external": ["-lm"]
  , "pure C": ["YES"]
  , "srcs": ["patched srcs"]
  , "private-hdrs": [["GLOB", null, "*.h"]]
  }
, "patched srcs":
  { "type": ["@", "rules", "data", "overlay"]
  , "deps": [["GLOB", null, "*.c"], "strfunc.c"]
  }
, "strfunc.c":
  { "type": ["@", "rules", "patch", "file"]
  , "src": [["FILE", ".", "strfunc.c"]]
  , "patch": [["@", "patches", "", "strfunc.c.diff"]]
  }
#+END_SRC

Building the new target, 2 actions have to be exectung: the patching, and
the compiling of the patched source file. As the patched file still generates
the same object file as the unpatched file (after all, we only wanted to get
rid of a warning), the linking step can be taken from cache.

#+BEGIN_SRC sh
$ just-mr build units
INFO: Requested target is [["@","units","","units"],{}]
INFO: Analysed target [["@","units","","units"],{}]
INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
INFO: Discovered 7 actions, 1 trees, 1 blobs
INFO: Building [["@","units","","units"],{}].
INFO: Processed 7 actions, 5 cache hits.
INFO: Artifacts built, logical paths are:
        units [718cb1489bd006082f966ea73e3fba3dd072d084:124488:x]
$
#+END_SRC

To finish the example, we also add a default target, staging
artifacts according to the usual conventions.

#+BEGIN_SRC js
, "": {"type": "install", "dirs": [["units", "bin"], ["data", "share/units"]]}
#+END_SRC

Then things work as expected

#+BEGIN_SRC sh
$ just-mr install -o /tmp/testinstall
INFO: Requested target is [["@","units","",""],{}]
INFO: Analysed target [["@","units","",""],{}]
INFO: Export targets found: 0 cached, 0 uncached, 0 not eligible for caching
INFO: Discovered 8 actions, 1 trees, 1 blobs
INFO: Building [["@","units","",""],{}].
INFO: Processed 8 actions, 8 cache hits.
INFO: Artifacts can be found in:
        /tmp/testinstall/bin/units [718cb1489bd006082f966ea73e3fba3dd072d084:124488:x]
        /tmp/testinstall/share/units/currency.units [ac6da8afaac0f34e114e123e4ab3a41e59121b10:14707:f]
        /tmp/testinstall/share/units/definitions.units [bfc9cebfce398205e94a441aaffc0f8a750b111e:342789:f]
$ /tmp/testinstall/bin/units 'area_saarland' 'area_soccerfield'
	* 359943.98
	/ 2.7782101e-06
$
#+END_SRC