diff options
author | Paul Cristian Sarbu <paul.cristian.sarbu@huawei.com> | 2023-11-09 15:42:58 +0100 |
---|---|---|
committer | Paul Cristian Sarbu <paul.cristian.sarbu@huawei.com> | 2023-11-14 13:35:01 +0100 |
commit | 5c9ea8725ebb58d42956309e7995e08837a25ff4 (patch) | |
tree | 8b8f6f165602e2440c9ef716543d4a90f9851d37 /doc/concepts | |
parent | a987adcbb97f878698018c1ec73536ad9bc2dcac (diff) | |
download | justbuild-5c9ea8725ebb58d42956309e7995e08837a25ff4.tar.gz |
alternative-mirrors design doc: Move to concepts folder
...and reword content to move from proposal to implemented state.
Diffstat (limited to 'doc/concepts')
-rw-r--r-- | doc/concepts/alternative-mirrors.md | 72 |
1 files changed, 72 insertions, 0 deletions
diff --git a/doc/concepts/alternative-mirrors.md b/doc/concepts/alternative-mirrors.md new file mode 100644 index 00000000..5572eda9 --- /dev/null +++ b/doc/concepts/alternative-mirrors.md @@ -0,0 +1,72 @@ +Alternative Mirrors +=================== + +Background +---------- + +The repository fetch tool `just-mr` reads a repository-configuration +file that describes `git` repositories and repositories given by +archives in a content-addressable way: by the `git` commit id, or +the blob identifier of the archive, respectively. Hence it does +not matter where these resources are fetched form; nevertheless, +a location to fetch from is needed for `just-mr` to be able to set +up those repositories. Typically the main upstream URL is given. + +While `just-mr` fetches each archive or commit only once and keeps +a local copy, large organizations still tend to set up a local mirror. +In this way, not every person working on the project has to fetch +from upstream (thus reducing load there) and there is a central +place archiving all the dependencies. Such an archive is needed +anyway, for audit and compliance reasons, as well as to be able +to continue the project independent of the dependencies' upstream. + +It is therefore desirable that `just-mr` support this workflow while +still clearly pointing out the upstream location, e.g., for updating. + +Implementation +-------------- + +We have introduced the following, backwards-compatible, extensions to +`just-mr`. + +### Mirrors field in `just-mr` repository config + +In the multi-repository configuration, in the definition of an +`"archive"`, `"zip"` or `"git"` repository, an addition field +`"mirrors"` was added. If given, this field has to be a list +of URLs that provide additional places where the respective +repository can be fetched from. `just-mr` will only consider a fetch +failed if the repository cannot be fetched neither from the main +location (described in `"fetch"` field for `"archive"` and `"zip"` +repositories, and `"repository"` field for `"git"` repositories), +nor from any of the specified mirrors. + +### Additional mirrors in the just local specification + +Local mirrors of organizations are often not available to the +general public and possibly not even available to everyone who +has access to the respective project. In order to avoid polluting +a multi-repository specification with the URLs of such restricted +mirrors, the `.just-local` file was extended to support additional +keys in its JSON object. +- For the optional key `"local mirrors"`, if given, a JSON object + is specified that maps primary URLs to a list of local (non-public) + mirrors. Those mirrors are always tried first (in the given order) + before any other URL is contacted. +- For the optional key `"preferred hostnames"`, if given, a list of + host names is specified. When fetching from a non-local URL, URLs + with host names in the given list are preferred (in the order + given) over URLs with other host names. + + +### `just-import-git` to support mirror specifications + +As multi-repository specifications are often generated from a +description of the local repositories by a chain of `just-import-git` +invocations, this tool needs to be able to also insert a `"mirrors"` +field for the repository imported. Therefore, this tool was +extended by an option `--mirror` where multiple occurrences accumulate +to specify additional mirrors for the URL fetched. These mirrors, +if any, form the `"mirrors"` field of the repository imported; they +are not used for the fetching in the import step itself, as the +head commit of the primary URL is to be taken. |