Skip to content

catalog-mirroring.md

Latest commit

 

History

History
130 lines (117 loc) · 9.51 KB

catalog-mirroring.md

File metadata and controls

130 lines (117 loc) · 9.51 KB
Raw

Table of Contents generated with DocToc

Mirror tool configuration

  • Starting with v1.8.0, to mirror the images to the final location, the plug-in provides support for mirror tool configuration to enable oc image mirror or oc mirror tool.
  • By default, plug-in is configured with oc image mirror tool. To enable oc mirror tool for mirroring, Run the following command: 
    oc ibm-pak config mirror-tools --enabled oc-mirror

  1. oc image mirror tool:

    • Uses an images mapping text file, which contains zero or more lines formatted as <source image>=<destination image>, with one line per image. This file is generated by the plugin, and uses the images provided by the product CASE(s) as the source image reference.
    • Accepts a list of arguments defining source images that should be pushed to the provided destination image tag.
    • Images in manifest list format will be copied as-is unless you use --filter-by-os flag to restrict the allowed images to copy in a manifest list.This flag has no effect on regular images.
    • When using file mirroring, the --dir and --from-dir flags control the location on disk that content will be stored to.

  2. oc mirror tool:

    • Uses a declarative image set configuration file where source images are provided either directly via additionalImages or indirectly via relatedImages which are defined within one or more catalog(s). It is also possible to limit which images are mirrored from a given catalog by specifying the packages and channels for a given operator so that you only mirror the images that your cluster needs. An image set configuration file is generated by the plugin when this tool is enabled.
    • Performs incremental mirroring, which reduces the size of future image sets.
    • Prunes images from the target mirror registry that were excluded from the image set configuration since the previous execution.
    • Provides a centralised method to mirror Operators and other images.
  • One of the important differences between these two tool choices is source for the images. In the oc image mirror scenario, the images are defined in the CASE(s) whereas in the oc mirror scenario the images are defined in the catalogs. The mirror tool configuration enable customers to control when to switch from CASE to CATALOG mirroring when they are ready.

Catalog curation

  • Catalog curation is the process of taking multiple catalogs as inputs, along with some constraints to produce a newly merged catalog. This capability is possible because we can merge the contents of a catalog (i.e. a File-Based Catalog (FBC)) in such a way to avoid merge conflicts.

  • A pre-curated catalog is purpose-built catalog of OLM operators that limit the scope of Operators that are available to just a few packages, channels and the specific operator versions (the bundles) within those channels.

  • Suppose you start with a catalog that contains product A and some time in the future you choose a catalog that contains product B. These catalogs will be merged together so that product B and product A will reside in the final catalog. This final catalog represents a single curated catalog which can then be mirrored using oc mirror. The curated catalog resides on disk in .ibm-pak/data/publish/latest/catalog-oci/manifest-list and the image set config references this location.

  • Catalog curation enable customers to build catalogs incrementally with the products pinned to the version which they need.

  • IBM product teams control when their Catalogs (defined within their CASE) become source of truth for catalog curation. In other words, an IBM product team has to opt into catalog curation, just as customers have to opt into catalog curation. This opt in behavior allows for a transition period to switch from CASE based mirroring to catalog based mirroring.

  • Run the following command to generate mirror manifests to be used when mirroring.

    oc ibm-pak generate mirror-manifests \
$CASE_NAME \
$TARGET_REGISTRY \
--version $CASE_VERSION

  • If product CASE supports catalog mirroring and you have enabled oc mirror tool, generate mirror manifests command will curate the catalog and generate image-set-config.yaml metadata file in ~/.ibm-pak/data/publish/latest directory. The following tree shows an example of the ~/.ibm-pak/data/publish directory.

    tree ~/.ibm-pak/data/publish/
/root/.ibm-pak/data/publish/
└── latest
    ├── catalog
    │   └── catalog.json
    ├── catalog-oci
    │   └── manifest-list
    │       ├── blobs
    │       │   └── sha256
    │       │       ├── xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    │       │       ├── xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    │       │       └── xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    │       ├── index.json
    │       └── oci-layout
    └── image-set-config.yaml

  • To mirror images to the TARGET_REGISTRY, first push the OCI image to a registry and update the image-set-config.yaml file to refer to the location where the image was pushed. You can push to the same TARGET_REGISTRY where all the images will be mirrored eventually. Support for referring OCI images directly in the image-set-config.yaml will be available in future release of oc mirror.

    oc mirror --config /root/.ibm-pak/data/publish/latest/image-set-config.yaml docker://$TARGET_REGISTRY --dest-skip-tls --max-per-registry=6

  • If product CASE doesn't support catalog mirroring but you have enabled oc mirror tool, generate mirror manifests command will generate image-set-config.yaml (without catalog curation) in ~/.ibm-pak/data/mirror directory.

    tree ~/.ibm-pak/data/mirror/
/root/.ibm-pak/data/mirror/
└── CASE-NAME
    └── CASE-VERSION
        ├── catalog-sources.yaml
        └── image-set-config.yaml

  • To mirror images to the TARGET_REGISTRY:

    oc mirror --config /root/.ibm-pak/data/mirror/<case-name>/<case-version>/image-set-config.yaml docker://$TARGET_REGISTRY --dest-skip-tls --max-per-registry=6

Online path

  • Starting with v1.8.0, plug-in supports generation of online manifests for the online consumption of products using oc ibm-pak generate online-manifests command.

  • The online manifests uses public image locations (i.e. icr.io is source of images). Hence, unlike mirror manifests,target-registry positional argument and --final-registry flag are not required.

  • The process for generating the catalog source is the same as the mirror manifests. But, the difference is that the catalog reference can be used as-is.

  • As mirroring is not required for online consumption, no ICSP needs to be generated.

  • Run the following command to generate online manifests

    oc ibm-pak generate online-manifests $CASE_NAME --version $CASE_VERSION

  • A new directory ~/.ibm-pak/online is created when you issue the oc ibm-pak generate online-manifests command. This directory holds the catalog-sources.yaml and images.txt (reference to souce images) files.

    tree ~/.ibm-pak/data/online
/root/.ibm-pak/data/online
└── CASE-NAME
    └── CASE-VERSION
        ├── catalog-sources.yaml
        └── images.txt

  • If product CASE supports catalog mirroring and you have enabled oc mirror tool, generate online manifests command will curate the catalog and generate the curated catalog in ~/.ibm-pak/data/publish/latest directory. The following tree shows an example of the ~/.ibm-pak/data/publish directory.

    tree ~/.ibm-pak/data/publish/
/root/.ibm-pak/data/publish/
└── latest
    ├── catalog
    │   └── catalog.json
    ├── catalog-oci
    │   └── manifest-list
    │       ├── blobs
    │       │   └── sha256
    │       │       ├── xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    │       │       ├── xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    │       │       └── xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    │       ├── index.json
    │       └── oci-layout
    ├── catalog-sources.yaml
    └── image-set-config.yaml

  • To copy the curated catalog to registry:

    skopeo copy --format v2s2 --all oci:///root/.ibm-pak/data/publish/latest/catalog-oci/manifest-list docker://<CATALOG_IMAGE_REF>

  • Note: In the above command and in /root/.ibm-pak/data/publish/latest/catalog-sources.yaml file, replace <CATALOG_IMAGE_REF> placeholder with an actual image reference where you want to copy the curated catalog. E.g setting <CATALOG_IMAGE_REF> to myregistry.com/images/curated-catalog:1.0, the above command becomes:

    skopeo copy --format v2s2 --all oci:///root/.ibm-pak/data/publish/latest/catalog-oci/manifest-list docker://myregistry.com/images/curated-catalog:1.0