EPIC: Prototype an OCM Download / Upload flow via Plugins

[jakobmoellerdev]

User Story: Delivery of a POC that can showcase the interaction of a Downloading and Uploading Plugin that interchange arbitrarily large (multiple gigabytes) of data locally (between two binaries) organized through an orchestrating core binary (this will later be OCM core).

Description:
As an OCM Ecosystem Developer, I want to have an easy to understand but also non-prohibitive way of including new access methods into OCM without having to contribute into the big OCM core.

To kickstart our ADR generation for the Plugin System, we should drive a POC that showcases how we can work with component versions in the future. We want to get rid of a monolithic system that forces people to understand the entirety of the OCM library and as such, the new binary system will kickstart a completely new version of OCM.

This POC should give the baseline of driving the ADR forwards for the new plugin system and should answer the following questions:

1. How do we make uploader and OCM repositories interact with each other so that plugins don’t need to access other plugins? AKA does all data flow need to go through the orchestrator?
2. We need a prototype that can showcase the reading of a component descriptor and interpreting both a correct uploader and downloader plugin and then establishing a byte stream from download to upload. We should start with CTFs and OCI. => The POC should have uploaders and downloaders from and to CTFs and OCI repositories.
3. We need a dynamic serialization system for type discovery => We should copy/clone the idea of current OCM and make the type system / decoder / encoder system work the same way for registrations (see compdesc package, v2 only)
4. We need a stable plugin system base and we should try out hashicorps plugin system because it allows for versioned plugins already (but if that turns out to be a wrong start because of complications, feel free to go back to a more basic system such as the one implemented in the current plugin approach)

Acceptance criteria:

• A CTF repository with a valid component version with basic attributes needs to be uploadable to an OCI registry. The syntax form of <REPO>//<Component>:<Version> should still work
• An OCI repository with a valid compoent version should be downloadable to a CTF
• The Uploader and Downloader for CTF (local files) and OCI should be in separate go modules and have 2 binaries
• The Uploader and Downloader for CTF and OCI should be selected based on the OCM Repository Type for the descriptor as well as the Access Type for resources
• The Uploader and Downloader for OCI and CTF should not communicate with each other and not have codependencies
• The Uploader and Downloader should not have to be forked as binaries N times for N resources. One transfer should trigger the start of a plugin once, even if it has many resources or descriptors.
• Credentials should be centrally discovered from the OCM configuration file and be passed securely to the plugin so that they cannot be intercepted

Related issues:

[Skarlso]

we should try out hashicorps plugin system because it allows for versioned plugins already

Heh. I contributed an extensive documentation to that thing. :D It's super nice, but we should be aware of the TCP / GRPC limitation. It's relying on local networking so VERY LARGE transfers might be a bit of a burdon there. But we'll have to try and play around with it. :)

[jakobmoellerdev]

Preemptive Flow considering these plugins:

sequenceDiagram
    actor User
    participant ocm
    participant plugin(CTF)
    participant plugin(OCI)
    participant plugin-registry


    User->>ocm: Transfer Component Versions From CTF to OCI
    rect rgba(100,50,100,0.5)
        critical Plugin Discovery
        Note right of ocm: Decode Request to determine necessary OCM Repository Types
            loop Every Needed OCM Repository Type (OCI, CTF)
                ocm->>plugin(CTF): Discover via Filesystem
                break Unavailable on Filesystem
                    ocm<<->>plugin-registry: Download Plugin from Registry
                end
                ocm<<->>plugin(CTF): GetCapabilities

                ocm->>plugin(OCI): Discover via Filesystem
                break Unavailable on Filesystem
                    ocm<<->>plugin-registry: Download Plugin from Registry
                end
                ocm<<->>plugin(OCI): GetCapabilities
            end
        end
    end

    rect rgba(50,50,100,0.5)
        critical Component Version Discovery
            option when more than one source version
                ocm<<->>plugin(CTF): ListComponentVersions
            option Recursive Resolution of Descriptors
                loop Component References
                    break Discover Repository Type of Component Reference
                    Note over ocm: This could be any OCM Repository type
                    ocm<<->>plugin(CTF): ListComponentVersions
                end
            end
        end
    end

    rect rgba(110,200,0,0.5)
        critical Transfer Process
            loop Every Discovered Component Version
                plugin(CTF)->>ocm: Download Component Descriptor

                loop Every Resource in Component Descriptor
                    plugin(CTF)<<->>ocm: Download Resource and Retrieve File Descriptor
                    ocm->>plugin(OCI): Upload Resource from File Descriptor and Update Resource Spec
                end

                ocm->>plugin(OCI): Upload Component Descriptor
            end
        end
    end
    ocm-->User: Confirmation of Transfer
    
    

Loading

[Skarlso]

• Create a new mono repo with the following layout (pocm)

	- .github
	- bindings
		- golang
			- descriptor bindings
			- ctf implementation
	- plugins
		- ctf
		- oci
	- cli

Individual tasks

Fabian

Create a Component Descriptor parser ( including the beginning of a plugin registration system ). Use a filepath for now, and throw an error if the plugin is not found.

Jakob + Gergely

Jakob

CTF Implementation

Gergely

• unix domain sockets -> implement this first, if it works as expected ( control channel bidirectional ) then we are done
• tcp + ports

Plugin implementation for CTF -> Talk with Jakob and agree on an API to use.

Everyone

• GetCapabilities -> Discuss how this looks like
• Discuss with other team members if they would like to do anything.

LocalBlob discussion

Every Plugin needs to have an implementation for the type localBlob.

OCMRepository -> This is a Capability -> To make it clear what a Plugin can do.