Add Native Build support

[ebma16]

I'm want to compile and run my operator as a native image in a setup with Kotlin/Java 21 + Spring Boot 3 + GraalVM.

So far Native Build is currently only explicitly supported by the Quarkus extension. A explicit Native Build support for Spring Boot is missing and has to be added.

If anybody knows already any examples or reference projects/repos that have done the same, please shout out :) Otherwise, I will look into adding support in a reasonable amount of time.

[bullshit]

A couple of weeks ago i tried to compile a spring boot based operator with graalvm v21. The operater uses spring cloud dependencies (kubernetes config and service discovery for example). i had to add many build args and RuntimeHints and reflection configuration to get it to work. Not sure how quarkus handles this. i'm wondering why i had to add the fabric8 stuff to it when native builds are supported by quarkus. figured they are using fabric8 as well

build.gradle.kts

dependencies {
    ...
    // fabric8 native compile
    implementation("org.reflections:reflections:0.10.2")
}
graalvmNative {
    binaries {
        named("main") {
            // fabric8 kubernetes client -> okHttp needs all charsets in native
            buildArgs.add("-H:+AddAllCharsets")

            // experimental features
            buildArgs.add("-H:+UnlockExperimentalVMOptions")
            //buildArgs.add("--enable-all-security-services")
            buildArgs.add("-H:+EnableSecurityServicesFeature")
            // @TODO: use gradle path selectors
            // because of ReconcilerUtils.loadYaml
            buildArgs.add("-H:ReflectionConfigurationFiles=../../resources/main/META-INF/native-image/reflection-config.json")
            buildArgs.add("-H:+ReportUnsupportedElementsAtRuntime")
            // nice to have
            buildArgs.add("-H:+ReportExceptionStackTraces")
            buildArgs.add("-H:+PrintClassInitialization")
        }
    }
}

KubernetesClientRuntimeHintsRegistrar.kt

import com.fasterxml.jackson.databind.annotation.JsonDeserialize
import com.fasterxml.jackson.databind.annotation.JsonSerialize
import io.fabric8.kubernetes.api.model.KubernetesResource
import io.fabric8.kubernetes.client.Client
import io.fabric8.kubernetes.client.VersionInfo
import org.apache.commons.logging.LogFactory
import org.reflections.Reflections
import org.springframework.aot.hint.MemberCategory
import org.springframework.aot.hint.RuntimeHints
import org.springframework.aot.hint.RuntimeHintsRegistrar
import org.springframework.aot.hint.registerType
import org.springframework.boot.autoconfigure.SpringBootApplication
import org.springframework.boot.runApplication
import org.springframework.context.annotation.ImportRuntimeHints

/* the following class is needed to configure fabric8 native compile */
@ImportRuntimeHints(KubernetesClientRuntimeHintsRegistrar::class)
class KubernetesClientRuntimeHintsRegistrar : RuntimeHintsRegistrar {

    private val log = LogFactory.getLog(javaClass)

    override fun registerHints(hints: RuntimeHints, classLoader: ClassLoader?) {
        hints.resources().registerPattern("reflection-config.json");

        hints.resources().registerPattern("META-INF/services/io.fabric8.kubernetes.api.model.KubernetesResource")
        hints.resources().registerPattern("META-INF/services/io.fabric8.kubernetes.client.http.HttpClient\$Factory")
        hints.reflection().registerType<VersionInfo>(*MemberCategory.values())
        registerClients(hints)
        registerJacksonKubernetesModels(hints)
    }

    private fun registerClients(hints: RuntimeHints) {
        val clazz = Client::class.java
        val reflections = Reflections(clazz.packageName, clazz)
        val clients = reflections.getSubTypesOf(Client::class.java) + Client::class.java

        for (client in clients) {
            hints.reflection().registerType(client, *MemberCategory.values())
            log.info { "registering ${client.name} for reflection" }
        }
    }

    private fun registerJacksonKubernetesModels(hints: RuntimeHints) {
        val clazz = KubernetesResource::class.java
        val reflections = Reflections(clazz.packageName, clazz)
        val kubernetesModels = reflections.getSubTypesOf(KubernetesResource::class.java)
        val combined = buildSet {
            addAll(kubernetesModels)
            addAll(resolveSerializationClasses<JsonSerialize>(reflections))
            addAll(resolveSerializationClasses<JsonDeserialize>(reflections))
        }

        for (model in combined) {
            hints.reflection().registerType(model, *MemberCategory.values())
            log.info { "registering ${model.name} for reflection" }
        }
    }

    /**
     * Extracts Jacksons Deserializer / Serializers specified in the classes annotations
     */
    private inline fun <reified R : Annotation> resolveSerializationClasses(reflections: Reflections): List<Class<*>> {
        val clazz = R::class.java
        val method = clazz.getMethod("using")
        val classes = reflections.getTypesAnnotatedWith(clazz)

        return classes.mapNotNull { clazzWithAnnotation ->
            val annotation = clazzWithAnnotation.getAnnotation(clazz)
            if (annotation != null) method.invoke(annotation) as Class<*> else null
        }
    }
}

reflection-config.json

[ { "name": "io.javaoperatorsdk.operator.processing.retry.GenericRetry", "allDeclaredConstructors": true, "allPublicConstructors": true, "allDeclaredMethods": true, "allPublicMethods": true, "allDeclaredClasses": true, "allPublicClasses": true }, { "name": "io.javaoperatorsdk.operator.processing.event.rate.LinearRateLimiter", "allDeclaredConstructors": true, "allPublicConstructors": true, "allDeclaredMethods": true, "allPublicMethods": true, "allDeclaredClasses": true, "allPublicClasses": true }, { "name": "pl.ows.springoperator.dependentresource.DeploymentDependentResource", "allDeclaredConstructors": true, "allPublicConstructors": true, "allDeclaredMethods": true, "allPublicMethods": true }, { "name": "io.javaoperatorsdk.operator.processing.dependent.kubernetes.KubernetesDependentConverter", "allDeclaredConstructors": true, "allPublicConstructors": true, "allDeclaredMethods": true, "allPublicMethods": true, "allDeclaredClasses": true, "allPublicClasses": true }, { "name": "pl.ows.springoperator.dependentresource.ServiceDependentResource", "allDeclaredConstructors": true, "allPublicConstructors": true, "allDeclaredMethods": true, "allPublicMethods": true, "allDeclaredClasses": true, "allPublicClasses": true }, { "name": "java.util.TreeMap", "allDeclaredConstructors": true, "allPublicConstructors": true, "allDeclaredMethods": true, "allPublicMethods": true, "allDeclaredClasses": true, "allPublicClasses": true }, { "name": "pl.ows.springoperator.customresource.SpringAppSpec", "allDeclaredConstructors": true, "allPublicConstructors": true, "allDeclaredMethods": true, "allPublicMethods": true, "allDeclaredClasses": true, "allPublicClasses": true }, { "name": "pl.ows.springoperator.customresource.SpringAppStatus", "allDeclaredConstructors": true, "allPublicConstructors": true, "allDeclaredMethods": true, "allPublicMethods": true, "allDeclaredClasses": true, "allPublicClasses": true }, { "name": "io.fabric8.kubernetes.api.model.IntOrString", "allDeclaredConstructors": true, "allPublicConstructors": true, "allDeclaredMethods": true, "allPublicMethods": true, "allDeclaredClasses": true, "allPublicClasses": true }, { "name": "io.fabric8.kubernetes.api.model.Config", "allDeclaredConstructors": true, "allPublicConstructors": true, "allDeclaredMethods": true, "allPublicMethods": true, "allDeclaredClasses": true, "allPublicClasses": true }, { "name": "io.fabric8.kubernetes.api.model.Preferences", "allDeclaredConstructors": true, "allPublicConstructors": true, "allDeclaredMethods": true, "allPublicMethods": true, "allDeclaredClasses": true, "allPublicClasses": true }, { "name": "io.fabric8.kubernetes.client.impl.KubernetesClientImpl", "allDeclaredConstructors": true, "allPublicConstructors": true, "allDeclaredMethods": true, "allPublicMethods": true, "allDeclaredClasses": true, "allPublicClasses": true } ]

[ebma16]

Hey @bullshit,

many thank's for your input! It's "calming" to see that you had to struggle with similar problems as me..

I used a similar principle. In the beginning I also added single dependencies, but when I added about 40-50 dependencies manually and it still didn't work, I stopped doing that and tried to load necessary dependencies more "generically", but this leads to loading more dependencies than I might really need (but I hope that my solution is now more robust and future changes in fabric8 or josdk won't have any impact so quickly..).

Unfortunately my operator application doesn't work with your solution either, I had to load more dependencies via reflection. My setup:
I have an operator with 3 different reconcilers. Each reconciler manages a CRD as well as certain DependentResources (Deployments, ConfigMaps, Secrets, ...). The reconcilers also use EventSources to listen to events from the other custom resources. I also have a validation webhook running that validates custom resources when they are created.

I've been playing around with my solution and yours to see the effects of loading more dependencies on the native image size..
Here some facts..

• native image size without loading any additional dependencies via reflection: 180 MB
• native image size loading my internal dependencies (1 MB) + your solution (2 MB): 183 MB
• native image size loading my internal dependencies (1 MB) + my fabric8 solution (32 MB) + my josdk solution (2MB): 215 MB

I generally

• loaded the entire JOSDK project (loading the entire JOSDK and limiting it to loading only necessary classes made only 2-3 MB difference)
• loaded a lot of Fabric8 classes (these make the main difference between my solution and yours).

That's my solution:

OperatorRuntimeHintsRegistrar.kt

import com.fasterxml.jackson.databind.annotation.JsonDeserialize
import com.fasterxml.jackson.databind.annotation.JsonSerialize
import io.fabric8.kubernetes.api.model.IntOrString
import io.fabric8.kubernetes.api.model.KubernetesResource
import io.fabric8.kubernetes.client.Client
import io.fabric8.kubernetes.client.extension.ExtensionAdapter
import io.fabric8.kubernetes.internal.KubernetesDeserializer
import org.reflections.Reflections
import org.reflections.scanners.SubTypesScanner
import org.reflections.scanners.TypeAnnotationsScanner
import org.slf4j.LoggerFactory
import org.springframework.aot.hint.MemberCategory
import org.springframework.aot.hint.RuntimeHints
import org.springframework.aot.hint.RuntimeHintsRegistrar
import java.util.*
import java.util.stream.Collectors


/**
 * This class provides functionality for registering runtime hints via reflection to include required depdencies that
 * are usually not included when building a Native Image.
 * It scans and registers internal types, dependencies required by Fabric8 and dependencies required by the
 * Java Operator SDK.
 */
class OperatorRuntimeHintsRegistrar : RuntimeHintsRegistrar {

    companion object {
        private val logger = LoggerFactory.getLogger(OperatorRuntimeHintsRegistrar::class.java)
    }

    private val reflectionsFabric8 = Reflections(
        "io.fabric8", TypeAnnotationsScanner(), SubTypesScanner(false)
    )

    private val reflectionsOperatorSDK = Reflections(
        "io.javaoperatorsdk", TypeAnnotationsScanner(), SubTypesScanner(false)
    )

    override fun registerHints(hints: RuntimeHints, classLoader: ClassLoader?) {
        registerInternalTypes(hints)
        registerFabric8Types(hints)
        registerJosdkTypes(hints)
    }

    /**
     * Register internal dependencies
     */
    private fun registerInternalTypes(hints: RuntimeHints) {
        val internalTypes = hashSetOf(
            ...some internal dependencies...
        )

        registerTypes(internalTypes, hints)
    }

    /**
     * Register default dependencies required by fabric8.
     * The required dependencies are based on own tests combined with this exploration: https://github.com/kubernetes-native-java/fabric8-spring-native/tree/main
     */
    private fun registerFabric8Types(hints: RuntimeHints) {

        val impls = reflectionsFabric8.allTypes.stream()
            .filter { it.endsWith("Impl") }
            .map { Class.forName(it) }
            .collect(Collectors.toSet())

        val othersToAddForReflection =
            mutableListOf(KubernetesDeserializer::class.java, IntOrString::class.java, TreeMap::class.java)

        val fabric8Types = HashSet<Class<*>>()
        fabric8Types.addAll(impls)
        fabric8Types.addAll(othersToAddForReflection)
        fabric8Types.addAll(reflectionsFabric8.getSubTypesOf(ExtensionAdapter::class.java))
        fabric8Types.addAll(reflectionsFabric8.getSubTypesOf(Client::class.java) + Client::class.java)
        fabric8Types.addAll(reflectionsFabric8.getSubTypesOf(KubernetesResource::class.java))
        fabric8Types.addAll(resolveSerializationClasses(JsonSerialize::class.java))
        fabric8Types.addAll(resolveSerializationClasses(JsonDeserialize::class.java))

        registerTypes(fabric8Types, hints)
    }

    /**
     * Register default dependencies required by java-operator-sdk.
     * Let's register all classes of josdk to ensure it is also working in future when adding new imlpementations.
     *   --> Native Image with all josdk dependencies is just ~1-2MB bigger than with minimal required dependencies.
     */
    private fun registerJosdkTypes(hints: RuntimeHints) {
        val josdkTypes = reflectionsOperatorSDK.getSubTypesOf(Object::class.java)
        registerTypes(josdkTypes, hints)
    }

    private fun registerTypes(
        internalDependencies: Set<Class<out Any>>,
        hints: RuntimeHints
    ) {
        internalDependencies
            .filter { Objects.nonNull(it) }
            .forEach {
                logger.info("trying to register " + it.name + " for reflection")
                hints.reflection().registerType(it, *MemberCategory.entries.toTypedArray())
            }
    }

    private fun <R : Annotation?> resolveSerializationClasses(annotationClazz: Class<R>): Set<Class<*>> {

        logger.debug("trying to resolve types annotated with " + annotationClazz.name)
        val method = annotationClazz.getMethod("using")
        val classes = reflectionsFabric8.getTypesAnnotatedWith(annotationClazz)

        return classes
            .mapNotNull {
                logger.debug("found " + it.name + " : " + annotationClazz.name)
                val annotation = it.getAnnotation(annotationClazz)
                if (annotation != null) {
                    method.invoke(annotation) as Class<*>
                } else {
                    null
                }
            }.toSet()
    }
}

[bullshit]

Hello @ebma16, did you measure your startup times as well? for me the differences where 7s normal start from IDE and 400ms with a native binary.

I'm to new to native builds to understand how quarkus manages to build native when fabric8 does not provide runtime hints.

I stumbled across this project https://github.com/kubernetes-native-java/fabric8-spring-native but is outdated and the intention was educational.

[ebma16]

Hello @bullshit,

I have not analysed the start time exactly, I have only started the application of the respective build three times in a row (I always created the image manually and loaded it into a local Kubernetes cluster where the application was started -> so same circumstance/config, but different images):
Native build (215 MB):

• Started OperatorApplicationKt in 0.087 seconds
• Started OperatorApplicationKt in 0.090 seconds
• Started OperatorApplicationKt in 0.090 seconds

Normal build (408 MB):

• Started OperatorApplicationKt in 2.926 seconds
• Started OperatorApplicationKt in 2.957 seconds
• Started OperatorApplicationKt in 2.910 seconds

I found the same project while researching how to create a native image with fabric8. I also had to adapt it as it still refers to Spring Boot 2, but it was a good basis and took a few tests/investigations off my hands :)

[heruan]

I'm also very much interested in this. Spring provides APIs to declare runtime hints for native image builds, such as RuntimeHintsRegistrar (see for example JacksonRuntimeHints). Basically any class that is inspected via reflection in JOSDK needs hints to work on a native image.

[heruan]

I tested around this for a while and for me this is working:

class JosdkRuntimeHints implements RuntimeHintsRegistrar {

    @Override
    public void registerHints(RuntimeHints hints, ClassLoader classLoader) {
        hints.reflection().registerTypes(
                TypeReference.listOf(GenericRetry.class, LinearRateLimiter.class, ControllerConfiguration.class),
                TypeHint.builtWith(MemberCategory.INVOKE_PUBLIC_CONSTRUCTORS, MemberCategory.INVOKE_PUBLIC_METHODS));
    }
}

This could be added to this module and registered in some @AutoConfiguration class by annotating it with

@RegisterRuntimeHints(JosdkRuntimeHints.class)

Note: I'm not sure if the registered types are enough, as I have very little knowledge of the JOSDK internals. I've just added those that caused exceptions for my very simple operator.