GraalVM
- GraalVM Native Image is an ahead-of-time compilation technology that generates native platform executables.
- Native executables are ideal for containers and cloud deployments as they are small, start very fast, and require significantly less CPU and memory.
- Deploy native executables on distroless and even Scratch container images for reduced size and improved security.
- With profile-guided optimization and the G1 garbage collector, native executables built with GraalVM Native Image can achieve peak throughput on par with the JVM.
- GraalVM Native Image enjoys significant adoption with support from leading Java frameworks such as Spring Boot, Micronaut, Quarkus, Gluon Substrate, etc.
Difference with traditional JVM application
Frameworks that support GraalVM
Benchmark
Scenario
- perform EDF+ conversion
Comparison
- application start time
- memory usage
- cpu usage
- conversion time (should not differ from each framework)
- docker image size
- framework documentations
- framework ease of use / bootstrap
- framework ease of deployment
- framework developer experience
Issues encounters
Build time and resource consumption
- really long build time (X times longer than traditional build)
- traditional build: ~15s
- native build: ~3m38s
- native build takes lots of resources (CPU+RAM)
- sometimes, my IDE / teams are killed because the build is stressing my computer a lot
- sometimes, my whole computer freezes
- no immediate feedback if something does not work in native mode because when we are developing, we are not constantly building the app in native
- newest version of GraalVM includes a faster compilation for development, hence not to be used for compiling for production
Native build errors encountered
UnsupportedFeatureException
I got the following error when trying to build the native app:
Error: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Detected an instance of Random/SplittableRandom class in the image heap. Instances created during image generation have cached seed values and don't behave as expected. To see how this object got instantiated use --trace-object-instantiation=java.util.Random. The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image runtime by using the option --initialize-at-run-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
According to the documentation (https://quarkus.io/guides/native-reference#build-time-vs-run-time-initialization), adding
-Dquarkus.native.additional-build-args=“—trace-object-instantiation=java.security.SecureRandom”` can help use track where the usage of this class is located.
In this poc, it was used by:
- annotations-service HTTP client
TempFileCreator.createTempFilePath
- AWS SDK apache HTTP client
The fix suggested by the documentation is to not call to Random
or SplittableRandom
in static methods/constructors. Or add the following property in the application.properties
to init at runtime instead:
We can mitigate for our code in TempFileCreator
, however, it’s not possible to change the behavior for libraries that we depends on… This is quite a drawbacks here.
Smallrye no Converter registered for class java.net.URI
The converter-worker is using a URI
as property for com.bioserenity.converter.application.quarkus.config.aws.AwsProperties.S3#endpoint
. It seems it’s not supported in native (don’t know why).
Mitigation
Transform into a String
instead.
SdkClientException: Unable to load an HTTP implementation
When starting the app in native, I got the following error:
2022-04-30 12:12:24,910 ERROR [io.qua.run.Application] (main) Failed to start application (with profile prod): software.amazon.awssdk.core.exception.SdkClientException: Unable to load an HTTP implementation from any provider in the chain. You must declare a dependency on an appropriate HTTP implementation or pass in an SdkHttpClient explicitly to the client builder.
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98)
at software.amazon.awssdk.core.internal.http.loader.DefaultSdkHttpClientBuilder.lambda$buildWithDefaults$1(DefaultSdkHttpClientBuilder.java:49)
at java.util.Optional.orElseThrow(Optional.java:403)
at software.amazon.awssdk.core.internal.http.loader.DefaultSdkHttpClientBuilder.buildWithDefaults(DefaultSdkHttpClientBuilder.java:43)
at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.lambda$resolveSyncHttpClient$5(SdkDefaultClientBuilder.java:279)
at java.util.Optional.orElseGet(Optional.java:364)
at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.resolveSyncHttpClient(SdkDefaultClientBuilder.java:279)
at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.finalizeSyncConfiguration(SdkDefaultClientBuilder.java:229)
at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.syncClientConfiguration(SdkDefaultClientBuilder.java:162)
at software.amazon.awssdk.services.s3.DefaultS3ClientBuilder.buildClient(DefaultS3ClientBuilder.java:27)
at software.amazon.awssdk.services.s3.DefaultS3ClientBuilder.buildClient(DefaultS3ClientBuilder.java:22)
at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.build(SdkDefaultClientBuilder.java:133)
at com.bioserenity.converter.application.quarkus.config.aws.AwsConfig.s3Client(AwsConfig.java:33)
Mitigation
When building the AWS clients, we must set the HTTP client:
- add dependency
Configure clients:
ManagedChannelProvider$ProviderNotFoundException: No functional channel service provider found
io.grpc.ManagedChannelProvider$ProviderNotFoundException: No functional channel service provider found. Try adding a dependency on the grpc-okhttp, grpc-netty, or grpc-netty-shaded artifact
at io.grpc.ManagedChannelProvider.provider(ManagedChannelProvider.java:43)
at io.grpc.ManagedChannelBuilder.forTarget(ManagedChannelBuilder.java:76)
at com.bioserenity.converter.application.quarkus.config.ConverterWorkerConfig.recordServiceChannel(ConverterWorkerConfig.java:69)
Mitigation
- add dependency
- configure gRPC channel
- add the following dependency:
IllegalArgumentException: cannot find a NameResolver for record.swpdev.bioserenity.cloud:9090
java.lang.IllegalArgumentException: cannot find a NameResolver for record.swpdev.bioserenity.cloud:9090
at io.grpc.internal.ManagedChannelImpl.getNameResolver(ManagedChannelImpl.java:776)
at io.grpc.internal.ManagedChannelImpl.getNameResolver(ManagedChannelImpl.java:785)
at io.grpc.internal.ManagedChannelImpl.<init>(ManagedChannelImpl.java:665)
at io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:630)
at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:297)
at com.bioserenity.converter.application.quarkus.config.ConverterWorkerConfig.recordServiceChannel(ConverterWorkerConfig.java:75)
Mitigation
- configure DNS name resolver manually
ClassNotFoundException: com.github.benmanes.caffeine.cache.SSMSW
java.lang.ClassNotFoundException: com.github.benmanes.caffeine.cache.SSMSW
at java.lang.Class.forName(DynamicHub.java:1338)
at java.lang.Class.forName(DynamicHub.java:1313)
at com.github.benmanes.caffeine.cache.LocalCacheFactory.newBoundedLocalCache(LocalCacheFactory.java:87)
at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache.<init>(BoundedLocalCache.java:3423)
at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache.<init>(BoundedLocalCache.java:3419)
at com.github.benmanes.caffeine.cache.Caffeine.build(Caffeine.java:1078)
at com.bioserenity.converter.infrastructure.swp.CachedM2MPassportFromAuthenticatedEntity.create(CachedM2MPassportFromAuthenticatedEntity.java:29)
Mitigation
https://github.com/ben-manes/caffeine/issues/434#issuecomment-653516398 https://quarkus.io/guides/writing-native-applications-tips#including-resources
- create a
src/main/resources/reflection-config.json
with the following content:
- add the flag to take this
reflection-config.json
file:
SqsJobMessageConsumer - Failed to deserialize Job
When testing a conversion, the quarkus application in native could not deserialize the SQS message body:
11:15:19.625 [main] [ERROR] c.b.c.i.w.sqs.SqsJobMessageConsumer - Failed to deserialize Job : {"id":"8141006c-1fb0-42c6-a25a-fe9d105ff9f8","recordKey":"wemu55618","creationTimestamp":1651403717,"conversionRequest":{"sourceFileFormat":"BIOSERENITY","targetFileFormat":"EDF_PLUS","targetSoftware":"PERSYST","timeZone":"UTC","patientIdentification":{"code":"foobar"}}}
om.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `com.bioserenity.converter.domain.model.ConversionRequest` (no Creators, like default constructor, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
at [Source: (String)"{"id":"863896d5-132d-4164-a27d-ef935503495d","recordKey":"wemu55618","creationTimestamp":1651408817,"conversionRequest":{"sourceFileFormat":"BIOSERENITY","targetFileFormat":"EDF_PLUS","targetSoftware":"PERSYST","timeZone":"UTC","patientIdentification":{"code":"foobar"}}}"; line: 1, column: 122] (through reference chain: com.bioserenity.converter.domain.model.Job["conversionRequest"])
Mitigation
https://quarkus.io/guides/writing-native-applications-tips#registering-for-reflection
Add Job
to be registered for reflection:
IllegalStateException: Could not find policy ‘pick_first’
2022-05-01 12:58:49,610 SEVERE [io.grp.int.ManagedChannelImpl] (main) [Channel<1>: (record.swpdev.bioserenity.cloud:9090)] Uncaught exception in the SynchronizationContext. Panic!: java.lang.IllegalStateException: Could not find policy 'pick_first'. Make sure its implementation is either registered to LoadBalancerRegistry or included in META-INF/services/io.grpc.LoadBalancerProvider from your jar files.
at io.grpc.internal.AutoConfiguredLoadBalancerFactory$AutoConfiguredLoadBalancer.<init>(AutoConfiguredLoadBalancerFactory.java:92)
at io.grpc.internal.AutoConfiguredLoadBalancerFactory.newLoadBalancer(AutoConfiguredLoadBalancerFactory.java:63)
Mitigation
- register loadbalancer provider
Memory usage increasing overtime in ladder form
The container memory usage is increasing overtime, even if the application does nothing, until it reaches ~250MB.
UnresolvedElementException: OkHttp
Error: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved method during parsing: okhttp3.RequestBody.create(java.lang.String, okhttp3.MediaType). This error is reported at image build time because class com.bioserenity.annotations.client.ApiClient is registered for linking at image build time by command line
Trace:
at parsing com.bioserenity.annotations.client.ApiClient.buildRequest(ApiClient.java:1098)
Call path from entry point to com.bioserenity.annotations.client.ApiClient.buildRequest(String, String, List, List, Object, Map, Map, Map, String[], ApiCallback):
at com.bioserenity.annotations.client.ApiClient.buildRequest(ApiClient.java:1072)
at com.bioserenity.annotations.client.ApiClient.buildCall(ApiClient.java:1050)
at com.bioserenity.annotations.client.api.QueryAnnotationsControllerApi.queryCall(QueryAnnotationsControllerApi.java:106)
Mitigation
There are conflicts on the okhttp clients… The one provided by the annotations-service does not match the one needed by the GraalVM/Quarkus to compile the project.
⇒ refactored exg-file-exporter
to use Java HttpClient
instead of the auto-generated client for communicating with annotations-service.
UnresolvedElementException: gRPC shaded Log4J2LoggerFactory & Log4JLoggerFactory
Error: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved type during parsing: io.grpc.netty.shaded.io.netty.util.internal.logging.Log4J2Logger. This error is reported at image build time because class io.grpc.netty.shaded.io.netty.util.internal.logging.Log4J2LoggerFactory is registered for linking at image build time by command line
Trace:
at parsing io.grpc.netty.shaded.io.netty.util.internal.logging.Log4J2LoggerFactory.newInstance(Log4J2LoggerFactory.java:33)
Call path from entry point to io.grpc.netty.shaded.io.netty.util.internal.logging.Log4J2LoggerFactory.newInstance(String):
no path found from entry point to target method
Error: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved type during parsing: org.apache.log4j.Logger. This error is reported at image build time because class io.grpc.netty.shaded.io.netty.util.internal.logging.Log4JLoggerFactory is registered for linking at image build time by command line
Trace:
at parsing io.grpc.netty.shaded.io.netty.util.internal.logging.Log4JLoggerFactory.newInstance(Log4JLoggerFactory.java:38)
Call path from entry point to io.grpc.netty.shaded.io.netty.util.internal.logging.Log4JLoggerFactory.newInstance(String):
no path found from entry point to target method
The library grpc-netty-shaded
has a io.grpc.netty.shaded.io.netty.util.internal.logging.InternalLoggerFactory
that has logic in static
to instanciate a LoggerFactory
depending on which is in the classpath, i.e., slf4j, log4j2 or log4j.
It’s the “incomplete classpath” issue mentioned above.
Mitigation
Add dependency (in provided scope):
Create new class:
UnresolvedElementException: PackagingDataCalculator
Error: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved type during parsing: sun.reflect.Reflection. This error is reported at image build time because class ch.qos.logback.classic.spi.PackagingDataCalculator is registered for linking at image build time by command line
Trace:
at parsing ch.qos.logback.classic.spi.PackagingDataCalculator.populateFrames(PackagingDataCalculator.java:85)
Call path from entry point to ch.qos.logback.classic.spi.PackagingDataCalculator.populateFrames(StackTraceElementProxy[]):
at ch.qos.logback.classic.spi.PackagingDataCalculator.populateFrames(PackagingDataCalculator.java:72)
at ch.qos.logback.classic.spi.PackagingDataCalculator.calculate(PackagingDataCalculator.java:58)
at ch.qos.logback.classic.spi.ThrowableProxy.calculatePackagingData(ThrowableProxy.java:142)
at ch.qos.logback.classic.spi.LoggingEvent.<init>(LoggingEvent.java:122)
at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:419)
at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
at ch.qos.logback.classic.Logger.info(Logger.java:579)
at com.bioserenity.converter.domain.worker.usecase.ConversionWorker.run(ConversionWorker.java:29)
Mitigation
- remove dependency to logback
KubernetesClientException: JcaPEMKeyConverter is provided by BouncyCastle
I bumped to quarkus 2.9.1.Final, and I got a new build error:
[ERROR] [error]: Build step io.quarkus.kubernetes.client.deployment.KubernetesClientBuildStep#process threw an exception: io.fabric8.kubernetes.client.KubernetesClientException: JcaPEMKeyConverter is provided by BouncyCastle, an optional dependency. To use support for EC Keys you must explicitly add this dependency to classpath.
[ERROR] at io.fabric8.kubernetes.client.internal.CertUtils.handleECKey(CertUtils.java:164)
[ERROR] at io.fabric8.kubernetes.client.internal.CertUtils.loadKey(CertUtils.java:134)
[ERROR] at io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:112)
[ERROR] at io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:247)
[ERROR] at io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:153)
[ERROR] at io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:147)
[ERROR] at io.fabric8.kubernetes.client.utils.HttpClientUtils.applyCommonConfiguration(HttpClientUtils.java:204)
[ERROR] at io.fabric8.kubernetes.client.okhttp.OkHttpClientFactory.createHttpClient(OkHttpClientFactory.java:89)
Mitigation
https://github.com/quarkusio/quarkus/issues/12417
- remove
~/.kube/config
- not quite a good solution…
Error on runtime
InvalidDefinitionException: Failed to access RecordComponents
Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Failed to access RecordComponents of type `com.bioserenity.format.convert.bios2edfplus.read.annotation.model.AnnotationQueryFilterInput`
Mitigation
https://github.com/oracle/graal/issues/3984
Java 17 record is only supported for newer GraalVM versions (22.1.0+ and 21.3.2+) ⇒ upgrade GraalVM:
- add the following dependency:
- add the classes to be registered for reflection
InvalidDefinitionException: Cannot construt instance of AnnotationViewOutput
Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `com.bioserenity.annotations.client.model.AnnotationViewOutput` (no Creators, like default constructor, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
Mitigation
- add the classes to be registered for reflection
LogConfigurationException: No suitable Log implementation
2022-05-20 07:58:04,312 ERROR [io.qua.run.Application] (main) Failed to start application (with profile prod): org.apache.commons.logging.LogConfigurationException: No suitable Log implementation
at org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:848)
at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:541)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:292)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:269)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:655)
at org.apache.http.conn.ssl.DefaultHostnameVerifier.<init>(DefaultHostnameVerifier.java:82)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.getDefaultHostnameVerifier(SSLConnectionSocketFactory.java:183)
at software.amazon.awssdk.http.apache.ApacheHttpClient$ApacheConnectionManagerFactory.getHostNameVerifier(ApacheHttpClient.java:669)
at software.amazon.awssdk.http.apache.ApacheHttpClient$ApacheConnectionManagerFactory.getPreferredSocketFactory(ApacheHttpClient.java:663)
at software.amazon.awssdk.http.apache.ApacheHttpClient$ApacheConnectionManagerFactory.create(ApacheHttpClient.java:641)
at software.amazon.awssdk.http.apache.ApacheHttpClient.createClient(ApacheHttpClient.java:156)
at software.amazon.awssdk.http.apache.ApacheHttpClient.<init>(ApacheHttpClient.java:130)
at software.amazon.awssdk.http.apache.ApacheHttpClient.<init>(ApacheHttpClient.java:109)
at software.amazon.awssdk.http.apache.ApacheHttpClient$DefaultBuilder.buildWithDefaults(ApacheHttpClient.java:633)
at software.amazon.awssdk.http.SdkHttpClient$Builder.build(SdkHttpClient.java:69)
at software.amazon.awssdk.http.apache.ApacheHttpClient.create(ApacheHttpClient.java:145)
at com.bioserenity.converter.application.quarkus.config.aws.AwsConfig.s3Client(AwsConfig.java:35)
at com.bioserenity.converter.application.quarkus.config.aws.AwsConfig_ProducerMethod_s3Client_82329382abfbbb6ffd3acf2778dbd46d3a4a87e1_Bean.create(Unknown Source)
at com.bioserenity.converter.application.quarkus.config.aws.AwsConfig_ProducerMethod_s3Client_82329382abfbbb6ffd3acf2778dbd46d3a4a87e1_Bean.create(Unknown Source)
Mitigation
- check which dependency has
commons-logging
- remove
commons-logging
from dependency:
ClassNotFoundException: LogFactoryImpl
Caused by java.lang.ClassNotFoundException: org.apache.commons.logging.impl.LogFactoryImpl
Mitigation
https://quarkus.io/guides/logging#logging-adapters
Resources
- https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/Limitations.md
- https://www.graalvm.org/22.1/reference-manual/native-image/Logging/
- https://www.graalvm.org/22.1/reference-manual/native-image/Reflection/
- https://simply-how.com/fix-graalvm-native-image-compilation-issues
- https://medium.com/graalvm/instant-netty-startup-using-graalvm-native-image-generation-ed6f14ff7692
- https://blog.frankel.ch/solving-substitution-graalvm-issue/
- https://www.infoq.com/articles/native-java-graalvm/