Code generation (for Java interop) throws exception on recent Kotlin versions

[fwcd]

While tinkering with #493, I've noticed that code generation seems to be broken when compiling the language server with Kotlin 1.9.10; the Kotlin compiler's JVM backend throws a lot of CompilationExceptions (see details for full stack trace).

[Error] org.jetbrains.kotlin.codegen.CompilationException: Back-end (JVM) Internal error: Failed to generate expression: KtNameReferenceExpression
    File being compiled: (2,5) in //Users/<user>/git/kotlin-language-server/server/build/resources/test/completions/BackquotedFunction.kt
    The root cause java.lang.AssertionError was thrown at: org.jetbrains.kotlin.codegen.ExpressionCodegen.visitSimpleNameExpression(ExpressionCodegen.java:1906)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.genQualified(ExpressionCodegen.java:356)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.genStatement(ExpressionCodegen.java:455)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.generateBlock(ExpressionCodegen.java:1336)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.generateBlock(ExpressionCodegen.java:1281)
    [Error]     at org.jetbrains.kotlin.codegen.CodegenStatementVisitor.visitBlockExpression(CodegenStatementVisitor.java:56)
    [Error]     at org.jetbrains.kotlin.codegen.CodegenStatementVisitor.visitBlockExpression(CodegenStatementVisitor.java:22)
    [Error]     at org.jetbrains.kotlin.psi.KtBlockExpression.accept(KtBlockExpression.java:79)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.genQualified(ExpressionCodegen.java:334)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.genStatement(ExpressionCodegen.java:455)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.gen(ExpressionCodegen.java:412)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.returnExpression(ExpressionCodegen.java:1840)
    [Error]     at org.jetbrains.kotlin.codegen.FunctionGenerationStrategy$FunctionDefault.doGenerateBody(FunctionGenerationStrategy.java:64)
    [Error]     at org.jetbrains.kotlin.codegen.FunctionGenerationStrategy$CodegenBased.generateBody(FunctionGenerationStrategy.java:86)
    [Error]     at org.jetbrains.kotlin.codegen.FunctionCodegen.generateMethodBody(FunctionCodegen.java:637)
    [Error]     at org.jetbrains.kotlin.codegen.FunctionCodegen.generateMethodBody(FunctionCodegen.java:481)
    [Error]     at org.jetbrains.kotlin.codegen.FunctionCodegen.generateMethod(FunctionCodegen.java:261)
    [Error]     at org.jetbrains.kotlin.codegen.FunctionCodegen.generateMethod(FunctionCodegen.java:166)
    [Error]     at org.jetbrains.kotlin.codegen.FunctionCodegen.gen(FunctionCodegen.java:137)
    [Error]     at org.jetbrains.kotlin.codegen.MemberCodegen.genSimpleMember(MemberCodegen.java:200)
    [Error]     at org.jetbrains.kotlin.codegen.PackagePartCodegen.generateBody(PackagePartCodegen.java:98)
    [Error]     at org.jetbrains.kotlin.codegen.MemberCodegen.generate(MemberCodegen.java:132)
    [Error]     at org.jetbrains.kotlin.codegen.PackageCodegenImpl.generateFile(PackageCodegenImpl.java:167)
    [Error]     at org.jetbrains.kotlin.codegen.PackageCodegenImpl.generate(PackageCodegenImpl.java:70)
    [Error]     at org.jetbrains.kotlin.codegen.DefaultCodegenFactory.generatePackage(CodegenFactory.kt:143)
    [Error]     at org.jetbrains.kotlin.codegen.DefaultCodegenFactory.invokeLowerings(CodegenFactory.kt:122)
    [Error]     at org.jetbrains.kotlin.codegen.CodegenFactory.generateModule(CodegenFactory.kt:44)
    [Error]     at org.jetbrains.kotlin.codegen.KotlinCodegenFacade.compileCorrectFiles(KotlinCodegenFacade.java:45)
    [Error]     at org.jetbrains.kotlin.codegen.KotlinCodegenFacade.compileCorrectFiles(KotlinCodegenFacade.java:54)
    [Error]     at org.javacs.kt.compiler.Compiler.generateCode(Compiler.kt:580)
    [Error]     at org.javacs.kt.SourcePath.save(SourcePath.kt:277)
    [Error]     at org.javacs.kt.SourcePath.saveAllFiles(SourcePath.kt:289)
    [Error]     at org.javacs.kt.KotlinTextDocumentService$lintAll$1.invoke(KotlinTextDocumentService.kt:270)
    [Error]     at org.javacs.kt.KotlinTextDocumentService$lintAll$1.invoke(KotlinTextDocumentService.kt:268)
    [Error]     at org.javacs.kt.util.Debouncer.submitImmediately$lambda$1(Debouncer.kt:27)
    [Error]     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    [Error]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    [Error]     at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    [Error]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    [Error]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    [Error]     at java.base/java.lang.Thread.run(Thread.java:829)
    [Error] Caused by: java.lang.AssertionError: Couldn't find descriptor for 'fu'
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.visitSimpleNameExpression(ExpressionCodegen.java:1906)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.visitSimpleNameExpression(ExpressionCodegen.java:127)
    [Error]     at org.jetbrains.kotlin.psi.KtNameReferenceExpression.accept(KtNameReferenceExpression.kt:59)
    [Error]     at org.jetbrains.kotlin.codegen.CodegenStatementVisitor.visitKtElement(CodegenStatementVisitor.java:31)
    [Error]     at org.jetbrains.kotlin.codegen.CodegenStatementVisitor.visitKtElement(CodegenStatementVisitor.java:22)
    [Error]     at org.jetbrains.kotlin.psi.KtVisitor.visitExpression(KtVisitor.java:186)
    [Error]     at org.jetbrains.kotlin.psi.KtVisitor.visitReferenceExpression(KtVisitor.java:202)
    [Error]     at org.jetbrains.kotlin.psi.KtVisitor.visitSimpleNameExpression(KtVisitor.java:198)
    [Error]     at org.jetbrains.kotlin.psi.KtNameReferenceExpression.accept(KtNameReferenceExpression.kt:59)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.genQualified(ExpressionCodegen.java:334)
    [Error]     ... 39 more
    [Error] org.jetbrains.kotlin.codegen.CompilationException: Back-end (JVM) Internal error: Failed to generate expression: KtNameReferenceExpression
    File being compiled: (4,13) in //Users/<user>/git/kotlin-language-server/server/build/resources/test/completions/Statics.kt
    The root cause java.lang.AssertionError was thrown at: org.jetbrains.kotlin.codegen.ExpressionCodegen.visitSimpleNameExpression(ExpressionCodegen.java:1906)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.genQualified(ExpressionCodegen.java:356)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.genQualified(ExpressionCodegen.java:316)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.visitDotQualifiedExpression(ExpressionCodegen.java:3497)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.visitDotQualifiedExpression(ExpressionCodegen.java:127)
    [Error]     at org.jetbrains.kotlin.psi.KtDotQualifiedExpression.accept(KtDotQualifiedExpression.kt:32)
    [Error]     at org.jetbrains.kotlin.codegen.CodegenStatementVisitor.visitKtElement(CodegenStatementVisitor.java:31)
    [Error]     at org.jetbrains.kotlin.codegen.CodegenStatementVisitor.visitKtElement(CodegenStatementVisitor.java:22)
    [Error]     at org.jetbrains.kotlin.psi.KtVisitor.visitExpression(KtVisitor.java:186)
    [Error]     at org.jetbrains.kotlin.psi.KtVisitor.visitQualifiedExpression(KtVisitor.java:290)
    [Error]     at org.jetbrains.kotlin.psi.KtVisitor.visitDotQualifiedExpression(KtVisitor.java:306)
    [Error]     at org.jetbrains.kotlin.psi.KtDotQualifiedExpression.accept(KtDotQualifiedExpression.kt:32)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.genQualified(ExpressionCodegen.java:334)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.genStatement(ExpressionCodegen.java:455)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.generateBlock(ExpressionCodegen.java:1336)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.generateBlock(ExpressionCodegen.java:1281)
    [Error]     at org.jetbrains.kotlin.codegen.CodegenStatementVisitor.visitBlockExpression(CodegenStatementVisitor.java:56)
    [Error]     at org.jetbrains.kotlin.codegen.CodegenStatementVisitor.visitBlockExpression(CodegenStatementVisitor.java:22)
    [Error]     at org.jetbrains.kotlin.psi.KtBlockExpression.accept(KtBlockExpression.java:79)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.genQualified(ExpressionCodegen.java:334)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.genStatement(ExpressionCodegen.java:455)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.gen(ExpressionCodegen.java:412)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.returnExpression(ExpressionCodegen.java:1840)
    [Error]     at org.jetbrains.kotlin.codegen.FunctionGenerationStrategy$FunctionDefault.doGenerateBody(FunctionGenerationStrategy.java:64)
    [Error]     at org.jetbrains.kotlin.codegen.FunctionGenerationStrategy$CodegenBased.generateBody(FunctionGenerationStrategy.java:86)
    [Error]     at org.jetbrains.kotlin.codegen.FunctionCodegen.generateMethodBody(FunctionCodegen.java:637)
    [Error]     at org.jetbrains.kotlin.codegen.FunctionCodegen.generateMethodBody(FunctionCodegen.java:481)
    [Error]     at org.jetbrains.kotlin.codegen.FunctionCodegen.generateMethod(FunctionCodegen.java:261)
    [Error]     at org.jetbrains.kotlin.codegen.FunctionCodegen.generateMethod(FunctionCodegen.java:166)
    [Error]     at org.jetbrains.kotlin.codegen.FunctionCodegen.gen(FunctionCodegen.java:137)
    [Error]     at org.jetbrains.kotlin.codegen.MemberCodegen.genSimpleMember(MemberCodegen.java:200)
    [Error]     at org.jetbrains.kotlin.codegen.PackagePartCodegen.generateBody(PackagePartCodegen.java:98)
    [Error]     at org.jetbrains.kotlin.codegen.MemberCodegen.generate(MemberCodegen.java:132)
    [Error]     at org.jetbrains.kotlin.codegen.PackageCodegenImpl.generateFile(PackageCodegenImpl.java:167)
    [Error]     at org.jetbrains.kotlin.codegen.PackageCodegenImpl.generate(PackageCodegenImpl.java:70)
    [Error]     at org.jetbrains.kotlin.codegen.DefaultCodegenFactory.generatePackage(CodegenFactory.kt:143)
    [Error]     at org.jetbrains.kotlin.codegen.DefaultCodegenFactory.invokeLowerings(CodegenFactory.kt:122)
    [Error]     at org.jetbrains.kotlin.codegen.CodegenFactory.generateModule(CodegenFactory.kt:44)
    [Error]     at org.jetbrains.kotlin.codegen.KotlinCodegenFacade.compileCorrectFiles(KotlinCodegenFacade.java:45)
    [Error]     at org.jetbrains.kotlin.codegen.KotlinCodegenFacade.compileCorrectFiles(KotlinCodegenFacade.java:54)
    [Error]     at org.javacs.kt.compiler.Compiler.generateCode(Compiler.kt:580)
    [Error]     at org.javacs.kt.SourcePath.save(SourcePath.kt:277)
    [Error]     at org.javacs.kt.SourcePath.saveAllFiles(SourcePath.kt:289)
    [Error]     at org.javacs.kt.KotlinTextDocumentService$lintAll$1.invoke(KotlinTextDocumentService.kt:270)
    [Error]     at org.javacs.kt.KotlinTextDocumentService$lintAll$1.invoke(KotlinTextDocumentService.kt:268)
    [Error]     at org.javacs.kt.util.Debouncer.submitImmediately$lambda$1(Debouncer.kt:27)
    [Error]     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    [Error]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    [Error]     at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    [Error]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    [Error]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    [Error]     at java.base/java.lang.Thread.run(Thread.java:829)
    [Error] Caused by: java.lang.AssertionError: Couldn't find descriptor for 'isN'
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.visitSimpleNameExpression(ExpressionCodegen.java:1906)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.visitSimpleNameExpression(ExpressionCodegen.java:127)
    [Error]     at org.jetbrains.kotlin.psi.KtNameReferenceExpression.accept(KtNameReferenceExpression.kt:59)
    [Error]     at org.jetbrains.kotlin.codegen.ExpressionCodegen.genQualified(ExpressionCodegen.java:334)
    [Error]     ... 50 more
    [Info] ..ounce126Linting .../completions/When.kt
    [Info] ..ounce126Reported 1 diagnostics in .../completions/When.kt
    [Info] async495  Completing at .../completions/When.kt 9:24

The codegen is invoked here:

kotlin-language-server/server/src/main/kotlin/org/javacs/kt/SourcePath.kt

Lines 272 to 283 in ba66720

	// If the code generation fails for some reason, we generate code for the other files anyway
	try {
	cp.compiler.removeGeneratedCode(listOfNotNull(it.lastSavedFile))
	it.module?.let { module ->
	it.compiledContext?.let { context ->
	cp.compiler.generateCode(module, context, listOfNotNull(it.compiledFile))
	it.lastSavedFile = it.compiledFile
	}
	}
	} catch (ex: Exception) {
	LOG.printStackTrace(ex)
	}

@daplf From what I can tell, you've added this for the JDT.LS extension, any thoughts? My best guess would be that a lot of internals changed due to the IR-based JVM backend.

[fwcd]

Looks like these errors are thrown mostly when the syntax tree is already broken (and apparently therefore contains nulls), e.g. fu in this test resource:

kotlin-language-server/server/src/test/resources/completions/BackquotedFunction.kt

Lines 1 to 3 in 8f96c29

	private fun completeBackquotedFunction() {
	fu
	}

[daplf]

@fwcd Thanks for the heads up! I hope to find some time to look at this soon. I like your idea here though:

[fwcd]

Yeah, that fixes some of the errors, but not all unfortunately. Feel free to dig deeper if you find the time, that branch contained a few experiments in that direction.

[derpda]

Not sure if this is related, but it is a CompilationException so I figured I'd post it here.

I'm getting this with both the 1.3.7 version downloaded automatically by the VSCode extension, and with a server built from from a7bc925 (1.3.5), so the cause for my issue may not be the Kotlin version as I believe that was changed in 1.3.6.

Click below for the error message.

Details

The root cause java.lang.IllegalStateException was thrown at: org.jetbrains.kotlin.codegen.state.KotlinTypeMapper$typeMappingConfiguration$1.processErrorType(KotlinTypeMapper.kt:124) [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.MemberCodegen.genSimpleMember(MemberCodegen.java:217) [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.ClassBodyCodegen.generateDeclaration(ClassBodyCodegen.java:170) [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.ClassBodyCodegen.generateBody(ClassBodyCodegen.java:88) [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.MemberCodegen.generate(MemberCodegen.java:132) [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.MemberCodegen.genClassOrObject(MemberCodegen.java:305) [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.MemberCodegen.genClassOrObject(MemberCodegen.java:289) [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.PackageCodegenImpl.generateClassesAndObjectsInFile(PackageCodegenImpl.java:119) [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.PackageCodegenImpl.generateFile(PackageCodegenImpl.java:138) [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.PackageCodegenImpl.generate(PackageCodegenImpl.java:70) [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.DefaultCodegenFactory.generatePackage(CodegenFactory.kt:143) [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.DefaultCodegenFactory.invokeLowerings(CodegenFactory.kt:122) [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.CodegenFactory.generateModule(CodegenFactory.kt:44) [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.KotlinCodegenFacade.compileCorrectFiles(KotlinCodegenFacade.java:45) [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.KotlinCodegenFacade.compileCorrectFiles(KotlinCodegenFacade.java:54) [Error - 4:39:19 PM] at org.javacs.kt.compiler.Compiler.generateCode(Compiler.kt:580) [Error - 4:39:19 PM] at org.javacs.kt.SourcePath.save(SourcePath.kt:277) [Error - 4:39:19 PM] at org.javacs.kt.SourcePath.saveAllFiles(SourcePath.kt:289) [Error - 4:39:19 PM] at org.javacs.kt.KotlinTextDocumentService$lintAll$1.invoke(KotlinTextDocumentService.kt:270) [Error - 4:39:19 PM] at org.javacs.kt.KotlinTextDocumentService$lintAll$1.invoke(KotlinTextDocumentService.kt:268) [Error - 4:39:19 PM] at org.javacs.kt.util.Debouncer.submitImmediately$lambda$1(Debouncer.kt:27) [Error - 4:39:19 PM] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [Error - 4:39:19 PM] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [Error - 4:39:19 PM] at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [Error - 4:39:19 PM] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [Error - 4:39:19 PM] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [Error - 4:39:19 PM] at java.base/java.lang.Thread.run(Thread.java:840) [Error - 4:39:19 PM] Caused by: java.lang.IllegalStateException: Error type encountered: [Error type: Unresolved type for MethodChannel] (ErrorType). [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.state.KotlinTypeMapper$typeMappingConfiguration$1.processErrorType(KotlinTypeMapper.kt:124) [Error - 4:39:19 PM] at org.jetbrains.kotlin.load.kotlin.DescriptorBasedTypeSignatureMappingKt.mapType(descriptorBasedTypeSignatureMapping.kt:83) [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.state.KotlinTypeMapper.mapType(KotlinTypeMapper.kt:276) [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.state.KotlinTypeMapper.mapType$default(KotlinTypeMapper.kt:267) [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.state.KotlinTypeMapper.mapType(KotlinTypeMapper.kt) [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.PropertyCodegen.generateBackingField(PropertyCodegen.java:409) [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.PropertyCodegen.generateBackingField(PropertyCodegen.java:360) [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.PropertyCodegen.genBackingFieldAndAnnotations(PropertyCodegen.java:170) [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.PropertyCodegen.gen(PropertyCodegen.java:138) [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.PropertyCodegen.gen(PropertyCodegen.java:101) [Error - 4:39:19 PM] at org.jetbrains.kotlin.codegen.MemberCodegen.genSimpleMember(MemberCodegen.java:211) [Error - 4:39:19 PM] ... 25 more

This is a gradle project that is part of a flutter plugin, and it seems that the error occurs during code generation for the MethodChannel, which is a class provided by flutter. Apologies if this is not actually related. I'll open a separate issue in that case.

[fwcd]

I have gated code generation behind the kotlin.codegen.enabled option for now, to avoid spamming users with exceptions. This unfortunately also means that Java interop will require setting this option, but given that compilation fails pretty frequently, I am not sure how reliable it worked on recent Kotlin versions anyway.

Once we figure out how to make this more stable, I would be more than happy to enable codegen by default again.