Skip to content

Commit

Permalink
PSI based parser (#309)
Browse files Browse the repository at this point in the history 
* Add `mapAllElements()` function

Populates a new `MutableMap<PsiElement, MutableList<FirElement>>` map.

* Add Fir tree printer && Implenment 1st version psi->fir mapping and verify type

* Tweak and disable some unrelated tests

* clean up

* Refactor code into a new `PsiElementAssociations` class

* Refactor to not depend on source offsets anymore

* Enable disabled tests again

* Validate more type attribution

* mark some expected failing tests with reasion

* update and more ExpectedToFail with reasons

* Add the psi based parser and have 1st test pass

* add variable type

* Handle comments and support nested comments

* Make the NestedComments test case more complicated

* support EOL_COMMENT

* Support binary

* Make parser extend `KtVisitor`

* Call `KtElement#accept()`

* Simplify parsing

* Allow for null initializer

* Added comment regarding space overlap

* Add simple fix for overlapping

* Implement TYPE_REFERENCE

* update fir tree printing

* Add visitClass

* support class modifiers

* move KotlinTreeParser.java to the java folder to pass compile error

* Disable all the unit tests during developing psi-based-parser features and will clean up all the failing tests

* Revert "move KotlinTreeParser.java to the java folder to pass compile error"

This reverts commit e2a8737.

* move KotlinTreeParser.java to the java folder

* Remove `@Disabled` annotations again

* Reset tests to state from main

* Reset tests to state from main

* Reset old parser to main version

* Fix some minor issues

* Fix literal parsing

* Fix failing tests

* Compare type attribution

* Always throw UnsupportedOperationException in the parser

* support CRLF

* adjust visitX methods to be alphabetical sorted

* support if-else statements

* support STRING_TEMPLATE

* support object declaration

* fix some spaces

* fix compile error

* fix space of anonymousObject

* more

* Fix some failing test and disable some recipe tests

* Fix a comment bug

* Get green status

* Use `PsiTreeUtil.getChildOfType()`

* implement visitNamedFunction

* fix compile error

* Implement `visitPackageDirective()`

* support compare kcuPsi and kcuFir LST structure and all types

* seperate type cache

* fix typo

* Add PSI-FIR 1 to N type mapping validation

* polish

* Implement `visitImportDirective()`

* Implement `visitImportDirective()`

* Add missing `UnsupportedOperationException`

* Avoid skip KtDotQualifiedExpression to get a type

* reduce log size for CI build

* add printing log back and udpate

* polish and make all tests pass

* support parsing destructuring

* Pass all test cases

* Add first implementation of `visitPostfixExpression()`

* Start work on `mapModifiers()`

* Polish. Revealed potential NPEs. Removed IntelliJ annotations.
Moved FirFile access to PsiElementAssociations.
Added base for mapping modifiers.

* Added visits.
Added modifier mapping.
Added to find expression type from fir.
Added switch for method invocations and new class in call expression.
Added support for parameterized type names.
Added lots of whitespace parsing issues.
Fixed various types.

* Added J.Empty for new class

* Added base for K.Binary conversions.

* Update Fir Tree printer to support desugaring

* implement visitLiteralStringTemplateEntry

* Fix some space issues

* fix method type for method call identifier

* implement visitReturnExpression

* Added type mapping tests for type overloads.

* Add DesugarVisitor

* polish

* Revised de-sugaring visitor for idempotent prints.

* Added missed changed.

* Revert "Added missed changed."

This reverts commit 2701fe4.

* Revert "Revised de-sugaring visitor for idempotent prints."

This reverts commit 79c30b5.

* Revert "polish"

This reverts commit f8399ce.

* Revert "Add DesugarVisitor"

This reverts commit f56b94b.

* Fix binary parsing

* KBinary toString

* Update

* support property getter()

* Change methodDeclarationType to be generic

* Make `printFirElement()` null-safe

* Better parsing for properties

* Some more minor parser fixes

* Support simple assignments

* Basic support for assignment operations

* Add a few more binary operations

* support getter return type reference

* fix some missing spaces and errors

* pass all tests in tree folder

* support method declaration parameters

* support method declaration with assignment

* support index access

* Support KString

* Correct mapping of package directive

* Map class literals

* Map member references

* Handle `null` receiver with method references

* Support trailing commas in method invocations

* Throw `UnsupportedOperationException` for named arguments

* Throw `UnsupportedOperationException` for vararg parameters

* Support text block

* support visitSimpleNameStringTemplateEntry

* support class type parameter

* Remove an `@ExpectedToFail` annotation

* Add owner to `JavaType.Variable`

* support wildcard type

* support abstract receriver

* Support `KtDoWhileExpression`

* support to keyword

* Support Lambda

* update

* Improve type attribution logic

* fix test case varargArgumentExpression

* Correct some spacing and type attributions

* Fix delimiter for K.String

* Move prefix back to `J.MethodInvocation` again

* Fix prefix again

* Better space handling for lambdas

* Support trailing lambdas

* Correct two tests

* Fix a few more tests

* Improve support for lambdas

* Support nullable types

* fix varargArgumentExpression

* expected to fail for varargArgumentExpression

* support implicit  type parameter

* fix test case methodName

* quick fix

* Small space fix

* Fix a few more corner cases

* Improve `visitQualifiedExpression()`

* support setter

* correct setter/getter order

* support property delegate

* update

* support trailing semicolon

* fix test case provideDelegateExtension

* Support `reified` type parameter

* Basic annotations support

* pass compile

* fix test provideDelegateBinaryType

* fix anonymousObjectWithoutSupertype

* fix anonymousObject

* pass all LiteralTest

* support KtTokens.VARARG_KEYWORD

* support primaryConstructor

* Object declarations are actually mapped to classes

* Object expressions and method parameter defaults

* Trailing comma in primary constructor

* Fix test whitespaceAfter

* support class SuperType

* support explicitInlineConstructor

* fix two test cases

* fix empty object body

* support interface

* support enum class

* support class init block

* support companion

* clean up

* Better type projection support

* support variance

* fix name

* support property leading annotation

* support annotation arguments

* support visitCollectionLiteralExpression

* support paramAnnotation

* fix empty annotation param

* fix bug of public keyword

* fix destructuringVariableDeclaration

* fix reduntant use site annotation

* fix trailing comma

* support binary type %

* support lambdaExpression

* support enum class entries

* support trailing comma of enum class entries

* support trailing semi-colon

* support visitInitializerList

* fix more

* fix enumImplementingInterface

* fix enumWithFunction

* handle both trailing comma and semi-colon works correctly

* Add psi-based-parser switch

* pass build

* turn off type printing to do parse to print idempotent only first

* Fix a space bug

* fix all tests of enumtest

* Clean up imports in tests

* Remove unnecessary `K.Binary.Type` constants

---------

Co-authored-by: Kun Li <[email protected]>
Co-authored-by: traceyyoshima <[email protected]>
  • Loading branch information
@knutwannheden @kunli2 @traceyyoshima
3 people authored Oct 17, 2023
1 parent 2c04664 commit f48f6c0
Show file tree
Hide file tree
Showing 22 changed files with 3,958 additions and 75 deletions.
53 changes: 47 additions & 6 deletions src/main/java/org/openrewrite/kotlin/KotlinParser.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -56,15 +56,12 @@
import org.jetbrains.kotlin.psi.KtFile;
import org.jetbrains.kotlin.resolve.jvm.platform.JvmPlatformAnalyzerServices;
import org.jetbrains.kotlin.utils.PathUtil;
import org.junit.platform.commons.util.StringUtils;
import org.openrewrite.*;
import org.openrewrite.internal.lang.Nullable;
import org.openrewrite.java.JavaParser;
import org.openrewrite.java.internal.JavaTypeCache;
import org.openrewrite.java.marker.JavaSourceSet;
import org.openrewrite.kotlin.internal.CompiledSource;
import org.openrewrite.kotlin.internal.KotlinParserVisitor;
import org.openrewrite.kotlin.internal.KotlinSource;
import org.openrewrite.kotlin.internal.*;
import org.openrewrite.kotlin.tree.K;
import org.openrewrite.style.NamedStyles;
import org.openrewrite.tree.ParseError;
Expand Down Expand Up @@ -93,6 +90,7 @@
import static org.jetbrains.kotlin.config.CommonConfigurationKeys.*;
import static org.jetbrains.kotlin.config.JVMConfigurationKeys.DO_NOT_CLEAR_BINDING_CONTEXT;
import static org.jetbrains.kotlin.incremental.IncrementalFirJvmCompilerRunnerKt.configureBaseRoots;
import static org.junit.jupiter.api.Assertions.assertEquals;

@RequiredArgsConstructor(access = AccessLevel.PRIVATE)
public class KotlinParser implements Parser {
Expand Down Expand Up @@ -165,7 +163,7 @@ public Stream<SourceFile> parseInputs(Iterable<Input> sources, @Nullable Path re
compilerCus.getSources().stream()
.map(kotlinSource -> {
try {
KotlinParserVisitor mappingVisitor = new KotlinParserVisitor(
KotlinParserVisitor firParserVisitor = new KotlinParserVisitor(
kotlinSource,
relativeTo,
styles,
Expand All @@ -175,7 +173,50 @@ public Stream<SourceFile> parseInputs(Iterable<Input> sources, @Nullable Path re
);

assert kotlinSource.getFirFile() != null;
SourceFile kcu = (SourceFile) mappingVisitor.visitFile(kotlinSource.getFirFile(), new InMemoryExecutionContext());
SourceFile kcuFir = (SourceFile) firParserVisitor.visitFile(kotlinSource.getFirFile(), ctx);
SourceFile kcu = kcuFir;

// Turn this flag on locally only to develop psi-based-parser
boolean switchToPsiParser = false;
if (switchToPsiParser) {
// debug purpose only, to be removed
System.out.println(PsiTreePrinter.print(kotlinSource.getInput()));
System.out.println(PsiTreePrinter.print(kotlinSource.getKtFile()));
System.out.println(PsiTreePrinter.print(kotlinSource.getFirFile()));

// PSI based parser
SourceFile kcuPsi;
KotlinTypeMapping typeMapping = new KotlinTypeMapping(new JavaTypeCache(), firSession, kotlinSource.getFirFile().getSymbol());
PsiElementAssociations psiFirMapping = new PsiElementAssociations(typeMapping, kotlinSource.getFirFile());
psiFirMapping.initialize();
KotlinTreeParserVisitor psiParser = new KotlinTreeParserVisitor(kotlinSource, firSession, typeMapping, psiFirMapping, styles, relativeTo, ctx);
try {
kcuPsi = psiParser.parse();
} catch (UnsupportedOperationException ignore) {
throw ignore;
}

if (kcuPsi == null) {
kcuPsi = kcuFir;
System.out.println("=========\n LST and types from FIR-based-parser");
System.out.println(PsiTreePrinter.print(kcuFir));
} else {
// compare kcuPsi and kcuFir LST structure and all types
String treeFir = PsiTreePrinter.print(kcuFir);
String treePsi = PsiTreePrinter.print(kcuPsi);

// Debug purpose only, to be removed
System.out.println("=========\n LST and types from FIR-based-parser");
System.out.println(treeFir);
System.out.println("=========\n LST and types from PSI-based-parser");
System.out.println(treePsi);

assertEquals(treeFir, treePsi);
}

kcu = kcuPsi;
}

parsingListener.parsed(kotlinSource.getInput(), kcu);
return requirePrintEqualsInput(kcu, kotlinSource.getInput(), relativeTo, ctx);
} catch (Throwable t) {
Expand Down
1 change: 1 addition & 0 deletions src/main/java/org/openrewrite/kotlin/internal/KotlinPrinter.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -775,6 +775,7 @@ public J visitLambda(J.Lambda lambda, PrintOutputCapture<P> p) {
p.append('{');
}

visitSpace(lambda.getParameters().getPrefix(), Space.Location.LAMBDA_PARAMETER, p);
visitLambdaParameters(lambda.getParameters(), p);
if (!lambda.getParameters().getParameters().isEmpty()) {
visitSpace(lambda.getArrow(), Space.Location.LAMBDA_ARROW_PREFIX, p);
Expand Down
Loading

0 comments on commit f48f6c0

Please sign in to comment.