Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PSI based parser #309

Merged
merged 229 commits into from
Oct 17, 2023
Merged
Show file tree
Hide file tree
Changes from 228 commits
Commits
Show all changes
229 commits
Select commit Hold shift + click to select a range
93cc385
Add `mapAllElements()` function
knutwannheden Sep 11, 2023
a4b95eb
Add Fir tree printer && Implenment 1st version psi->fir mapping and v…
kunli2 Sep 11, 2023
3548b2a
Tweak and disable some unrelated tests
kunli2 Sep 12, 2023
aaee252
clean up
kunli2 Sep 12, 2023
a88bd47
Refactor code into a new `PsiElementAssociations` class
knutwannheden Sep 12, 2023
0934292
Refactor to not depend on source offsets anymore
knutwannheden Sep 12, 2023
116ee3e
Enable disabled tests again
knutwannheden Sep 12, 2023
e68e6cc
Validate more type attribution
knutwannheden Sep 12, 2023
4947c9b
mark some expected failing tests with reasion
kunli2 Sep 13, 2023
b0b1a0c
update and more ExpectedToFail with reasons
kunli2 Sep 13, 2023
37784dc
Merge branch 'main' into psi-based-parser
kunli2 Sep 14, 2023
75400e8
Add the psi based parser and have 1st test pass
kunli2 Sep 14, 2023
61030b0
Merge branch 'main' into psi-based-parser
kunli2 Sep 14, 2023
daa08bf
add variable type
kunli2 Sep 15, 2023
5050442
Merge branch 'main' into psi-based-parser
kunli2 Sep 15, 2023
5fda653
Handle comments and support nested comments
kunli2 Sep 15, 2023
4573fdb
Make the NestedComments test case more complicated
kunli2 Sep 15, 2023
2ae8d5d
support EOL_COMMENT
kunli2 Sep 15, 2023
4daedb8
Support binary
kunli2 Sep 16, 2023
dbf82b9
Make parser extend `KtVisitor`
knutwannheden Sep 16, 2023
63ca409
Call `KtElement#accept()`
knutwannheden Sep 16, 2023
519aa88
Simplify parsing
knutwannheden Sep 16, 2023
a28aa2d
Allow for null initializer
knutwannheden Sep 16, 2023
d58a88e
Added comment regarding space overlap
knutwannheden Sep 16, 2023
799d379
Add simple fix for overlapping
knutwannheden Sep 16, 2023
44bad50
Implement TYPE_REFERENCE
kunli2 Sep 17, 2023
639ab7f
update fir tree printing
kunli2 Sep 17, 2023
c1f4a87
Add visitClass
kunli2 Sep 17, 2023
a799a43
support class modifiers
kunli2 Sep 17, 2023
e2a8737
move KotlinTreeParser.java to the java folder to pass compile error
kunli2 Sep 17, 2023
7604549
Disable all the unit tests during developing psi-based-parser feature…
kunli2 Sep 17, 2023
f487490
Revert "move KotlinTreeParser.java to the java folder to pass compile…
kunli2 Sep 17, 2023
761e5ae
move KotlinTreeParser.java to the java folder
kunli2 Sep 17, 2023
bd7c32d
Remove `@Disabled` annotations again
knutwannheden Sep 17, 2023
9980031
Reset tests to state from main
knutwannheden Sep 18, 2023
adfa044
Merge remote-tracking branch 'origin/main' into psi-based-parser
knutwannheden Sep 18, 2023
84687bc
Reset tests to state from main
knutwannheden Sep 18, 2023
a3cb933
Reset old parser to main version
knutwannheden Sep 18, 2023
e1aae43
Fix some minor issues
knutwannheden Sep 18, 2023
b9abb58
Fix literal parsing
knutwannheden Sep 18, 2023
d941506
Fix failing tests
knutwannheden Sep 18, 2023
5e6db37
Compare type attribution
knutwannheden Sep 18, 2023
916c635
Always throw UnsupportedOperationException in the parser
kunli2 Sep 18, 2023
48531f5
support CRLF
kunli2 Sep 18, 2023
3f345e3
adjust visitX methods to be alphabetical sorted
kunli2 Sep 18, 2023
7cfadd2
Merge branch 'main' into psi-based-parser
kunli2 Sep 18, 2023
29f6a08
support if-else statements
kunli2 Sep 18, 2023
f942dd1
support STRING_TEMPLATE
kunli2 Sep 18, 2023
30b75ca
support object declaration
kunli2 Sep 19, 2023
2c95733
fix some spaces
kunli2 Sep 19, 2023
c991c72
Merge branch 'main' into psi-based-parser
kunli2 Sep 20, 2023
fe7b21f
fix compile error
kunli2 Sep 20, 2023
465ac44
fix space of anonymousObject
kunli2 Sep 20, 2023
9aa58bf
more
kunli2 Sep 20, 2023
f41a0c0
Fix some failing test and disable some recipe tests
kunli2 Sep 20, 2023
2120107
Fix a comment bug
kunli2 Sep 20, 2023
ac7696b
Get green status
kunli2 Sep 20, 2023
2b19d7f
Use `PsiTreeUtil.getChildOfType()`
knutwannheden Sep 20, 2023
ef1a762
Merge branch 'main' into psi-based-parser
knutwannheden Sep 20, 2023
84363ed
implement visitNamedFunction
kunli2 Sep 20, 2023
03554fe
fix compile error
kunli2 Sep 20, 2023
70a99ba
Implement `visitPackageDirective()`
knutwannheden Sep 20, 2023
435595a
Merge branch 'main' into psi-based-parser
kunli2 Sep 20, 2023
ae25e62
support compare kcuPsi and kcuFir LST structure and all types
kunli2 Sep 20, 2023
180e9a2
seperate type cache
kunli2 Sep 20, 2023
75e063c
fix typo
kunli2 Sep 20, 2023
bed8926
Merge branch 'main' into psi-based-parser
kunli2 Sep 20, 2023
8fcb54b
Add PSI-FIR 1 to N type mapping validation
kunli2 Sep 20, 2023
bcd2977
polish
kunli2 Sep 20, 2023
e42c828
Implement `visitImportDirective()`
knutwannheden Sep 20, 2023
f145490
Implement `visitImportDirective()`
knutwannheden Sep 21, 2023
ec482ad
Add missing `UnsupportedOperationException`
knutwannheden Sep 21, 2023
728ce4c
Avoid skip KtDotQualifiedExpression to get a type
kunli2 Sep 21, 2023
ef7b1d4
reduce log size for CI build
kunli2 Sep 21, 2023
3f56c04
add printing log back and udpate
kunli2 Sep 21, 2023
71e1057
Merge branch 'main' into psi-based-parser
kunli2 Sep 21, 2023
d857e27
polish and make all tests pass
kunli2 Sep 21, 2023
268aad8
support parsing destructuring
kunli2 Sep 22, 2023
40c73b1
Pass all test cases
kunli2 Sep 22, 2023
88e73e0
Add first implementation of `visitPostfixExpression()`
knutwannheden Sep 22, 2023
2875b18
Start work on `mapModifiers()`
knutwannheden Sep 22, 2023
8df0d5a
Polish. Revealed potential NPEs. Removed IntelliJ annotations.
traceyyoshima Sep 22, 2023
e8a88ca
Added visits.
traceyyoshima Sep 23, 2023
85d6e4d
Added J.Empty for new class
traceyyoshima Sep 23, 2023
bb00158
Added base for K.Binary conversions.
traceyyoshima Sep 23, 2023
b981e7d
Update Fir Tree printer to support desugaring
kunli2 Sep 24, 2023
c78675d
implement visitLiteralStringTemplateEntry
kunli2 Sep 25, 2023
34b6366
Fix some space issues
kunli2 Sep 25, 2023
9e2a89f
Merge branch 'main' into psi-based-parser
kunli2 Sep 25, 2023
e4e7377
fix method type for method call identifier
kunli2 Sep 25, 2023
29a0827
implement visitReturnExpression
kunli2 Sep 25, 2023
274c2bf
Added type mapping tests for type overloads.
traceyyoshima Sep 25, 2023
f56b94b
Add DesugarVisitor
kunli2 Sep 25, 2023
f8399ce
polish
kunli2 Sep 25, 2023
79c30b5
Revised de-sugaring visitor for idempotent prints.
traceyyoshima Sep 26, 2023
2701fe4
Added missed changed.
traceyyoshima Sep 26, 2023
2f96a5d
Revert "Added missed changed."
traceyyoshima Sep 28, 2023
694a93a
Revert "Revised de-sugaring visitor for idempotent prints."
traceyyoshima Sep 28, 2023
a19a942
Revert "polish"
traceyyoshima Sep 28, 2023
7799ff1
Revert "Add DesugarVisitor"
traceyyoshima Sep 28, 2023
09a09c6
Merge branch 'main' into psi-based-parser
kunli2 Sep 28, 2023
1705bb7
Merge branch 'main' into psi-based-parser
kunli2 Oct 4, 2023
760baaa
Fix binary parsing
kunli2 Oct 4, 2023
16cc87e
KBinary toString
kunli2 Oct 4, 2023
83d8908
Update
kunli2 Oct 4, 2023
6568486
support property getter()
kunli2 Oct 5, 2023
702b9e2
Change methodDeclarationType to be generic
kunli2 Oct 5, 2023
a49e372
Make `printFirElement()` null-safe
knutwannheden Oct 5, 2023
b4887f7
Better parsing for properties
knutwannheden Oct 5, 2023
fdb38c7
Some more minor parser fixes
knutwannheden Oct 5, 2023
04b6038
Support simple assignments
knutwannheden Oct 5, 2023
9835734
Basic support for assignment operations
knutwannheden Oct 5, 2023
8bddca9
Add a few more binary operations
knutwannheden Oct 5, 2023
024344e
support getter return type reference
kunli2 Oct 5, 2023
70f361f
fix some missing spaces and errors
kunli2 Oct 5, 2023
bef1f08
pass all tests in tree folder
kunli2 Oct 5, 2023
654ee82
support method declaration parameters
kunli2 Oct 5, 2023
cfc2ef1
support method declaration with assignment
kunli2 Oct 5, 2023
ca3557c
support index access
kunli2 Oct 5, 2023
e09752b
Support KString
kunli2 Oct 6, 2023
79e4560
Correct mapping of package directive
knutwannheden Oct 6, 2023
7e83a70
Map class literals
knutwannheden Oct 6, 2023
d8dc427
Map member references
knutwannheden Oct 6, 2023
1585652
Handle `null` receiver with method references
knutwannheden Oct 6, 2023
390d891
Support trailing commas in method invocations
knutwannheden Oct 6, 2023
94f6d45
Throw `UnsupportedOperationException` for named arguments
knutwannheden Oct 6, 2023
9ce25a0
Throw `UnsupportedOperationException` for vararg parameters
knutwannheden Oct 6, 2023
c8f57d0
Merge branch 'main' into psi-based-parser
kunli2 Oct 6, 2023
bafc7b6
Merge branch 'main' into psi-based-parser
kunli2 Oct 6, 2023
01a3059
Support text block
kunli2 Oct 6, 2023
07db50d
support visitSimpleNameStringTemplateEntry
kunli2 Oct 6, 2023
4573d0a
support class type parameter
kunli2 Oct 6, 2023
0ccb7cb
Remove an `@ExpectedToFail` annotation
knutwannheden Oct 8, 2023
d93bbed
Add owner to `JavaType.Variable`
knutwannheden Oct 8, 2023
fc1afa9
support wildcard type
kunli2 Oct 8, 2023
023de9c
support abstract receriver
kunli2 Oct 8, 2023
56cfa8b
Support `KtDoWhileExpression`
knutwannheden Oct 8, 2023
5fd3d96
support to keyword
kunli2 Oct 9, 2023
b532700
Support Lambda
kunli2 Oct 9, 2023
b708220
update
kunli2 Oct 9, 2023
676249a
Merge remote-tracking branch 'origin/main' into psi-based-parser
knutwannheden Oct 9, 2023
f589b9b
Improve type attribution logic
knutwannheden Oct 9, 2023
44d59a6
fix test case varargArgumentExpression
kunli2 Oct 10, 2023
b38e80d
Merge remote-tracking branch 'origin/main' into psi-based-parser
knutwannheden Oct 10, 2023
2ceb05d
Correct some spacing and type attributions
knutwannheden Oct 10, 2023
0d5f025
Fix delimiter for K.String
knutwannheden Oct 10, 2023
7fcd53a
Move prefix back to `J.MethodInvocation` again
knutwannheden Oct 10, 2023
f981ddd
Fix prefix again
knutwannheden Oct 10, 2023
92e6f23
Better space handling for lambdas
knutwannheden Oct 10, 2023
43c2b80
Support trailing lambdas
knutwannheden Oct 10, 2023
0460c32
Correct two tests
knutwannheden Oct 10, 2023
aeaf1d6
Fix a few more tests
knutwannheden Oct 10, 2023
0596925
Improve support for lambdas
knutwannheden Oct 10, 2023
d720f2f
Support nullable types
knutwannheden Oct 10, 2023
01ceca1
fix varargArgumentExpression
kunli2 Oct 10, 2023
4ed38f6
expected to fail for varargArgumentExpression
kunli2 Oct 10, 2023
37dde26
support implicit type parameter
kunli2 Oct 10, 2023
a4f6cf3
fix test case methodName
kunli2 Oct 10, 2023
e4b8b9b
quick fix
kunli2 Oct 10, 2023
d508e48
Small space fix
knutwannheden Oct 10, 2023
2992a2f
Fix a few more corner cases
knutwannheden Oct 10, 2023
039572f
Improve `visitQualifiedExpression()`
knutwannheden Oct 10, 2023
e11d2c7
Merge branch 'main' into psi-based-parser
kunli2 Oct 10, 2023
09f65b1
support setter
kunli2 Oct 10, 2023
33a1cbf
correct setter/getter order
kunli2 Oct 10, 2023
c0a2dbd
support property delegate
kunli2 Oct 11, 2023
b0b7ce0
update
kunli2 Oct 11, 2023
f5e8775
support trailing semicolon
kunli2 Oct 11, 2023
cd321a3
fix test case provideDelegateExtension
kunli2 Oct 11, 2023
195d6e1
Support `reified` type parameter
knutwannheden Oct 11, 2023
b210cd0
Basic annotations support
knutwannheden Oct 11, 2023
dfefe3a
Merge branch 'main' into psi-based-parser
kunli2 Oct 11, 2023
e479ca8
pass compile
kunli2 Oct 11, 2023
670704b
fix test provideDelegateBinaryType
kunli2 Oct 11, 2023
2766dc6
fix anonymousObjectWithoutSupertype
kunli2 Oct 11, 2023
5b7bbf3
fix anonymousObject
kunli2 Oct 11, 2023
6a57f41
pass all LiteralTest
kunli2 Oct 11, 2023
506f158
support KtTokens.VARARG_KEYWORD
kunli2 Oct 12, 2023
c9782f6
support primaryConstructor
kunli2 Oct 12, 2023
a7c1997
Object declarations are actually mapped to classes
knutwannheden Oct 12, 2023
3c2846a
Object expressions and method parameter defaults
knutwannheden Oct 12, 2023
ab08210
Trailing comma in primary constructor
knutwannheden Oct 12, 2023
1afd27c
Merge branch 'main' into psi-based-parser
kunli2 Oct 12, 2023
9d0d92a
Fix test whitespaceAfter
kunli2 Oct 12, 2023
602e9b1
support class SuperType
kunli2 Oct 12, 2023
44e9360
support explicitInlineConstructor
kunli2 Oct 12, 2023
79300e4
Merge branch 'main' into psi-based-parser
kunli2 Oct 12, 2023
458eb42
fix two test cases
kunli2 Oct 12, 2023
abc08c6
fix empty object body
kunli2 Oct 12, 2023
f2cf10e
support interface
kunli2 Oct 12, 2023
fbf7c15
support enum class
kunli2 Oct 12, 2023
0fa1663
support class init block
kunli2 Oct 12, 2023
16f3fdb
support companion
kunli2 Oct 13, 2023
8b664e9
Merge branch 'main' into psi-based-parser
kunli2 Oct 13, 2023
20aa322
clean up
kunli2 Oct 13, 2023
08b5cb8
Better type projection support
knutwannheden Oct 13, 2023
6917453
Merge branch 'main' into psi-based-parser
kunli2 Oct 13, 2023
064dd24
support variance
kunli2 Oct 13, 2023
aef3af0
Merge branch 'main' into psi-based-parser
kunli2 Oct 13, 2023
b9c0335
fix name
kunli2 Oct 13, 2023
e29d41b
Merge branch 'main' into psi-based-parser
kunli2 Oct 13, 2023
c648e39
support property leading annotation
kunli2 Oct 13, 2023
d3f180b
support annotation arguments
kunli2 Oct 13, 2023
0e259d1
support visitCollectionLiteralExpression
kunli2 Oct 14, 2023
4bff50a
support paramAnnotation
kunli2 Oct 15, 2023
0e8a472
fix empty annotation param
kunli2 Oct 15, 2023
b125281
fix bug of public keyword
kunli2 Oct 15, 2023
bcc0216
fix destructuringVariableDeclaration
kunli2 Oct 15, 2023
3a5834b
Merge branch 'main' into psi-based-parser
kunli2 Oct 16, 2023
19d76bc
fix reduntant use site annotation
kunli2 Oct 16, 2023
805b045
fix trailing comma
kunli2 Oct 16, 2023
8373da4
support binary type %
kunli2 Oct 16, 2023
71cafa6
support lambdaExpression
kunli2 Oct 16, 2023
700aceb
support enum class entries
kunli2 Oct 16, 2023
6973c36
support trailing comma of enum class entries
kunli2 Oct 16, 2023
4f15405
support trailing semi-colon
kunli2 Oct 16, 2023
57f97b6
support visitInitializerList
kunli2 Oct 16, 2023
6afe6ae
fix more
kunli2 Oct 16, 2023
38ca659
fix enumImplementingInterface
kunli2 Oct 16, 2023
31acea2
fix enumWithFunction
kunli2 Oct 16, 2023
69db658
handle both trailing comma and semi-colon works correctly
kunli2 Oct 16, 2023
43e4e0c
Merge branch 'main' into psi-based-parser
kunli2 Oct 16, 2023
eb00968
Add psi-based-parser switch
kunli2 Oct 17, 2023
aa6522c
pass build
kunli2 Oct 17, 2023
f742268
turn off type printing to do parse to print idempotent only first
kunli2 Oct 17, 2023
6d2c4d0
Fix a space bug
kunli2 Oct 17, 2023
5e6485a
fix all tests of enumtest
kunli2 Oct 17, 2023
65a753c
Clean up imports in tests
knutwannheden Oct 17, 2023
175f4be
Remove unnecessary `K.Binary.Type` constants
knutwannheden Oct 17, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 47 additions & 6 deletions src/main/java/org/openrewrite/kotlin/KotlinParser.java
Original file line number Diff line number Diff line change
Expand Up @@ -56,15 +56,12 @@
import org.jetbrains.kotlin.psi.KtFile;
import org.jetbrains.kotlin.resolve.jvm.platform.JvmPlatformAnalyzerServices;
import org.jetbrains.kotlin.utils.PathUtil;
import org.junit.platform.commons.util.StringUtils;
import org.openrewrite.*;
import org.openrewrite.internal.lang.Nullable;
import org.openrewrite.java.JavaParser;
import org.openrewrite.java.internal.JavaTypeCache;
import org.openrewrite.java.marker.JavaSourceSet;
import org.openrewrite.kotlin.internal.CompiledSource;
import org.openrewrite.kotlin.internal.KotlinParserVisitor;
import org.openrewrite.kotlin.internal.KotlinSource;
import org.openrewrite.kotlin.internal.*;
import org.openrewrite.kotlin.tree.K;
import org.openrewrite.style.NamedStyles;
import org.openrewrite.tree.ParseError;
Expand Down Expand Up @@ -93,6 +90,7 @@
import static org.jetbrains.kotlin.config.CommonConfigurationKeys.*;
import static org.jetbrains.kotlin.config.JVMConfigurationKeys.DO_NOT_CLEAR_BINDING_CONTEXT;
import static org.jetbrains.kotlin.incremental.IncrementalFirJvmCompilerRunnerKt.configureBaseRoots;
import static org.junit.jupiter.api.Assertions.assertEquals;

@RequiredArgsConstructor(access = AccessLevel.PRIVATE)
public class KotlinParser implements Parser {
Expand Down Expand Up @@ -165,7 +163,7 @@ public Stream<SourceFile> parseInputs(Iterable<Input> sources, @Nullable Path re
compilerCus.getSources().stream()
.map(kotlinSource -> {
try {
KotlinParserVisitor mappingVisitor = new KotlinParserVisitor(
KotlinParserVisitor firParserVisitor = new KotlinParserVisitor(
kotlinSource,
relativeTo,
styles,
Expand All @@ -175,7 +173,50 @@ public Stream<SourceFile> parseInputs(Iterable<Input> sources, @Nullable Path re
);

assert kotlinSource.getFirFile() != null;
SourceFile kcu = (SourceFile) mappingVisitor.visitFile(kotlinSource.getFirFile(), new InMemoryExecutionContext());
SourceFile kcuFir = (SourceFile) firParserVisitor.visitFile(kotlinSource.getFirFile(), ctx);
SourceFile kcu = kcuFir;

// Turn this flag on locally only to develop psi-based-parser
boolean switchToPsiParser = false;
if (switchToPsiParser) {
// debug purpose only, to be removed
System.out.println(PsiTreePrinter.print(kotlinSource.getInput()));
System.out.println(PsiTreePrinter.print(kotlinSource.getKtFile()));
System.out.println(PsiTreePrinter.print(kotlinSource.getFirFile()));

// PSI based parser
SourceFile kcuPsi;
KotlinTypeMapping typeMapping = new KotlinTypeMapping(new JavaTypeCache(), firSession, kotlinSource.getFirFile().getSymbol());
PsiElementAssociations psiFirMapping = new PsiElementAssociations(typeMapping, kotlinSource.getFirFile());
psiFirMapping.initialize();
KotlinTreeParserVisitor psiParser = new KotlinTreeParserVisitor(kotlinSource, firSession, typeMapping, psiFirMapping, styles, relativeTo, ctx);
try {
kcuPsi = psiParser.parse();
} catch (UnsupportedOperationException ignore) {
throw ignore;
}

if (kcuPsi == null) {
kcuPsi = kcuFir;
System.out.println("=========\n LST and types from FIR-based-parser");
System.out.println(PsiTreePrinter.print(kcuFir));
} else {
// compare kcuPsi and kcuFir LST structure and all types
String treeFir = PsiTreePrinter.print(kcuFir);
String treePsi = PsiTreePrinter.print(kcuPsi);

// Debug purpose only, to be removed
System.out.println("=========\n LST and types from FIR-based-parser");
System.out.println(treeFir);
System.out.println("=========\n LST and types from PSI-based-parser");
System.out.println(treePsi);

assertEquals(treeFir, treePsi);
}

kcu = kcuPsi;
}

parsingListener.parsed(kotlinSource.getInput(), kcu);
return requirePrintEqualsInput(kcu, kotlinSource.getInput(), relativeTo, ctx);
} catch (Throwable t) {
Expand Down
13 changes: 13 additions & 0 deletions src/main/java/org/openrewrite/kotlin/internal/KotlinPrinter.java
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,18 @@ public J visitBinary(K.Binary binary, PrintOutputCapture<P> p) {
case RangeUntil:
keyword = "..<";
break;
case Plus:
keyword = "+";
break;
case Minus:
keyword = "-";
break;
case Mul:
keyword = "*";
break;
case Div:
keyword = "/";
break;
}
visit(binary.getLeft(), p);
visitSpace(binary.getPadding().getOperator().getBefore(), KSpace.Location.BINARY_OPERATOR, p);
Expand Down Expand Up @@ -775,6 +787,7 @@ public J visitLambda(J.Lambda lambda, PrintOutputCapture<P> p) {
p.append('{');
}

visitSpace(lambda.getParameters().getPrefix(), Space.Location.LAMBDA_PARAMETER, p);
visitLambdaParameters(lambda.getParameters(), p);
if (!lambda.getParameters().getParameters().isEmpty()) {
visitSpace(lambda.getArrow(), Space.Location.LAMBDA_ARROW_PREFIX, p);
Expand Down
Loading