Code too large

[ssteinhauser]

Hi,

I am currently struggling with the Code too large problem while I am trying to implement the IMPORT and EXPORT statements of Exasol (https://docs.exasol.com/db/latest/sql/import.htm / https://docs.exasol.com/db/latest/sql/export.htm).
You can see the current progress of my implementation here: https://github.com/ssteinhauser/JSqlParser/tree/feature/exasol-import-export

Did anyone already experience this problem and can provide a solution for that?

Thank you in advance!

[manticore-projects]

Greetings.

I was worried that we will hit this at a time. Its a limitation of JavaCC when the Grammar is too complex and the generated Parser becomes to large (beyond the 64kb mark).
As far as I do understand it, there is not immediate solution to it. You can only try to split your production in smaller chunks.

In general, its a JavaCC issue and should be discussed there.
The only prompt ideas I have are:

1. split Productions into smaller chunks
2. split off DDL from DML and Queries
3. switch to CongoCC which appears to be the superior, but toxic technology

[manticore-projects]

I opened a case at JavaCC, see javacc/javacc#297

[ssteinhauser]

Hi @manticore-projects ,
thank you for the quick reply.
That's bad to hear, that there is not actual solution for that. However, thank you for opening the issue at JavaCC.

The error occurs in the TokenManager class, so I guess splitting the productions further won't fix the issue. From my understanding (possibly I am wrong), it is due to the number of tokens.

So I guess my implementation is being blocked by that problem and we need to wait for a solution or workaround provided at JavaCC.

[manticore-projects]

@ssteinhauser: please give it a few days to collect feedback and then we will look for an acceptable way forward.

[manticore-projects]

@ssteinhauser: I have slept over this and cleared my thoughts and without any promise, the most likely scenario now looks like this

1. I am aiming for a final JSQLParser 5.2 release (based on JavaCC) because I need the new PipeSQL parser released

2. after that, I will likely look for a migration to CongoCC although I have many concerns around that:

   a) I will need to overhaul the complete Grammar

   b) I will need to rewrite parts of the tool chain (drop Maven, write a Gradle module, rewrite the Rail Road builder, rewrite the Reserved Keywords logic)

   c) there are some "peoples issues" to overcome

So it looks like an aim for a JSQLParser 6 release with a lot of work to-do.

@wumpz: Your opinion and endorsement will be needed for this one to work.

[ssteinhauser]

Hi @manticore-projects ,
alright, thank you for sharing your thoughts.

Regarding the overhaul of the complete Grammar, you can give the official syntax converter a try.

Please also let me know, if I can or should already provide a PR with my changes, so in case you can probably also convert that syntax to CongoCC so I don't have to do the same thing again on my end for my newly implemented syntax.

[MarcMazas]

Hi.
We advice you to take some time checking that JavaCC 8 solves your issue without functional & performance regressions.
I cloned https://github.com/ssteinhauser/JSqlParser/tree/feature/exasol-import-export, modified pom.xml:

...
        <!-- needed for parsing the Keywords via JTree in ParserKeywordsUtils -->
        <dependency>
<!--            <groupId>net.java.dev.javacc</groupId>-->
<!--            <artifactId>javacc</artifactId>-->
<!--            <version>[7.0.13,)</version>-->
            <groupId>org.javacc.generator</groupId>
            <artifactId>java</artifactId>
            <version>8.0.1</version>
<!--            <version>8.1.0-SNAPSHOT</version>-->
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.javacc</groupId>
            <artifactId>core</artifactId>
            <version>8.0.1</version>
<!--            <version>8.1.0-SNAPSHOT</version>-->
            <scope>test</scope>
        </dependency>

...
            <plugin>
                <groupId>org.javacc.plugin</groupId>
                <artifactId>javacc-maven-plugin</artifactId>
                <version>3.0.3</version>
                <executions>
                    <execution>
                        <id>javacc</id>
                        <phase>generate-sources</phase>
                        <goals>
                            <goal>jjtree-javacc</goal>
                        </goals>
                        <configuration>
                            <codeGenerator>java</codeGenerator>
<!--                            <grammarEncoding>UTF-8</grammarEncoding>-->
<!--                            <isStatic>false</isStatic>-->
<!--                            <jdkVersion>1.8</jdkVersion>-->
                        </configuration>
                    </execution>
                </executions>
                <dependencies>
<!--                    <dependency>-->
<!--                        <groupId>net.java.dev.javacc</groupId>-->
<!--                        <artifactId>javacc</artifactId>-->
<!--                        <version>[7.0.13,)</version>-->
<!--                    </dependency>-->
                    <dependency>
                        <groupId>org.javacc.generator</groupId>
                        <artifactId>java</artifactId>
                        <version>8.0.1</version>
<!--                        <version>8.1.0-SNAPSHOT</version>-->
                    </dependency>
                    <dependency>
                        <groupId>org.javacc</groupId>
                        <artifactId>core</artifactId>
                        <version>8.0.1</version>
<!--                        <version>8.1.0-SNAPSHOT</version>-->
                    </dependency>
                </dependencies>
            </plugin>
...

You'll have to change yous SimpleNode references to Node (and other things in your tests, on which I did not investigate).

Tell me how things evolve.

[manticore-projects]

@MarcMazas: Thank you a lot! I definitely will give this a try soonest and appreciate your help and support. I will return with findings and hopefully a success story.

Cheers and best
Andreas

[MarcMazas]

Hi By the way: the version you�ll find on Maven Central is 8.0.1; I�m on a 8.1.0 version, currently not released; it has a lot of �laundry� changes but with normally no impact; main benefits for users should be prettier generated code and better debug traces (lookahead, token manager, parser). Will it be possible for you to check without too much effort that JSqlParser works fine with 8.0.1, and then spend more time checking it works with 8.1.0 (for example on performance, traces�)? TIA Regards Marc De : manticore-projects ***@***.***> Envoyé : mardi 11 février 2025 12:16 À : JSQLParser/JSqlParser ***@***.***> Cc : Marc MAZAS ***@***.***>; Mention ***@***.***> Objet : Re: [JSQLParser/JSqlParser] Code too large (Discussion #2161) @MarcMazas <https://github.com/MarcMazas> : Thank you a lot! I definitely will give this a try soonest and appreciate your help and support. I will return with findings and hopefully a success story. Cheers and best Andreas � Reply to this email directly, view it on GitHub <#2161 (comment) -12139319> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFQZRCZ7KCWMSSRUQHJOKUL2P HLVHAVCNFSM6AAAAABW2GOQPOVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTE MJTHEZTCOI> . You are receiving this because you were mentioned. <https://github.com/notifications/beacon/AFQZRCYXTCVE7Z6DJ6MD5GL2PHLVHA5CNFS M6AAAAABW2GOQPOWGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3T UL5UWJTQAXE5TO.gif> Message ID: ***@***.*** ***@***.*** .com> >

[manticore-projects]

Sure, I will try this out too -- but only next week.
Thank you again for helping!

[MarcMazas]

Hi,

I am currently struggling with the Code too large problem while I am trying to implement the IMPORT and EXPORT statements of Exasol (https://docs.exasol.com/db/latest/sql/import.htm / https://docs.exasol.com/db/latest/sql/export.htm). You can see the current progress of my implementation here: https://github.com/ssteinhauser/JSqlParser/tree/feature/exasol-import-export

Did anyone already experience this problem and can provide a solution for that?

Thank you in advance!

About the "Code too large" issue: from what I've seen, you had 2 errrors:

• one for the static part (which is the one discussed in the article you mentioned)
• one for a generated method (and the author of the article does not advertise he guarantees that the same problem will not happen with his parser).

[manticore-projects]

@ssteinhauser: There is some good news. After running into Code too large myself I have been finally forced to take this seriously and I managed to start a JavaCC-8 migration.

1. it solves indeed the code too large problem from too many tokens
2. it works in general
3. but it fails on two instances of token manipulation (T-SQL squared brackets and advanced text escaping)

Here is the new branch: https://github.com/JSQLParser/JSqlParser/tree/javacc8

You can continue your implementation on that.