Copying or visiting the same tree multiple times?

[juli1]

I have a program that does multiple analyses on the same AST, one analysis per thread. The analysis is done with a Visitor. For now, each thread parses the program input, transforms it as an AST, and runs its visitor. However, this is very time-consuming to parse the same program multiple times.

I tried to parse the program once and have the threads to use the visitors, but it does not seem multiple visitors can share the same tree (especially concurrently).

One alternative would be to parse the program, build the AST and make a copy of the AST that is later analyzed.

Any recommendations on how to proceed?

1. Is there any way to safely process the same AST by different threads?
2. Is there a way to make a copy (e.g. deep copy) of an AST?

[ericvergnaud]

This seems weird since in theory the visitor does not touch the AST. Which language are you targeting ?Envoyé de mon iPhoneLe 18 oct. 2022 à 08:44, Julien ***@***.***> a écrit : I have a program that does multiple analyses on the same AST, one analysis per thread. The analysis is done with a Visitor. For now, each thread parses the program input, transforms it as an AST, and runs its visitor. However, this is very time-consuming to parse the same program multiple times. I tried to parse the program once and have the threads to use the visitors, but it does not seem multiple visitors can share the same tree (especially concurrently). One alternative would be to parse the program, build the AST and make a copy of the AST that is later analyzed. Any recommendations on how to proceed? Is there any way to safely process the same AST by different threads? Is there a way to make a copy (e.g. deep copy) of an AST? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[juli1]

Thanks for your reply! I am targeting Python.

This is how I parse the code

PythonLexer lexer = new PythonLexer(CharStreams.fromString(code));
CommonTokenStream tokens = new CommonTokenStream(lexer);
PythonParser parser = new PythonParser(tokens);
parser.setBuildParseTree(true);

Then, in each thread, I build my visitor

MyVisitor visitor = new MyVisitor();
visitor.visit(parser.root());

If I have only one thread, everything runs fine. If I have two threads, results are then different.

[ericvergnaud]

Can you share your visitor's code ? It could be touching the parse tree without knowing...

[juli1]

Unfortunately, I cannot share everything.

The visitor executes rules for each node.

So we have functions in the visitor that analyze each node like the code below. The transformStmtToTryStatement converts the Antlr AST into our own representation.

    @Override
    public List<Violation> visitTry_stmt(PythonParser.Try_stmtContext ctx) {
       Optional<TryStatement> tryStatementOptional = transformStmtToTryStatement(ctx, root);
        tryStatementOptional.ifPresent(this::executeRule);
        

        return visitChildren(ctx);
    }

[ericvergnaud]

Actually you are not targeting Python but Java i.e. your parser parses Python code but runs in Java…I suspect you’re changing the parse tree in the piece of code that you cannot share…Envoyé de mon iPhoneLe 19 oct. 2022 à 01:11, Julien ***@***.***> a écrit : Unfortunately, I cannot share everything. The visitor executes rules for each node. So we have functions in the visitor that analyze each node like the code below. The transformStmtToTryStatement converts the Antlr AST into our own representation. @OverRide public List<Violation> visitTry_stmt(PythonParser.Try_stmtContext ctx) { Optional<TryStatement> tryStatementOptional = transformStmtToTryStatement(ctx, root); tryStatementOptional.ifPresent(this::executeRule); return visitChildren(ctx); } —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>