Indexing single Lingua Franca projects #181

vinzbarbuto · 2024-10-10T01:17:36Z

This PR introduces a new indexing method for the TreeView. If the data provider detects local libraries, lingo packages, or source files within a single LF project (i.e., a root directory containing a ./src folder), the TreeView will be displayed as follows:

If an LF project is not found, the following message will be displayed:

Note: For the LF project to appear in the TreeView, it must adhere to the following structure:

├── root
...
│   ├── src/ # Contains LF source files
│   │  ├── SourceFile.lf
│   │  ├── lib/ # Directory for storing reusable reactors
│   │  │   ├── Input.lf # Ex: reactor capturing external inputs (e.g., Microphone, Camera)
│   ├── build/   # directory containing lingo packages
│   │  ├── lfc_include/ # Directory for storing reusable reactors
│   │  │   ├── lingo_downloaded_library/
...

At least one of the following must be present in the structure for the TreeView to be displayed: source files, library files, or lingo files.

vinzbarbuto · 2024-10-10T17:38:03Z

After the latest commit, if no LF project is found, the following message will be displayed:

Open Lingua Franca Project seems like a more suitable option than Refresh

vinzbarbuto · 2024-10-10T18:05:51Z

The question is, what if we have the following structure?

├── main_lf_root/
...
│   ├── src/ # Contains LF source files
│   │  ├── lf_project_1/ 
│   │  │   ├── src/ # Contains LF source files of lf_project_1
│   │  │   │  ├──SourceFile.lf
│   │  │   │  ├── lib/ # Directory for storing reusable reactors of lf_project_1
│   │  │   │  │   ├── Input.lf # Ex: reactor capturing external inputs (e.g., Microphone, Camera)
│   │  │   ├── build/   # directory containing lingo packages of lf_project_1
│   │  │   │  ├── lfc_include/ # Directory for storing reusable reactors
│   │  │   │  │   ├── lingo_downloaded_library/
│   │  ├── lf_project_2/  # Same structure as lf_project_1
...

Should we consider this an LF project because it has a src/ folder? If so, we should define what qualifies as an LF project to proceed with the implementation accordingly. What do you suggest? @lhstrh @edwardalee @tanneberger

lhstrh · 2024-10-10T18:58:09Z

The question is, what if we have the following structure?

├── main_lf_root/
...
│   ├── src/ # Contains LF source files
│   │  ├── lf_project_1/ 
│   │  │   ├── src/ # Contains LF source files of lf_project_1
│   │  │   │  ├──SourceFile.lf
│   │  │   │  ├── lib/ # Directory for storing reusable reactors of lf_project_1
│   │  │   │  │   ├── Input.lf # Ex: reactor capturing external inputs (e.g., Microphone, Camera)
│   │  │   ├── build/   # directory containing lingo packages of lf_project_1
│   │  │   │  ├── lfc_include/ # Directory for storing reusable reactors
│   │  │   │  │   ├── lingo_downloaded_library/
│   │  ├── lf_project_2/  # Same structure as lf_project_1
...

Should we consider this an LF project because it has a src/ folder? If so, we should define what qualifies as an LF project to proceed with the implementation accordingly. What do you suggest? @lhstrh @edwardalee @tanneberger

AFAIK, what we agreed on is that the source files of a package should be in ./src. If you create subdirectories, that's fine (even if you call them src, although this would be confusing), but that doesn't turn your package into a container that hosts multiple "projects" (whatever we happen to mean with that). I don't see any reason to accommodate this usage pattern.

If you open VS Code in main_lf_root and open one of the .lf files and build them, the build artifacts should be produced relative to the root, not adjacent to whatever first src directory you might find in the directory tree that the .lf file is in. TBH, I'm not sure how lfc handles this currently.

edwardalee · 2024-10-10T19:07:13Z

If you open VS Code in main_lf_root and open one of the .lf files and build them, the build artifacts should be produced relative to the root, not adjacent to whatever first src directory you might find in the directory tree that the .lf file is in. TBH, I'm not sure how lfc handles this currently.

lfc knows nothing about where you opened VS Code, so it can't possibly do this. I think what it does is find the first src directory above the .lf file and puts the build artifacts there. The only other option I can see would be for lfc to use the current working directory, but this would make the behavior quite sensitive to where you run lfc from. Anyway, changing what lfc does should be out-of-scope for this effort.

With the above file structure, the subdirectories with src directories will be treated by lfc as separate "project" roots (meaning that's where the src-gen and bin get put).

So the @vinzbarbuto 's question is valid: If you open VS Code in directory main_lf_root, what should it do? If we say "we don't accommodate this use case", then I suppose it would need to search the directory for embedded src directories and then what. Issue an error message? Disable the extension?

lhstrh · 2024-10-10T19:09:18Z

Out of scope? Don't we want a consistent approach to what our build tools do?

edwardalee · 2024-10-10T19:32:01Z

Out of scope? Don't we want a consistent approach to what our build tools do?

What I meant is that this design should take what lfc does as a given. It defines what an LF project is. So yes, we should be consistent, and the subdirectories with src should be considered separate projects, and the enclosing directory with src should not be a project unless it also contains .lf files.

This adds a lot of complexity to this tool. Do we really need this?

An alternative would be to change what lfc does, but to what? Will we have to give up running lfc on the command line and only use it within the IDE? Who should take on this project?

vinzbarbuto · 2024-10-10T20:57:32Z

With the implementation proposed in this PR, an LF project is defined as a project with the following structure:

├── root
...
│   ├── src/ # Contains LF source files
│   │  ├── SourceFile.lf # Can also be in a subfolder, e.g. source1/SourceFile.lf
│   │  ├── lib/ # Directory for storing reusable reactors
│   │  │   ├── Input.lf # Example: reactor capturing external inputs (e.g., Microphone, Camera)
│   ├── build/   # Directory containing lingo packages
│   │  ├── lfc_include/ # Directory for storing reusable reactors
│   │  │   ├── lingo_downloaded_library/
...

If a project does not follow the structure above, as the one showed in my previous comment, it will not be considered an LF project. In this case, if an LF project with the incorrect structure is detected, a message will prompt the user to refer to the LF documentation on how to correctly structure the project. The user can then click the button Open Lingua Franca Project to open a properly structured LF project.

After the latest commit, if no LF project is found, the following message will be displayed:

Open Lingua Franca Project seems like a more suitable option than Refresh

lhstrh · 2024-10-10T21:05:54Z

What I meant is that this design should take what lfc does as a given. It defines what an LF project is. So yes, we should be consistent, and the subdirectories with src should be considered separate projects, and the enclosing directory with src should not be a project unless it also contains .lf files.

I thought that we were implementing what's described here, and there is no mention of nested package structures in that discussion. Why do we need to support it at all?

As per my earlier comment, I don't actually see a problem with the presented scenario.

✔️ there is a ./src directory (it is a valid project)
✔️ building ./src/lf_project_1/src/SourceFile.lf would produce a binary in ./bin/SourceFile

If that's not what you want, then either open the IDE in a different location or reorganize your files.

lhstrh · 2024-10-10T21:08:22Z

If a project does not follow the structure above, as the one showed in my previous comment, it will not be considered an LF project. In this case, if an LF project with the incorrect structure is detected, a message will prompt the user to refer to the LF documentation on how to correctly structure the project. The user can then click the button Open Lingua Franca Project to open a properly structured LF project.

This sounds fine, but my only question would be: how do you validate the structure of the project layout? I would keep this as simple as possible.

lhstrh · 2024-10-10T21:11:43Z

src/lfview/lf-data-provider.ts

@@ -577,7 +574,7 @@ export class LFDataProvider implements vscode.TreeDataProvider<LFDataProviderNod
     */
    buildRoot(uri: string, type: LFDataProviderNodeType | null): LFDataProviderNode {
        const splittedUri = uri.split('/');
-            const srcIdx = splittedUri.indexOf(!type || type == LFDataProviderNodeType.LIBRARY ? 'build' : 'src');
+            const srcIdx = splittedUri.lastIndexOf(!type || type == LFDataProviderNodeType.LIBRARY ? 'build' : 'src');


Why this change? This seems to relate to our discussion. The question is whether we should find the file relative to the root of the workspace, or fine the root of the package by traversing up from the file. In lfc, we have no workspace root, so we do the latter. I think that for an IDE it makes more sense to do the former.

I use the last index because the uri passed to the buildRoot function is an absolute file path. For example, if my workspace is opened for a project named project with the following structure:

├── project ... │ ├── src/ # Contains LF source files │ │ ├── SourceFile.lf # Can also be in a subfolder, e.g. source1/SourceFile.lf │ │ ├── lib/ # Directory for storing reusable reactors │ │ │ ├── Input.lf # Example: reactor capturing external inputs (e.g., Microphone, Camera) │ ├── build/ # Directory containing lingo packages │ │ ├── lfc_include/ # Directory for storing reusable reactors │ │ │ ├── lingo_downloaded_library/ ...

Let's assume the project in your file system has an src folder in its absolute path. The uri passed to the function could be something like /root/src/project/src/SourceFile.lf. If you use indexOf, it will find the index of the first src folder, and the label shown in the tree view will be root instead of project. However, we want to locate the closest src folder to SourceFile.lf and use that.

This makes sense depending on whether we want to let the open file dictate the behavior of the editor or whether we want the workspace to do that. Here, you're switching from the latter to the former. We have to make sure this is what we want.

It also leaves unanswered the question of what to do when no file is opened in the editor. Do we look down in case no file is opened and up in case there is an open file?

I think the answer to this is here as well

edwardalee · 2024-10-10T21:40:59Z

What I meant is that this design should take what lfc does as a given. It defines what an LF project is. So yes, we should be consistent, and the subdirectories with src should be considered separate projects, and the enclosing directory with src should not be a project unless it also contains .lf files.

I thought that we were implementing what's described here, and there is no mention of nested package structures in that discussion. Why do we need to support it at all?

As per my earlier comment, I don't actually see a problem with the presented scenario.

✔️ there is a ./src directory (it is a valid project)

✔️ building ./src/lf_project_1/src/SourceFile.lf would produce a binary in ./bin/SourceFile

If that's not what you want, then either open the IDE in a different location or reorganize your files.

This isn't what lfc currently does. I just tested it. If I run

lfc ./src/lf_project_1/src/SourceFile.lf

it produces the binary in ./src/lf_project_1/bin/SourceFile not ./bin/SourceFile. This design dates back several years, and I recall quite a bit of discussion back then. Do we need to reopen that discussion and find another design? I'm not sure what the alternative design would be.

lhstrh · 2024-10-10T21:48:16Z

This isn't what lfc currently does. I just tested it. If I run
lfc ./src/lf_project_1/src/SourceFile.lf
it produces the binary in ./src/lf_project_1/bin/SourceFile not ./bin/SourceFile. This design dates back several years, and I recall quite a bit of discussion back then. Do we need to reopen that discussion and find another design? I'm not sure what the alternative design would be.

As you pointed out yourself, this is because lfc is a command line tool that has no notion of a workspace. Moreover, this design predates the notion of a package, because we had no package manager back then. While developing lingo, the idea has always been that it in ways would obviate direct use of lfc in a similar way as cargo obviates the direct use of rustc.

In a package created by lingo, there will always be a Lingo.toml at the root of the package (alongside the src directory as the location to place sources). I think the goal here should be to make sure that the VS Code extension and lingo do similar things.

vinzbarbuto · 2024-10-10T21:49:30Z

This sounds fine, but my only question would be: how do you validate the structure of the project layout? I would keep this as simple as possible.

To be consistent with lfc, an LF project is considered to have a correct structure if at least one of the following conditions is met:

It contains *.lf files in a folder named src/. In this case, the src/ folder is identified, and the directory above it will be considered the project root. These files will populate the Source Files section of the tree view.
It contains *.lf files in a folder named src/lib/. Similarly, the src/ folder is identified, and the directory above it will be considered the project root. These files will populate the Local Libraries section of the tree view.
It contains *.lf files in a folder named build/lfc_include/src/lib/. Here, the build/ folder is identified, and the directory above it will be considered the project root. These files will populate the Lingo Packages section of the tree view.

If at least one of these conditions is satisfied, the tree view will be displayed. Otherwise, the message will be shown.

lhstrh · 2024-10-11T03:44:52Z

This sounds fine, but my only question would be: how do you validate the structure of the project layout? I would keep this as simple as possible.

To be consistent with lfc, an LF project is considered to have a correct structure if at least one of the following conditions is met: (...)

If at least one of these conditions is satisfied, the tree view will be displayed. Otherwise, the message will be shown.

Yes, this makes sense, but are you reasoning up from the location of the current file or down from the current workspace? I think the code suggests the former, but I want to make sure I understand you correctly.

If my understanding is correct -- going back to your example -- if you open VS Code in main_lf_root, the tree view is not shown. Then, if you navigate to ./src/lf_project_1/ in the file explorer and open SourceFile.lf, the package explorer will show whatever if will find in lf_project_1. Then, if you open./src/lf_project_2/SourceFile.lf, it will show what's in lf_project_2, and when you switch back to ./src/lf_project_1/SourceFile.lf again, the package explorer will show what's in lf_project_1 again, etc.

lhstrh · 2024-10-11T04:08:19Z

package.json

+        "viewsWelcome": [
+            {
+              "view": "lf-lang-projects",
+              "contents": "No Lingua Franca project found. [Learn more](https://www.lf-lang.org/docs/) about setting up a Lingua Franca project structure.\n[Open Lingua Franca Project](command:linguafranca.openFolder)"


Should we point to a specific page?

Yes, that’s the idea. Do you know which specific page?

https://www.lf-lang.org/docs/tools/code-extension/#usage but this page it rather incomplete. Also, there is no functionality to create a new project, or a description of how to do so. Should be easy to let lingo do it via VS Code once it's available as a library...

vinzbarbuto · 2024-10-11T07:00:04Z

If my understanding is correct -- going back to your example -- if you open VS Code in main_lf_root, the tree view is not shown. Then, if you navigate to ./src/lf_project_1/ in the file explorer and open SourceFile.lf, the package explorer will show whatever if will find in lf_project_1. Then, if you open./src/lf_project_2/SourceFile.lf, it will show what's in lf_project_2, and when you switch back to ./src/lf_project_1/SourceFile.lf again, the package explorer will show what's in lf_project_1 again, etc.

No. Let's consider 3 cases:

I open VS Code in a project named project with the following structure:

├── project
...
│   ├── src/ # Contains LF source files
│   │  ├── SourceFile.lf # Can also be in a subfolder, e.g. source1/SourceFile.lf
│   │  ├── lib/ # Directory for storing reusable reactors
│   │  │   ├── Input.lf # Example: reactor capturing external inputs (e.g., Microphone, Camera)
│   ├── build/   # Directory containing lingo packages
│   │  ├── lfc_include/ # Directory for storing reusable reactors
│   │  │   ├── lingo_downloaded_library/
...

Regardless of the file opened in the editor, the tree view will display the following structure:

├── project
│   ├── Lingo Packages/ 
│   │  ├──lingo_downloaded_library/
│   ├── Local Libraries/
│   │  ├── Input.lf # Example: reactor capturing external inputs (e.g., Microphone, Camera)
│   ├── Source Files/ 
│   │  ├── SourceFile.lf
...

If I open a file in the active editor that is under the same root as project, such as path/to/project/src/SourceFile.lf, the tree view turns green; otherwise, it stays gray.

I open VS Code in the main_lf_root folder, which contains multiple projects, such as:

├── main_lf_root/
│   ├── lf_project_1/ 
│   │   ├── src/ # Contains LF source files of lf_project_1
│   │   │  ├──SourceFile.lf
│   │   │  ├── lib/ # Directory for storing reusable reactors of lf_project_1
│   │   │  │   ├── Input.lf # Ex: reactor capturing external inputs (e.g., Microphone, Camera)
│   │   ├── build/   # directory containing lingo packages of lf_project_1
│   │   │  ├── lfc_include/ # Directory for storing reusable reactors
│   │   │  │   ├── lingo_downloaded_library/
│   ├── lf_project_2/  # Same structure as lf_project_1
...

In this corner case, since this structure does not match the intended LF project structure, no tree view is shown. Instead, a message appears saying that no LF project has been found. To view the tree, you should open VS Code with one of the lf_project_x folders (which are proper LF projects).

I open VS Code as in case 2, but this time, multiple projects are located under main_lf_root/src, as follows:

├── main_lf_root/
│   ├── src/ # Contains LF source files
│   │  ├── lf_project_1/ 
│   │  │   ├── src/ # Contains LF source files of lf_project_1
│   │  │   │  ├──SourceFile.lf
│   │  │   │  ├── lib/ # Directory for storing reusable reactors of lf_project_1
│   │  │   │  │   ├── Input.lf # Ex: reactor capturing external inputs (e.g., Microphone, Camera)
│   │  │   ├── build/   # directory containing lingo packages of lf_project_1
│   │  │   │  ├── lfc_include/ # Directory for storing reusable reactors
│   │  │   │  │   ├── lingo_downloaded_library/
│   │  ├── lf_project_2/  # Same structure as lf_project_1
...

This is a very strong corner case because, according to the definition I provided above, it satisfies the first condition:

It contains *.lf files in a folder named src/. In this case, the src/ folder is identified, and the directory above it will be considered the project root. These files will populate the Source Files section of the tree view.

Since the search regex for Source Files is src/**/*.lf, it accommodates cases where the source file is located in a subfolder within src/, e.g., src/test/Test.lf. Consequently, all source files contained in both lf_project_1 and lf_project_2 will be indexed, as the pattern main_lf_root/src/lf_project_1/src/Source.lf matches the regex. In this case, the tree view will show the following structure:

├── lf_project_1
│   ├── Source Files/ 
│   │  ├── SourceFile.lf
├── lf_project_2
│   ├── Source Files/ 
│   │  ├── SourceFile.lf
...

However, since this is not the behavior we want, and the directory containing multiple projects is not considered an LF project, we aim to encourage the desired behavior of opening a single project. I'm working on fixing this by excluding the following path from the regex search: src/**/src/*.lf. By excluding this path, corner case 3 effectively becomes corner case 2, meaning no tree view will be shown—only the message will appear.

To conclude, as you can see, what is displayed in the tree view is determined by the workspace you open. It does not change dynamically based on the active file opened in the editor but is defined statically when you first open a project in VSCode. The behavior we want to enforce, based on my understanding, is scenario number 1: a single LF project.

src/lfview/lf-data-provider.ts

lhstrh

These are all great improvements! Let's go ahead and merge.

vinzbarbuto added 2 commits October 9, 2024 18:02

Indixing limited to single LF Projects

726df97

openFolder command instead of refreshEntries in viewsWelcome

372af0e

vinzbarbuto requested review from edwardalee and lhstrh October 10, 2024 18:06

Modified search path for source files

f53f2bc

lhstrh reviewed Oct 10, 2024

View reviewed changes

lhstrh reviewed Oct 11, 2024

View reviewed changes

lhstrh mentioned this pull request Oct 11, 2024

Updated README to reflect recent UI changes #180

Merged

vinzbarbuto added 3 commits October 12, 2024 13:07

Excluded nested 'src' folders from searchSourceFiles

4293ccd

Exclude the path 'src/lib/**' from 'searchSourceFiles'

5b03d0b

Added support to Windows path

4d80754

vinzbarbuto mentioned this pull request Oct 15, 2024

Updated documentation for upcoming VSCode extension release lf-lang/lf-lang.github.io#288

Merged

vinzbarbuto marked this pull request as ready for review October 15, 2024 18:50

vinzbarbuto added 3 commits October 15, 2024 16:57

Fix Windows path on VSCode extension - v2

2d617a2

Fix Windows path on VSCode extension - full tested

8693054

Renamed view name and Lingo Packages section

7d2b1f7

lhstrh reviewed Oct 16, 2024

View reviewed changes

src/lfview/lf-data-provider.ts Outdated Show resolved Hide resolved

Update src/lfview/lf-data-provider.ts

3f0a776

lhstrh approved these changes Oct 16, 2024

View reviewed changes

lhstrh merged commit 0a34f75 into main Oct 16, 2024
3 checks passed

lhstrh deleted the improvement/lf-project-indexing branch October 16, 2024 18:15

lhstrh added the enhancement New feature or request label Oct 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Indexing single Lingua Franca projects #181

Indexing single Lingua Franca projects #181

vinzbarbuto commented Oct 10, 2024

vinzbarbuto commented Oct 10, 2024

vinzbarbuto commented Oct 10, 2024

lhstrh commented Oct 10, 2024

edwardalee commented Oct 10, 2024 •

edited

Loading

lhstrh commented Oct 10, 2024

edwardalee commented Oct 10, 2024

vinzbarbuto commented Oct 10, 2024

lhstrh commented Oct 10, 2024 •

edited

Loading

lhstrh commented Oct 10, 2024

lhstrh Oct 10, 2024

vinzbarbuto Oct 10, 2024

lhstrh Oct 11, 2024

lhstrh Oct 11, 2024

vinzbarbuto Oct 11, 2024

edwardalee commented Oct 10, 2024

lhstrh commented Oct 10, 2024

vinzbarbuto commented Oct 10, 2024

lhstrh commented Oct 11, 2024 •

edited

Loading

lhstrh Oct 11, 2024

vinzbarbuto Oct 11, 2024

lhstrh Oct 11, 2024

vinzbarbuto commented Oct 11, 2024

lhstrh left a comment

Indexing single Lingua Franca projects #181

Indexing single Lingua Franca projects #181

Conversation

vinzbarbuto commented Oct 10, 2024

vinzbarbuto commented Oct 10, 2024

vinzbarbuto commented Oct 10, 2024

lhstrh commented Oct 10, 2024

edwardalee commented Oct 10, 2024 • edited Loading

lhstrh commented Oct 10, 2024

edwardalee commented Oct 10, 2024

vinzbarbuto commented Oct 10, 2024

lhstrh commented Oct 10, 2024 • edited Loading

lhstrh commented Oct 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edwardalee commented Oct 10, 2024

lhstrh commented Oct 10, 2024

vinzbarbuto commented Oct 10, 2024

lhstrh commented Oct 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vinzbarbuto commented Oct 11, 2024

lhstrh left a comment

Choose a reason for hiding this comment

edwardalee commented Oct 10, 2024 •

edited

Loading

lhstrh commented Oct 10, 2024 •

edited

Loading

lhstrh commented Oct 11, 2024 •

edited

Loading