feat: add sas 9 remote (via IOM bridge) support

[scottdover]

Summary
This adds sas 9 IOM bridge support. There are a few things to note here:

• This uses ITC to connect to SAS 9 via the IOM bridge protocol. Our current local sas 9 connection uses the same underlying connection. Therefore, instead of recreating the wheel, our current com session was modified.
• All of the connection files were moved from com -> itc to better match connection info
• fetching file results no longer assumes the file is on your local machine. Instead, it uses the file service to grab the results from the last run

Testing

• Test running multiple sas programs via SAS 9.4 (remote - IOM)
• Test running multiple sas programs via SAS 9.4 (local)
• Test running with and without ods results enabled
• Tested printing ODS results of various sizes (small tables, tables w/ 1k+ rows)

[scnwwu]

Looks good to me. Thanks.

[Zhirong2022]

Create a new profile with SAS 9.4 (Remote-IOM). The property description is not consistent with the connection type. Put mouse over 'host', 'port' and 'username'.

[Zhirong2022]

I went through all the issues raised in PR.
And now only two left for verification:

1. Unreadable Chinese characters, the root cause is that I use SAS Dark theme to render the specific SAS code. So, it is nothing to do with OS locale, VSC language configure or locale94/IOM connection.
2. As @clangsmith mentioned the server configured with encrypted communications (specifically, Triple DES algorithm). It cannot connect to the server. I have no idea how to do verification.

[scottdover]

As @clangsmith mentioned the server configured with encrypted communications (specifically, Triple DES algorithm). It cannot connect to the server. I have no idea how to do verification.

Fwiw, we believe this one is now resolved. @clangsmith tested this yesterday I believe

[clangsmith]

As @clangsmith mentioned the server configured with encrypted communications (specifically, Triple DES algorithm). It cannot connect to the server. I have no idea how to do verification.

Fwiw, we believe this one is now resolved. @clangsmith tested this yesterday I believe

Correct. I have tested it and it is now (after Scott's fix) working as expected for me -- the ODS results are now appearing. I played around with that server using our extension for awhile and did not encounter any other issues.

[Zhirong2022]

The issue that Chinese characters were not rendered correctly has been fixed in the latest package. However I found a performance issue with coding running.
1.Suppose I ran the same sample code against old package and new package.
old package: 5-8 seconds to show result page
new package 50seconds-1miniute30seconds to show result page
2. If try to run the given code below to show a big table with same IOM connection.

proc print data=maps.africa;

old package: 1minute50seconds to show result table
new package: sometimes it will keep showing 'SAS code running' without any response.
At such scenario, try to cancel the running. The running man icon will be disabled even switching to a new profile.

[scottdover]

The issue that Chinese characters were not rendered correctly has been fixed in the latest package. However I found a performance issue with coding running. 1.Suppose I ran the same sample code against old package and new package. old package: 5-8 seconds to show result page new package 50seconds-1miniute30seconds to show result page 2. If try to run the given code below to show a big table with same IOM connection.

proc print data=maps.africa;

old package: 1minute50seconds to show result table new package: sometimes it will keep showing 'SAS code running' without any response. At such scenario, try to cancel the running. The running man icon will be disabled even switching to a new profile.

This was one of my concerns in making a change to "collect all the bytes" first (in ad48cd1). Before this commit, we were reading bytes and writing them directly to a file. Now, we are storing all of the bytes in memory (which we'll eventually exhaust) and then attempting to encode all of them at once (which again, could be a problem for a huge number of bytes).

I'm not sure the best approach to take here. It seems the options are:

• Write to the file system as often as possible, keeping a lower memory overhead but ending up with some encoding issues, or...
• Write to the file system once, with a much higher memory footprint and potential for failed runs, but no encoding issues

@clangsmith @smorrisj @scnwwu Thoughts on how to proceed, here?

[scnwwu]

The issue that Chinese characters were not rendered correctly has been fixed in the latest package. However I found a performance issue with coding running. 1.Suppose I ran the same sample code against old package and new package. old package: 5-8 seconds to show result page new package 50seconds-1miniute30seconds to show result page 2. If try to run the given code below to show a big table with same IOM connection.

proc print data=maps.africa;

old package: 1minute50seconds to show result table new package: sometimes it will keep showing 'SAS code running' without any response. At such scenario, try to cancel the running. The running man icon will be disabled even switching to a new profile.

This was one of my concerns in making a change to "collect all the bytes" first (in ad48cd1). Before this commit, we were reading bytes and writing them directly to a file. Now, we are storing all of the bytes in memory (which we'll eventually exhaust) and then attempting to encode all of them at once (which again, could be a problem for a huge number of bytes).

I'm not sure the best approach to take here. It seems the options are:

• Write to the file system as often as possible, keeping a lower memory overhead but ending up with some encoding issues, or...
• Write to the file system once, with a much higher memory footprint and potential for failed runs, but no encoding issues

@clangsmith @smorrisj @scnwwu Thoughts on how to proceed, here?

Some thoughts:

1. How about simply read files with Nodejs for local connection (same as main branch)? Currently it already checks this._config.protocol === ITCProtocol.COM to resolve file path. I don't know if it will be faster than the IOM approach. If we're not able to make a perfect solution for remote IOM, at least it's not a regression compared to v1.5.
2. How about replace System.IO.StreamWriter with System.IO.FileStream? So we read bytes and write bytes thus can be safely sliced. I guess this is a desired way if it works, as it supports text and non-text files uniformly.
3. If the above doesn't work, how about replace $objFile.OpenBinaryStream with $objFile.OpenTextStream? It looks like designed for characters.

[clangsmith]

The issue that Chinese characters were not rendered correctly has been fixed in the latest package. However I found a performance issue with coding running. 1.Suppose I ran the same sample code against old package and new package. old package: 5-8 seconds to show result page new package 50seconds-1miniute30seconds to show result page 2. If try to run the given code below to show a big table with same IOM connection.

proc print data=maps.africa;

old package: 1minute50seconds to show result table new package: sometimes it will keep showing 'SAS code running' without any response. At such scenario, try to cancel the running. The running man icon will be disabled even switching to a new profile.

This was one of my concerns in making a change to "collect all the bytes" first (in ad48cd1). Before this commit, we were reading bytes and writing them directly to a file. Now, we are storing all of the bytes in memory (which we'll eventually exhaust) and then attempting to encode all of them at once (which again, could be a problem for a huge number of bytes).
I'm not sure the best approach to take here. It seems the options are:

• Write to the file system as often as possible, keeping a lower memory overhead but ending up with some encoding issues, or...
• Write to the file system once, with a much higher memory footprint and potential for failed runs, but no encoding issues

@clangsmith @smorrisj @scnwwu Thoughts on how to proceed, here?

Some thoughts:

1. How about simply read files with Nodejs for local connection (same as main branch)? Currently it already checks this._config.protocol === ITCProtocol.COM to resolve file path. I don't know if it will be faster than the IOM approach. If we're not able to make a perfect solution for remote IOM, at least it's not a regression compared to v1.5.
2. How about replace System.IO.StreamWriter with System.IO.FileStream? So we read bytes and write bytes thus can be safely sliced. I guess this is a desired way if it works, as it supports text and non-text files uniformly.
3. If the above doesn't work, how about replace $objFile.OpenBinaryStream with $objFile.OpenTextStream? It looks like designed for characters.

@scottdover and I stepped through the EG code and we are now doing essentially the same thing that EG is (pseudo-code):

binaryStream = SAS.IFileref.OpenBinaryStream(Mode.StreamOpenModeForReading)
while (true)
{
	binaryStream.Read(CommonDownloadHelper.ChunkSize, out data); // EG ChunkSize is 8192
	if (data == null || data.Length == 0)
		break;
	fileStream.Write((byte[])data, 0, data.Length);
}

The previous issue is that we were encoding the byte chunks as we received them, which would break up DBCS characters if they fell on the border.

The testing I have done since the last fix appears to have resolved the issue. I also feel good knowing this is now the same way that EG handles it, considering EG has been doing it for many years (field-proven experience).

(Btw, given that printing maps.africa generates a 20MB results file, I consider that to be an edge-case scenario. EG doesn't even attempt to open results files larger than 5MB by default. I still tried this scenario in the extension with the latest changes, and it was successful, but took about a minute total before I saw the results. This is not unreasonable in my opinion.)

@scnwwu, @Zhirong2022, when you all return next week, please take another look, and if you feel this is reasonably addressed, don't encounter any other serious issues, or have any other concerns, please feel free to go ahead and merge this in.

[Zhirong2022]

All issues have been fixed properly.

[scottdover]

Thanks, @scnwwu and @Zhirong2022 🎉