Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SNOW-950923 fix to provide QueryID for failures during GET/PUT file operations #825

Merged
merged 17 commits into from
Feb 21, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
512cd0b
SNOW-950923 fix to provide QueryID for failures during GET/PUT file o…
sfc-gh-mhofman Nov 30, 2023
3305fb6
SNOW-950923 fix for two different SnowflakeDbExceptions not to collide
sfc-gh-mhofman Dec 5, 2023
d11a361
SNOW-950923 ensuring any exception throw during file transfer provide…
sfc-gh-mhofman Dec 6, 2023
03e9669
SNOW-950923 changed type of exception thrown when directory/file not …
sfc-gh-mhofman Dec 6, 2023
5100674
SNOW-950923 Review and github actions improvements
sfc-gh-mhofman Jan 12, 2024
3179729
SNOW-950923 codecov fixes, review suggestions to provide more checks …
sfc-gh-mhofman Jan 17, 2024
156784c
Github warning fixes, removal of some other warnings related to rethr…
sfc-gh-mhofman Jan 30, 2024
4ef2c85
Reverting last commit change as it doesn't work for NET 4.7.*
sfc-gh-mhofman Jan 30, 2024
a6dbb09
SNOW-950923 Description of the changes regarding query id in exceptio…
sfc-gh-mhofman Feb 12, 2024
8f474e1
SNOW-950923 Description fix for the changes regarding query id in exc…
sfc-gh-mhofman Feb 13, 2024
8f394c8
SNOW-950923 review fixes and improved docs on GET/PUT execution failures
sfc-gh-mhofman Feb 16, 2024
9ed5361
SNOW-950923 Wrapping Exceptions on GET/PUT commands
sfc-gh-mhofman Feb 16, 2024
1a7d139
SNOW-950923 revert repackaging exception when non-put/get command are…
sfc-gh-mhofman Feb 19, 2024
3d90e7f
Fix failures on mac
sfc-gh-mhofman Feb 19, 2024
7ca5fe9
SNOW-950923 fix flaky test for login retries for 404 error
sfc-gh-mhofman Feb 21, 2024
3cbf4a3
SNOW-950923 review fixes
sfc-gh-mhofman Feb 21, 2024
9297cff
SNOW-950923 fixed typo in refactor
sfc-gh-mhofman Feb 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
SNOW-950923 Wrapping Exceptions on GET/PUT commands
  • Loading branch information
sfc-gh-mhofman committed Feb 16, 2024
commit 9ed53617eb012feff70cf0481df5bec4aad0784a
21 changes: 6 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -691,20 +691,16 @@ using (IDbConnection conn = new SnowflakeDbConnection())

cmd.CommandText = "PUT file://some_data.csv @my_schema.my_stage AUTO_COMPRESS=TRUE";
var reader = cmd.ExecuteReader();
Assert.IsTrue(reader.read()); // on success
Assert.IsTrue(reader.read());
Assert.DoesNotThrow(() => Guid.Parse(cmd.GetQueryId()));
}
catch (SnowflakeDbException e)
{
Assert.DoesNotThrow(() => Guid.Parse(e.QueryId)); // when failed
Assert.That(e.InnerException.GetType(), Is.EqualTo(typeof(FileNotFoundException)));
sfc-gh-mhofman marked this conversation as resolved.
Show resolved Hide resolved
}
catch (Exception e)
{
// for instance on a query execution if a connection was not opened
}
```
In case of a failure an exception will be thrown.
In case of a failure a SnowflakeDbException exception will be thrown with affected QueryId if possible.
If it was after the query got executed this exception will be a SnowflakeDbException containing affected QueryId.
In case of the initial phase of execution QueryId might not be provided.
Inner exception (if applicable) will provide some details on the failure cause and
Expand All @@ -726,22 +722,17 @@ To use the command in a driver similar code can be executed in a client app:

cmd.CommandText = "GET @my_schema.my_stage/stage_file.csv file://local_file.csv AUTO_COMPRESS=TRUE";
var reader = cmd.ExecuteReader();
Assert.IsTrue(reader.read()); // if succeeded
Assert.IsTrue(reader.read()); // True on success, False if failure
Assert.DoesNotThrow(() => Guid.Parse(cmd.GetQueryId()));
}
catch (SnowflakeDbException e)
{
Assert.DoesNotThrow(() => Guid.Parse(e.QueryId)); // on failure
}
catch (Exception e)
{
// some other failure has occurred before query got started
}
```
In case of a failure a SnowflakeDbException will be thrown or DBDataReader will return a False response.
In any case QueryId should be available from the exception or command property.
Similarly to PUT command in some rare cases QueryID might not be possible to provide and a generic Exception
can be thrown.
In case of a failure a SnowflakeDbException will be thrown with affected QueryId if possible.
When no technical or syntax errors occurred but the DBDataReader has no data to process it returns False
without throwing an exception.

Close the Connection
--------------------
Expand Down
69 changes: 68 additions & 1 deletion Snowflake.Data.Tests/IntegrationTests/SFPutGetTest.cs
Original file line number Diff line number Diff line change
Expand Up @@ -209,7 +209,61 @@ public void TestPutFileRelativePathWithoutDirectory()
VerifyFilesAreUploaded(conn, new List<string> { t_inputFilePath }, t_internalStagePath);
}
}


[Test]
public void TestPutGetOnClosedConnectionThrowsWithoutQueryId([Values("GET", "PUT")] string command)
{
t_inputFilePath = "unexisting_file.csv";
t_internalStagePath = $"@{t_schemaName}.{t_stageName}";

// Act
using (var conn = new SnowflakeDbConnection(ConnectionString))
{
// conn.Open(); // intentionally closed
var snowflakeDbException = Assert.Throws<SnowflakeDbException>(() => ProcessFile(command, conn));
Assert.NotNull(snowflakeDbException);
Assert.IsNull(snowflakeDbException.QueryId);
SnowflakeDbExceptionAssert.HasErrorCode(snowflakeDbException, SFError.EXECUTE_COMMAND_ON_CLOSED_CONNECTION);
}
}

[Test]
public void TestGetNonExistentFileReturnsFalseAndDoesNotThrow()
{
t_inputFilePath = "non_existent_file.csv";
t_internalStagePath = $"@{t_schemaName}.{t_stageName}";

// Act
using (var conn = new SnowflakeDbConnection(ConnectionString))
{
conn.Open();
var sql = $"GET {t_internalStagePath}/{t_fileName} file://{s_outputDirectory}";
using (var command = conn.CreateCommand())
{
command.CommandText = sql;
var reader = command.ExecuteReader();
Assert.AreEqual(false, reader.Read());
}
}
}

[Test]
public void TestPutNonExistentFileThrowsWithQueryId()
{
t_inputFilePath = "non_existent_file.csv";
t_internalStagePath = $"@{t_schemaName}.{t_stageName}";

// Act
using (var conn = new SnowflakeDbConnection(ConnectionString))
{
conn.Open();
var snowflakeDbException = Assert.Throws<SnowflakeDbException>(() => PutFile(conn));
Assert.IsNotNull(snowflakeDbException);
Assert.IsNotNull(snowflakeDbException.QueryId);
SnowflakeDbExceptionAssert.HasErrorCode(snowflakeDbException, SFError.IO_ERROR_ON_GETPUT_COMMAND);
}
}

[Test]
public void TestPutFileProvidesQueryIdOnFailure()
{
Expand Down Expand Up @@ -646,6 +700,19 @@ private void GetFile(DbConnection conn)
}
}

private void ProcessFile(String command, SnowflakeDbConnection connection)
{
switch (command)
{
case "GET":
GetFile(connection);
break;
case "PUT":
PutFile(connection);
break;
}
}

private static string[] ReadOutputFileLines()
{
using (var outputStream = File.OpenRead(t_outputFilePath))
Expand Down
7 changes: 6 additions & 1 deletion Snowflake.Data/Client/SnowflakeDbCommand.cs
Original file line number Diff line number Diff line change
Expand Up @@ -339,12 +339,17 @@ private static Dictionary<string, BindingDTO> convertToBindList(List<SnowflakeDb

private void SetStatement()
{
if (connection == null)
{
throw new SnowflakeDbException(SFError.EXECUTE_COMMAND_ON_CLOSED_CONNECTION);
}

var session = (connection as SnowflakeDbConnection).SfSession;

// SetStatement is called when executing a command. If SfSession is null
// the connection has never been opened. Exception might be a bit vague.
if (session == null)
throw new Exception("Can't execute command when connection has never been opened");
throw new SnowflakeDbException(SFError.EXECUTE_COMMAND_ON_CLOSED_CONNECTION);

this.sfStatement = new SFStatement(session);
}
Expand Down
3 changes: 3 additions & 0 deletions Snowflake.Data/Core/ErrorMessages.resx
Original file line number Diff line number Diff line change
Expand Up @@ -185,5 +185,8 @@
</data>
<data name="IO_ERROR_ON_GETPUT_COMMAND" xml:space="preserve">
<value>IO operation failed. Error: {0}</value>
</data>
<data name="EXECUTE_COMMAND_ON_CLOSED_CONNECTION" xml:space="preserve">
<value>Executing command on a non-opened connection.</value>
</data>
</root>
5 changes: 4 additions & 1 deletion Snowflake.Data/Core/SFError.cs
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,10 @@ public enum SFError
BROWSER_RESPONSE_TIMEOUT,

[SFErrorAttr(errorCode = 270058)]
IO_ERROR_ON_GETPUT_COMMAND
IO_ERROR_ON_GETPUT_COMMAND,

[SFErrorAttr(errorCode = 270059)]
EXECUTE_COMMAND_ON_CLOSED_CONNECTION
}

class SFErrorAttr : Attribute
Expand Down
12 changes: 7 additions & 5 deletions Snowflake.Data/Core/SFStatement.cs
Original file line number Diff line number Diff line change
Expand Up @@ -409,14 +409,16 @@ internal SFBaseResultSet Execute(int timeout, string sql, Dictionary<string, Bin
return BuildResultSet(response, CancellationToken.None);
}
}
catch (SnowflakeDbException ex)
{
logger.Error($"Query execution failed, QueryId: {ex.QueryId??"unavailable"}", ex);
_lastQueryId = ex.QueryId ?? _lastQueryId;
throw;
}
catch (Exception ex)
{
logger.Error("Query execution failed.", ex);
if (ex is SnowflakeDbException snowflakeDbException)
{
this._lastQueryId = snowflakeDbException.QueryId;
}
throw;
throw new SnowflakeDbException(ex, SFError.INTERNAL_ERROR);
}
finally
{
Expand Down
Loading