OleDB has crashes and access violation exceptions when using databases in specific versions of 64 bit MSAccess

[jkemphbk]

Description

I don't honestly know if this is an Office 365 issue or a .NET issue, but it's becoming a major problem for us, and it seems that the OleDB calls may be having issues.

As of Office 365 update 16.0.18129.20116 (and newer), we are seeing sporadic crashes to desktop in our software that uses OleDB to interact with MSAccess databases, during normal operation.

Most of the time, an Event Log entry is produced like:

Faulting application name: ConsoleApp15.exe, version: 1.0.0.0, time stamp: 0xcbaaa94a
Faulting module name: ucrtbase.dll, version: 10.0.22621.3593, time stamp: 0x10c46e71
Exception code: 0xc0000409
Fault offset: 0x000000000007f6fe
Faulting process id: 0x0x67E4

But no message or error box is produced from the application, it just crashes to desktop.

In Visual Studio, a warning exception can usually be seen some time before the fatal crash, when all exceptions are configured to break when thrown:

Managed Debugging Assistant 'ReportAvOnComRelease' : 'An exception was caught but handled while releasing a COM interface pointer through Marshal.Release or Marshal.ReleaseComObject or implicitly after the corresponding RuntimeCallableWrapper was garbage collected. This is the result of a user refcount error or other problem with a COM object's Release. Make sure refcounts are managed properly. The COM interface pointer's original vtable pointer was 0x6f481198. While these types of exceptions are caught by the CLR, they can still lead to corruption and data loss so if possible the issue causing the exception should be addressed'

This ususally occurs on OleDbCommand.cs, CloseInternalCommand().

Even more rarely, a FatalEngineExecution Exception occurs, I think it's only limited to our main application... I have never seen it in my test application.

Managed Debugging Assistant 'FatalExecutionEngineError' : 'The runtime has encountered a fatal error. The address of the error was at 0xae18e241, on thread 0x4934. The error code is 0xc0000005. This error may be a bug in the CLR or in the unsafe or non-verifiable portions of user code. Common sources of this bug include user marshaling errors for COM-interop or PInvoke, which may corrupt the stack.'

This occurred in OleDbConnectionInternal.cs, ICommandText(), on the return line.

None of these occur if the computer has an older version of Office 365 installed. They also do not occur if the application is configured to run as 32 bit (as it will use Jet instead of ACE). ODBC calls also seem to function as expected... the problem is seemingly only with OleDB.

Reproduction Steps

OleDBException.zip

Seems to be more likely with a higher quantity (and complexity) of DB operations.
Presence of multiple DBCommand objects, as well as calls to all forms of DB command functionality (ExecuteScalar, ExecuteNonQuery, ExecuteReader) seem to be necessary for some reason.

Attached example is about as simple as I could reduce it to. It opens a DB, does a specific sequence of DB actions, then closes it again. This loop is repeated 10,000 times. Usually a crash occurs within the first few hundred. Rarely, more than that.

Run this in debug in Visual Studio 2022, with x64 as the project configuration, and all exceptions configured to break when thrown.

Repeatedly opening and closing the DB in this manner seemed to be the most reliable trigger of this crash, but it isn't the only one. It can and does happen during standard operations of reading and writing to a DB.

Expected behavior

Test program will repeatedly perform DB operations with no issues, no matter how many of them there are.

Actual behavior

Test program will eventually crash to desktop (usually producing a debug-visible exception first) within the first 10k actions on the database.

Regression?

This does not specifically seem to be tied to .NET Framework. Something in MSAccess changed that is now causing OleDB to throw an exception.

Known Workarounds

Difficult to say. For my small example, any small change to the actions or order will cause the exception to stop happening.

For a more complex example, it appears that reducing the DBCommand count to a single instance helps prevent the issue, but I don't know if this is a workable solution for us yet.

Configuration

.NET Framework 4.8.1
Verified on Windows 10 (22H2) and 11 (22H2).
x64

The only environmental factor that seems to affect this is having Office 365 64-bit, version 16.0.18129.20116 or greater.

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @roji, @ajcvickers
See info in area-owners.md if you want to be subscribed.

[roji]

@jkemphbk in 99% of these cases, the problem is in the native OleDB driver, as opposed to the System.Data.OleDB bridge that simply exposes it to .NET; System.Data.OleDB is implemented in pure .NET, and so in principle shouldn't be able to produce an access violation exception.

In other words, if you were to implement your repro above (OleDBException.zip) in C/C++, it's extremely likely that you'd see the same problem. I'd recommend doing that as a way to isolate things.

[jkemphbk]

@roji You are probably right. I can see what I can do, but I don't think I have the time or knowledge to pull off converting the project in a reasonable timeframe. Would I not have to replicate how the System.Data.OleDB bridge calls the driver? The state of my small .NET project is already shaky enough as it is, I would most likely have to find a way of getting the order of everything perfect to get the crash to stay consistent.

[roji]

Would I not have to replicate how the System.Data.OleDB bridge calls the driver?

In theory yes, but there's really not much going on in System.Data.OleDB - it's a pretty thin wrapped, so i'd imagine it shouldn't be too difficult to reproduce the problem outside of .NET.

In any case, there's not much we can do without a clear indication that there's something wrong with System.Data.OleDB itself, so for now, I'll go ahead and close the issue. However, if you need further assistance or manage to show that the same usage is working well outside of .NET, please post back here and we can revisit and reopen as necessary.

[jkemphbk]

Where would the best place to send this issue be? I've tried navigating the tech support hierarchy before to no luck with a previous issue, and I'd like to avoid that.

[roji]

I'm honestly not sure where reports on the Access OleDB driver - I'm assuming that would be part of Office? I'm sorry I can't be of more help here on that...

[jkemphbk]

Darn... well, thanks anyways.

[danmoseley]

Where would the best place to send this issue be? I've tried navigating the tech support hierarchy before to no luck with a previous issue, and I'd like to avoid that.

If nothing else, the crew in https://github.com/dotnet/SqlClient would know.