Coral TPU: "Unable to run inference" unhandled error.

[mac22321]

Area of Concern

• Server

Server Info

Server version:   2.9.5
System:           Windows
Operating System: Windows (Windows 11 24H2)
CPUs:             Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (Intel)
                  1 CPU x 6 cores. 12 logical processors (x64)
GPU (Primary):    Quadro P1000 (4 GiB) (NVIDIA) 
                  Driver: 552.74, CUDA: 12.4 (up to: 12.4), Compute: 6.1, cuDNN: 
System RAM:       32 GiB
Platform:         Windows
BuildConfig:      Release
Execution Env:    Native
Runtime Env:      Production
Runtimes installed:
  .NET runtime:     9.0.0
  .NET SDK:         7.0.410
  Default Python:   Not found
  Go:               Not found
  NodeJS:           18.17.0
  Rust:             Not found
Video adapter info:
  NVIDIA Quadro P1000:
    Driver Version     31.0.15.5274
    Video Processor    Quadro P1000
System GPU info:
  GPU 3D Usage       2%
  GPU RAM Usage      3.8 GiB
Global Environment variables:
  CPAI_APPROOTPATH = <root>
  CPAI_PORT        = 32168

Describe the bug
Using CPAI within the context of Blue Iris NVR software. When Coral TPU object detection module is enabled, an unhandled exception frequently occurs with error message "Unable to run inference: There is at least 1 reference to internal data in the interpreter in the form of a numpy array or slice. Be sure to only hold the function returned from tensor() if you are using raw data access.".

Actual error:

T-394 msec  [475 msec]   

[
	{
		"api":"objects",
		"found":{
			"success":true,
			"inferenceMs":7,
			"processMs":85,
			"message":"No objects found",
			"count":0,
			"predictions":[
				]
			,
			"moduleId":"ObjectDetectionCoral",
			"moduleName":"Object Detection (Coral)",
			"code":200,
			"command":"detect",
			"requestId":"d1c26a48-68ae-4714-8725-807e8d1aa788",
			"inferenceDevice":"TPU",
			"analysisRoundTripMs":161,
			"processedBy":"localhost",
			"timestampUTC":"Thu,
			19 Dec 2024 21:07:38 GMT"}
		}
	,
	{
		"api":"ipcam-dark",
		"found":{
			"success":false,
			"error":"Unable to run inference: There is at least 1 reference to internal data\n      in the interpreter in the form of a numpy array or slice. Be sure to\n      only hold the function returned from tensor() if you are using raw\n      data access.",
			"inferenceMs":0,
			"processMs":80,
			"predictions":[
				]
			,
			"message":"",
			"count":0,
			"moduleId":"ObjectDetectionCoral",
			"moduleName":"Object Detection (Coral)",
			"code":500,
			"command":"custom",
			"requestId":"3634a008-e41c-4179-bd56-697a75ae66f4",
			"inferenceDevice":"TPU",
			"analysisRoundTripMs":198,
			"processedBy":"localhost",
			"timestampUTC":"Thu,
			19 Dec 2024 21:07:38 GMT"}
		}
	]

Expected behavior
CPAI should work similarly to how it does when using CPU/GPU hardware and process the image handed to it, returning either a "No objects found" payload

T-28 msec  [380 msec]  *

[
	{
		"api":"objects",
		"found":{
			"message":"No objects found",
			"count":0,
			"predictions":[
				]
			,
			"inferenceDevice":"GPU",
			"inferenceMs":147,
			"processMs":148,
			"analysisRoundTripMs":154,
			"success":true,
			"moduleName":"Object Detection (YOLOv5 .NET)",
			"moduleId":"ObjectDetectionYOLOv5Net",
			"command":"detect",
			"requestId":"11dd1b12-31be-461d-9858-6e8b971ac252",
			"processedBy":"localhost",
			"timestampUTC":"Wed,
			11 Dec 2024 07:13:35 GMT"}
		}
	,
	{
		"api":"ipcam-combined",
		"found":{
			"message":"No objects found",
			"count":0,
			"predictions":[
				]
			,
			"inferenceDevice":"GPU",
			"inferenceMs":150,
			"processMs":150,
			"analysisRoundTripMs":154,
			"success":true,
			"moduleName":"Object Detection (YOLOv5 .NET)",
			"moduleId":"ObjectDetectionYOLOv5Net",
			"command":"custom",
			"requestId":"dc6a3e7d-d679-4832-8023-24aee843fe1b",
			"processedBy":"localhost",
			"timestampUTC":"Wed,
			11 Dec 2024 07:13:35 GMT"}
		}
	]

or an "object found" payload.

T+206 msec  [576 msec]  *

[
	{
		"api":"objects",
		"found":{
			"message":"Found person,
			chair",
			"count":2,
			"predictions":[
				{
					"label":"person",
					"confidence":0.8436014,
					"y_min":1410,
					"x_min":1347,
					"y_max":1724,
					"x_max":1937}
				,
				{
					"label":"chair",
					"confidence":0.47627652,
					"y_min":810,
					"x_min":0,
					"y_max":1366,
					"x_max":653}
				]
			,
			"inferenceDevice":"GPU",
			"inferenceMs":237,
			"processMs":237,
			"analysisRoundTripMs":245,
			"success":true,
			"moduleName":"Object Detection (YOLOv5 .NET)",
			"moduleId":"ObjectDetectionYOLOv5Net",
			"command":"detect",
			"requestId":"807846c8-be05-44c4-96b4-a0377700a893",
			"processedBy":"localhost",
			"timestampUTC":"Wed,
			11 Dec 2024 19:47:03 GMT"}
		}
	,
	{
		"api":"ipcam-combined",
		"found":{
			"message":"No objects found",
			"count":0,
			"predictions":[
				]
			,
			"inferenceDevice":"GPU",
			"inferenceMs":196,
			"processMs":196,
			"analysisRoundTripMs":211,
			"success":true,
			"moduleName":"Object Detection (YOLOv5 .NET)",
			"moduleId":"ObjectDetectionYOLOv5Net",
			"command":"custom",
			"requestId":"7fc2b07d-d5f5-49ef-bedc-446bd14a6e5e",
			"processedBy":"localhost",
			"timestampUTC":"Wed,
			11 Dec 2024 19:47:04 GMT"}
		}
	]

Screenshots

Additional context
https://coral.ai/products/m2-accelerator-ae

[Hardware Resources]

Resource	Device	Status	

<snip>
IRQ 4294967272	Coral PCIe Accelerator	OK	
IRQ 4294967271	Coral PCIe Accelerator	OK	
IRQ 4294967270	Coral PCIe Accelerator	OK	
IRQ 4294967269	Coral PCIe Accelerator	OK	
<snip>

Resource	Device	Status	
<snip>
0xFC000-0xFFFFF	Coral PCIe Accelerator	OK	
0x100000-0x1FFFFF	Coral PCIe Accelerator	OK	
<snip>

[System Drivers]

Name	Description	File	Type	Started	Start Mode	State	Status	Error Control	Accept Pause	Accept Stop	
<snip>
coral	coral	c:\windows\system32\drivers\coral.sys	Kernel Driver	Yes	Manual	Running	OK	Normal	No	Yes	
<snip>

[hpmkruit]

Having similair issues on Agent DVR. Using coral tpu on windows sometimes gives the same error.