BDX transfer support for Python tests (project-chip#34821)

* Add the python-C++ translation.

* Add a BDX transfer server to handle unsolicited BDX init messages.

* Add the manager to implement the transfer pool.

* Add the initial implementation of a BDX transfer.

* Use BdxTransfer in the other classes.

* Update constructors to set the delegates etc. correctly.

* Implement the C++ side of the barrier. Move the data callback into the transfer delegate.

* Add a way to map the transfer to the python contexts.

* Fix some of the minor TODOs.

* Add init/shutdown to the transfer server.

* Start on the implementation of the Python side.

Also add the transfer obtained context to the C++ methods relating to expecting transfers.

* Listen for all BDX protocol messages rather than just the init messages.

* Fix minor issues in the transfer server.

* Implement a good chunk of the python side.

* Fix compile errors.

* Fix a number of issues preventing the BDX python code from running at all.

* Return the results of the python-C methods.

* Fix the async-ness of the methods that prepare the system to receive a BDX transfer.

Also run the python BDX initialisation.

* Initialise the BDX transfer server.

Also ignore the BDX transfer server implementation that only handles diagnostic logs.

* Fixes necessary to await on the future from PrepareToReceive/SendBdxData.

* Call Responder::PrepareForTransfer from BdxTransfer.
* Correctly schedule satisfying the future on the event loop.
* Use the real property to determine if a PyChipError was a success.

* Fix sending the accept message.

* Acknowledge received blocks so the BDX transfer continues.

Also don't ignore all messages after the init.

* Fix the parameters of the python callback methods.

* Add another async transaction class to handle the transfer completed callback.

* Add comments to the C++ code.

* Add a test for the BDX transfer that uses the diagnostic logs cluster.

* Move the calls to release a transfer out of the manager so it works the way one would expect.

* Delay releasing the C++ BDX transfer object until after it's no longer in use.

* Verify the diagnostic logs response is a success.

* Restyled by whitespace

* Restyled by clang-format

* Restyled by gn

* Restyled by autopep8

* Restyled by isort

* Improve BdxTransferManager's comments.

* Use a vector for the data to send over a BDX transfer rather than a raw pointer.

* Minor renames.

* Improve the error message when the BDX transfer pool is exhausted.

* Minor fixes.

* remove a check that was inadvertently kept.
* print a log message when something that shouldn't happen inevitably does.
* use user_params to get the end user support log test parameter.

* Pass the status report's status code up the stack.

* Merge the BDX transfer server into the manager.

* Rename BdxTransferManager to TestBdxTransferServer.

* Minor cleanup.

* Rename TransferData to TransferInfo.
* Change `!=` to `is not` in python.
* Add missing type annotation.

* Improve the documentation of the ownership in the C++ side.

* Restyled by clang-format

* Restyled by autopep8

* Update the new test to work with the new formatting.

Also remove an unnecessary conversion to bytearray.

* Lint fixes.

* Fix clang-tidy errors.

* Several fixes suggested by Andrei.

* Fix a name in a comment.

* Fix issues preventing test from working.

Also:
* Split the accept function into one for sending data and one for receiving data.
* Return bytes instead of a bytearray when receiving data.
* Add typing to the data callback.

* Rename the methods that accept transfers so it's clear which way the data is flowing.

* Add doc comments to the Python classes and methods.

* Fix issues found by mypy.

* Restyled by clang-format

* Restyled by autopep8

* Fix python lint error.

* Explicitly truncate the status code when generating the error.

* Generate the diagnostic log to transfer in the test.

---------

Co-authored-by: Restyled.io <[email protected]>

src/controller/python/BUILD.gn

@@ -63,6 +63,11 @@ shared_library("ChipDeviceCtrl") {
       "ChipDeviceController-StorageDelegate.cpp",
       "ChipDeviceController-StorageDelegate.h",
       "OpCredsBinding.cpp",
+      "chip/bdx/bdx-transfer.cpp",
+      "chip/bdx/bdx-transfer.h",
+      "chip/bdx/bdx.cpp",
+      "chip/bdx/test-bdx-transfer-server.cpp",
+      "chip/bdx/test-bdx-transfer-server.h",
       "chip/clusters/attribute.cpp",
       "chip/clusters/command.cpp",
       "chip/commissioning/PlaceholderOperationalCredentialsIssuer.h",
@@ -166,6 +171,10 @@ chip_python_wheel_action("chip-core") {
         "chip/ChipStack.py",
         "chip/FabricAdmin.py",
         "chip/__init__.py",
+        "chip/bdx/Bdx.py",
+        "chip/bdx/BdxProtocol.py",
+        "chip/bdx/BdxTransfer.py",
+        "chip/bdx/__init__.py",
         "chip/ble/__init__.py",
         "chip/ble/commissioning/__init__.py",
         "chip/ble/get_adapters.py",
@@ -235,6 +244,7 @@ chip_python_wheel_action("chip-core") {
 
   py_packages = [
     "chip",
+    "chip.bdx",
     "chip.ble",
     "chip.ble.commissioning",
     "chip.configuration",

src/controller/python/chip/ChipDeviceCtrl.py

@@ -48,6 +48,7 @@
 from . import FabricAdmin
 from . import clusters as Clusters
 from . import discovery
+from .bdx import Bdx
 from .clusters import Attribute as ClusterAttribute
 from .clusters import ClusterObjects as ClusterObjects
 from .clusters import Command as ClusterCommand
@@ -1343,6 +1344,36 @@ def WriteGroupAttribute(
         # An empty list is the expected return for sending group write attribute.
         return []
 
+    def TestOnlyPrepareToReceiveBdxData(self) -> asyncio.Future:
+        '''
+        Sets up the system to expect a node to initiate a BDX transfer. The transfer will send data here.
+
+        Returns:
+            - a future that will yield a BdxTransfer with the init message from the transfer.
+        '''
+        self.CheckIsActive()
+
+        eventLoop = asyncio.get_running_loop()
+        future = eventLoop.create_future()
+
+        Bdx.PrepareToReceiveBdxData(future).raise_on_error()
+        return future
+
+    def TestOnlyPrepareToSendBdxData(self, data: bytes) -> asyncio.Future:
+        '''
+        Sets up the system to expect a node to initiate a BDX transfer. The transfer will send data to the node.
+
+        Returns:
+            - a future that will yield a BdxTransfer with the init message from the transfer.
+        '''
+        self.CheckIsActive()
+
+        eventLoop = asyncio.get_running_loop()
+        future = eventLoop.create_future()
+
+        Bdx.PrepareToSendBdxData(future, data).raise_on_error()
+        return future
+
     def _parseAttributePathTuple(self, pathTuple: typing.Union[
         None,  # Empty tuple, all wildcard
         typing.Tuple[int],  # Endpoint

src/controller/python/chip/ChipStack.py

@@ -36,6 +36,7 @@
 import chip.native
 from chip.native import PyChipError
 
+from .bdx import Bdx
 from .clusters import Attribute as ClusterAttribute
 from .clusters import Command as ClusterCommand
 from .exceptions import ChipStackError, ChipStackException, DeviceError
@@ -175,6 +176,7 @@ def HandleChipThreadRun(callback):
         im.InitIMDelegate()
         ClusterAttribute.Init()
         ClusterCommand.Init()
+        Bdx.Init()
 
         builtins.chipStack = self
 

src/controller/python/chip/bdx/Bdx.py

@@ -0,0 +1,230 @@
+#
+#    Copyright (c) 2024 Project CHIP Authors
+#    All rights reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+#
+
+import asyncio
+import builtins
+import ctypes
+from asyncio.futures import Future
+from ctypes import CFUNCTYPE, POINTER, c_char_p, c_size_t, c_uint8, c_uint16, c_uint64, c_void_p, py_object
+from typing import Callable, Optional
+
+import chip
+from chip.native import PyChipError
+
+from . import BdxTransfer
+
+c_uint8_p = POINTER(c_uint8)
+
+
+_OnTransferObtainedCallbackFunct = CFUNCTYPE(
+    None, py_object, c_void_p, c_uint8, c_uint16, c_uint64, c_uint64, c_uint8_p, c_uint16, c_uint8_p, c_size_t)
+_OnFailedToObtainTransferCallbackFunct = CFUNCTYPE(None, py_object, PyChipError)
+_OnDataReceivedCallbackFunct = CFUNCTYPE(None, py_object, c_uint8_p, c_size_t)
+_OnTransferCompletedCallbackFunct = CFUNCTYPE(None, py_object, PyChipError)
+
+
+class AsyncTransferObtainedTransaction:
+    ''' The Python context when obtaining a transfer. This is passed into the C++ code to be sent back to Python as part
+    of the callback when a transfer is obtained, and sets the result of the future after being called back.
+    '''
+
+    def __init__(self, future, event_loop, data=None):
+        self._future = future
+        self._data = data
+        self._event_loop = event_loop
+
+    def _handleTransfer(self, bdxTransfer, initMessage: BdxTransfer.InitMessage):
+        transfer = BdxTransfer.BdxTransfer(bdx_transfer=bdxTransfer, init_message=initMessage, data=self._data)
+        self._future.set_result(transfer)
+
+    def handleTransfer(self, bdxTransfer, initMessage: BdxTransfer.InitMessage):
+        self._event_loop.call_soon_threadsafe(self._handleTransfer, bdxTransfer, initMessage)
+
+    def _handleError(self, result: PyChipError):
+        self._future.set_exception(result.to_exception())
+
+    def handleError(self, result: PyChipError):
+        self._event_loop.call_soon_threadsafe(self._handleError, result)
+
+
+class AsyncTransferCompletedTransaction:
+    ''' The Python context when accepting a transfer. This is passed into the C++ code to be sent back to Python as part
+    of the callback when the transfer completes, and sets the result of the future after being called back.
+    '''
+
+    def __init__(self, future, event_loop):
+        self._future = future
+        self._event_loop = event_loop
+
+    def _handleResult(self, result: PyChipError):
+        if result.is_success:
+            self._future.set_result(result)
+        else:
+            self._future.set_exception(result.to_exception())
+
+    def handleResult(self, result: PyChipError):
+        self._event_loop.call_soon_threadsafe(self._handleResult, result)
+
+
+@_OnTransferObtainedCallbackFunct
+def _OnTransferObtainedCallback(transaction: AsyncTransferObtainedTransaction, bdxTransfer, transferControlFlags: int,
+                                maxBlockSize: int, startOffset: int, length: int, fileDesignator, fileDesignatorLength: int,
+                                metadata, metadataLength: int):
+    fileDesignatorData = ctypes.string_at(fileDesignator, fileDesignatorLength)
+    metadataData = ctypes.string_at(metadata, metadataLength)
+
+    initMessage = BdxTransfer.InitMessage(
+        transferControlFlags,
+        maxBlockSize,
+        startOffset,
+        length,
+        fileDesignatorData[:],
+        metadataData[:],
+    )
+
+    transaction.handleTransfer(bdxTransfer, initMessage)
+
+
+@_OnFailedToObtainTransferCallbackFunct
+def _OnFailedToObtainTransferCallback(transaction: AsyncTransferObtainedTransaction, result: PyChipError):
+    transaction.handleError(result)
+
+
+@_OnDataReceivedCallbackFunct
+def _OnDataReceivedCallback(context, dataBuffer: c_uint8_p, bufferLength: int):
+    data = ctypes.string_at(dataBuffer, bufferLength)
+    context(data)
+
+
+@_OnTransferCompletedCallbackFunct
+def _OnTransferCompletedCallback(transaction: AsyncTransferCompletedTransaction, result: PyChipError):
+    transaction.handleResult(result)
+
+
+def _PrepareForBdxTransfer(future: Future, data: Optional[bytes]) -> PyChipError:
+    ''' Prepares the BDX system for a BDX transfer. The BDX transfer is set as the future's result. This must be called
+    before the BDX transfer is initiated.
+
+    Returns the CHIP_ERROR result from the C++ side.
+    '''
+    handle = chip.native.GetLibraryHandle()
+    transaction = AsyncTransferObtainedTransaction(future=future, event_loop=asyncio.get_running_loop(), data=data)
+
+    ctypes.pythonapi.Py_IncRef(ctypes.py_object(transaction))
+    res = builtins.chipStack.Call(
+        lambda: handle.pychip_Bdx_ExpectBdxTransfer(ctypes.py_object(transaction))
+    )
+    if not res.is_success:
+        ctypes.pythonapi.Py_DecRef(ctypes.py_object(transaction))
+    return res
+
+
+def PrepareToReceiveBdxData(future: Future) -> PyChipError:
+    ''' Prepares the BDX system for a BDX transfer where this device receives data. This must be called before the BDX
+    transfer is initiated.
+
+    When a BDX transfer is found it's set as the future's result. If an error occurs while waiting it is set as the future's exception.
+
+    Returns an error if there was an issue preparing to wait a BDX transfer.
+    '''
+    return _PrepareForBdxTransfer(future, None)
+
+
+def PrepareToSendBdxData(future: Future, data: bytes) -> PyChipError:
+    ''' Prepares the BDX system for a BDX transfer where this device sends data. This must be called before the BDX
+    transfer is initiated.
+
+    When a BDX transfer is found it's set as the future's result. If an error occurs while waiting it is set as the future's exception.
+
+    Returns an error if there was an issue preparing to wait a BDX transfer.
+    '''
+    return _PrepareForBdxTransfer(future, data)
+
+
+def AcceptTransferAndReceiveData(transfer: c_void_p, dataReceivedClosure: Callable[[bytes], None], transferComplete: Future):
+    ''' Accepts a BDX transfer with the intent of receiving data.
+
+    The data will be returned block-by-block in dataReceivedClosure.
+    transferComplete will be fulfilled when the transfer completes.
+
+    Returns an error if one is encountered while accepting the transfer.
+    '''
+    handle = chip.native.GetLibraryHandle()
+    complete_transaction = AsyncTransferCompletedTransaction(future=transferComplete, event_loop=asyncio.get_running_loop())
+    ctypes.pythonapi.Py_IncRef(ctypes.py_object(dataReceivedClosure))
+    ctypes.pythonapi.Py_IncRef(ctypes.py_object(complete_transaction))
+    res = builtins.chipStack.Call(
+        lambda: handle.pychip_Bdx_AcceptTransferAndReceiveData(transfer, dataReceivedClosure, complete_transaction)
+    )
+    if not res.is_success:
+        ctypes.pythonapi.Py_DecRef(ctypes.py_object(dataReceivedClosure))
+        ctypes.pythonapi.Py_DecRef(ctypes.py_object(complete_transaction))
+    return res
+
+
+def AcceptTransferAndSendData(transfer: c_void_p, data: bytearray, transferComplete: Future):
+    ''' Accepts a BDX transfer with the intent of sending data.
+
+    The data will be copied by C++.
+    transferComplete will be fulfilled when the transfer completes.
+
+    Returns an error if one is encountered while accepting the transfer.
+    '''
+    handle = chip.native.GetLibraryHandle()
+    complete_transaction = AsyncTransferCompletedTransaction(future=transferComplete, event_loop=asyncio.get_running_loop())
+    ctypes.pythonapi.Py_IncRef(ctypes.py_object(complete_transaction))
+    res = builtins.chipStack.Call(
+        lambda: handle.pychip_Bdx_AcceptTransferAndSendData(transfer, c_char_p(data), len(data), complete_transaction)
+    )
+    if not res.is_success:
+        ctypes.pythonapi.Py_DecRef(ctypes.py_object(complete_transaction))
+    return res
+
+
+async def RejectTransfer(transfer: c_void_p):
+    ''' Rejects a BDX transfer.
+
+    Returns an error if one is encountered while rejecting the transfer.
+    '''
+    handle = chip.native.GetLibraryHandle()
+    return await builtins.chipStack.CallAsyncWithResult(
+        lambda: handle.pychip_Bdx_RejectTransfer(transfer)
+    )
+
+
+def Init():
+    handle = chip.native.GetLibraryHandle()
+    # Uses one of the type decorators as an indicator for everything being initialized.
+    if not handle.pychip_Bdx_ExpectBdxTransfer.argtypes:
+        setter = chip.native.NativeLibraryHandleMethodArguments(handle)
+
+        setter.Set('pychip_Bdx_ExpectBdxTransfer',
+                   PyChipError, [py_object])
+        setter.Set('pychip_Bdx_StopExpectingBdxTransfer',
+                   PyChipError, [py_object])
+        setter.Set('pychip_Bdx_AcceptTransferAndReceiveData',
+                   PyChipError, [c_void_p, py_object, py_object])
+        setter.Set('pychip_Bdx_AcceptTransferAndSendData',
+                   PyChipError, [c_void_p, c_uint8_p, c_size_t])
+        setter.Set('pychip_Bdx_RejectTransfer',
+                   PyChipError, [c_void_p])
+        setter.Set('pychip_Bdx_InitCallbacks', None, [
+                   _OnTransferObtainedCallbackFunct, _OnFailedToObtainTransferCallbackFunct, _OnDataReceivedCallbackFunct,
+                   _OnTransferCompletedCallbackFunct])
+
+    handle.pychip_Bdx_InitCallbacks(
+        _OnTransferObtainedCallback, _OnFailedToObtainTransferCallback, _OnDataReceivedCallback, _OnTransferCompletedCallback)

src/controller/python/chip/bdx/BdxProtocol.py

@@ -0,0 +1,23 @@
+#
+#    Copyright (c) 2024 Project CHIP Authors
+#    All rights reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+#
+
+# These BDX constants are defined in the spec.
+
+# SendInit/ReceiveInit Proposed Transfer Control field structure.
+SENDER_DRIVE = 0x10
+RECEIVER_DRIVE = 0x20
+ASYNC = 0x40