Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
SSL support in asyncio, sorted qstr pools, common machine module bindings This release of MicroPython introduces SSL/TLS support to asyncio, for both the client and server sides. The interface matches CPython: `asyncio.open_connection()` and `asyncio.start_serve()` now both accept an `ssl` argument to supply an `SSLContext` object. As part of this, new methods were added to `SSLContext` to load certificates, and certificate date/time validation was enabled on all ports that use mbedTLS. Qstr pools are now sorted, which provides a significant performance boost for `qstr_find_strn()`, which is called a lot during parsing and loading of .mpy files, as well as interning of string objects, which happens in most string methods that return new strings. The static pool (part of the .mpy ABI) isn't currently sorted, but could be in the future. There have been many internal changes to the `machine` module (and on some ports the `os` module) to factor the Python bindings to a common location, reduce code duplication and make the API more consistent across all the ports. And a new `boardgen.py` script has been added to factor pin generation and enable a more consistent `machine.Pin` across ports. For consistency, the following user-facing changes have been made: - cc3200 port: The `machine` module gains `soft_reset()`, `mem8`, `mem16`, `mem32` and `Signal`; it loses `POWER_ON` (replaced by `PWRON_RESET`). `disable_irq()` now returns an (opaque) integer rather than a bool, and `enable_irq(state)` must be passed an argument which is the return value of `disable_irq()`, rather than a bool. In the `os` module, `dupterm()` has been converted to use the common implementation and has semantics the same as other ports, and `uname()` is removed to save space (sys.version and sys.implementation can be used instead). - esp32 port: In the `machine` module, `lightsleep()` and `deepsleep()` no longer take the `sleep` keyword argument, instead it's positional to match other ports. Also, passing 0 here will now do a 0ms sleep instead of acting like nothing was passed. And `reset_cause()` no longer accepts any arguments (before it would just ignore them). - esp8266 port: `machine.idle()` now returns None instead of the time elapsed. The `machine.WDT()` constructor now takes keyword arguments, and accepts the `timeout` argument but raises an exception if it's not the default value (this port doesn't support changing the timeout). - mimxrt port: `machine.freq()` now accepts an argument but raises `NotImplementedError`, and `machine.lightsleep()` has been added but also just raises `NotImplementedError` (this is to make these functions use an implementation common to the other ports). - nrf port: The `machine` module gains `unique_id()` (returns an empty bytes object), `freq()` (raises `NotImplementedError`) and `Signal`. `UART.sendbreak()` is removed, but this method previously did nothing. The `os.dupterm()` function has changed to match the semantics used by all other ports (except it's restricted to accept only `machine.UART` objects). - qemu-arm port: The `machine` module gains `soft_reset()` and `idle()`. - samd port: The `machine.deepsleep()` function now resets after sleeping. - unix port: Gains `machine.soft_reset()`. - zephyr port: The `machine` module gains `soft_reset()`, `mem8`, `mem16`, and `mem32`. The `UART` class gains the following methods: `init()` which supports setting `timeout` and `timeout_char`, `deinit()` which does nothing, `flush()` which raises `OSError(EINVAL)` because it's not implemented, and `any()` and `txdone()` which both raise `NotImplementedError`. The teensy port has been removed in this release. This port was largely unmaintained, had limited features (the only hardware support was for GPIO and timer, and no `machine` module), and only supported a small number of Teensy boards. A new preview versioning scheme has been introduced, whereby non-release builds are a preview of the next, upcoming release. This scheme is compatible with semver and should help to eliminate confusion matching documentation and firmware version numbers, among other things. Black has been replaced with ruff format as the Python code formatter. This required a few small changes to Python code, and now allows linting and formatting with ruff. Bound method instances now support comparison and hashing, matching CPython semantics. The .mpy sub-version has been updated from 6.1 to 6.2 due to a change in the native .mpy ABI. A new option `MICROPY_PREVIEW_VERSION_2` has been added which provides a way to enable features and changes slated for MicroPython 2.x, by running `make MICROPY_PREVIEW_VERSION_2=1`. This is an alternative to having a 2.x development branch, and any feature or change that needs to be "hidden" until 2.x will use this flag. LittleFS has been updated to v2.8.1. The associated MicroPython VfsLfs2 driver can read existing LFS2 filesystems, but any writes will update the filesystem to a newer LFS2 version that cannot be read by older drivers, so take this into account when updating, for example update mboot first. The VFS sub-system has a new file ioctl to set the read-buffer size, which is used by mpremote to significantly increase performance of the "mpremote mount" feature. Manifest files now allow registering an external library path via `add_library(name, path)`. `sys.stdout.buffer.write()` now returns the actual number of bytes written (although this is complicated when output goes to multiple destinations). The esp32 port has been updated to use IDF version 5.0.4, and the initial GC heap size tuned so that, after doubling the heap size, WiFi can still be started and an SSL connection made. `RMT.source_freq()` is now a class method, socket connect timeout has been implemented, RTC user memory is now preserved over most reset causes, and `hashlib.md5` enabled. The mimxrt port has RTC alarm/wakeup functionality added, along with support for `machine.deepsleep()`. The rp2 port sees the introduction of a new `rp2.DMA` class for control over DMA transfers. It has switched to use the same math library as other ports to get more accurate floating point behaviour, and enabled `os.dupterm_notify()` for WebREPL use. The TinyUSB stack is now scheduled to run from the IRQ handler (instead of polled in the VM) which slightly improves performance of the VM and USB. The port also makes better use of event scheduling and WFE to be more efficient. It also has added support for external ADC channels (for example when using the ninaw10 driver). The stm32 port has improved support for STM32H5xx MCUs, including Ethernet support, frequency scaling with HSI, sleep mode and SD card support. The NUCLEO_WL55 board now freezes in the LoRa driver, the I2S driver has improved accuracy of the clock frequency, and mboot now supports Microsoft WCID to set the USB driver. New boards added in this release are: UM_TINYWATCHS3 (esp32 port), POLOLU_3PI_2040_ROBOT, POLOLU_ZUMO_2040_ROBOT and SIL_RP2040_SHIM (rp2 port), NUCLEO_H563ZI (stm32 port). The change in code size since the previous release for various ports is (absolute and percentage change in the text section): bare-arm: +216 +0.381% minimal x86: +624 +0.340% unix x64: +8283 +1.050% stm32: +1368 +0.350% cc3200: +1184 +0.649% esp8266: +800 +0.114% esp32: +35348 +2.100% mimxrt: +2172 +0.602% renesas-ra: +96 +0.015% nrf: +1460 +0.785% rp2: +6100 +1.880% samd: +1476 +0.568% The changes that dominate these numbers are: - bare-arm, minimal: comparing and hashing bound methods, sorted qstr pools - unix: updating LittleFS to 2.8.1, enabling certificate date/time validation, adding SSLContext certificate methods, asyncio SSL support - stm32: sorted qstrs, updating LittleFS to 2.8.1, I2S clock frequency improvements, asyncio SSL support - cc3200: sorted qstrs, more machine module functions, use of the common `os.dupterm` implementation - esp32: switching ESP-IDF from 5.0.2 to 5.0.4 - esp8266: updating LittleFS to 2.8.1 - mimxrt: adding RTC alarm/wakeup functionality, updating LittleFS to 2.8.1, asyncio SSL support - nrf: updating LittleFS to 2.8.1, enabling `machine.Signal`, asyncio SSL support - rp2: using locally-provided math library, adding new `rp2.DMA` class - samd: sorted qstrs, updating LittleFS to 2.8.1, asyncio SSL support With the new sorted qstr pools, performance is significantly improved for qstr-heavy operations, between +50% and +200% improvement. Other areas have their performance unchanged since the last release. Thanks to everyone who contributed to this release: Alessandro Gatti, Andrew Leech, Angus Gratton, Carlosgg, Christian Walther, Damien George, Daniël van de Giessen, Elias Wimmer, Glenn Moloney, iabdalkader, Ihor Nehrutsa, Jeff Epler, Jim Mussared, Kwabena W. Agyeman, Maarten van der Schrieck, Mark Blakeney, Mathieu Serandour, Matthias Urlichs, MikeTeachman, Ned Konz, Nicko van Someren, Pascal Brunot, Patrick Van Oosterwijck, Paul Grayson, Peter Züger, Rene Straub, robert-hh, Scott Zhao, Sebastian Romero, Seon Rozenblum, stijn, Thomas Ackermann, Thomas Wenrich, ThomHPL, Trent Piepho. Contributions were made from the following timezones: -0800, -0700, -0600, -0500, +0000, +0100, +0200, +1000, +1100. The work done in this release was funded in part through GitHub Sponsors, and in part by George Robotics, Planet Innovation, Espressif, Arduino, LEGO Education and OpenMV. What follows is a detailed list of changes, generated from the git commit history, and organised into sections. Main components =============== all: - switch to new preview build versioning scheme - replace "black" with "ruff format" - update Python formatting to ruff-format py core: - vm: don't emit warning when using "raise ... from None" - builtinevex: handle invalid filenames for execfile - objboundmeth: support comparing and hashing bound methods - objboundmeth: optimise check for types in binary_op - obj: generalise mp_get_buffer so it can raise if a flag is set - dynruntime: add mp_get_buffer - persistentcode: bump .mpy sub-version - modthread: initialise nlr_jump_callback_top on threads - makeqstrdefs.py: print a nicer error when preprocessing stage fails - mkrules.mk: add MICROPY_PREVIEW_VERSION_2 - asm: remove unused asm helper macros - qstr: add support for sorted qstr pools - mkrules.mk: add rule for compiling auto-generated source files - runtime: remove declaration of function from inside function - lexer: change token position for new lines - misc: change sizeof to offsetof for variable-length alloc - qstr: special case qstr_find_strn for empty string - obj: fix mp_obj_is_type compilation with C++ - objslice: validate that the argument to indices() is an integer - mkrules: add support for custom manifest variables - modbuiltins: share vstr_add_char's implementation of utf8 encoding - mphal: move configuration of ATOMIC_SECTION macros to mphal.h - add port-agnostic inline functions for event handling - modsys: implement optional sys.intern - mkrules.mk: list hash files as byproducts - makeqstrdefs.py: don't skip output for stale hash file - makeqstrdefs.py: stop generating temporary intermediate file - gc: improve calculation of new heap size in split-heap-auto mode - mkrules.mk: fix dependency file generation for compiler wrappers extmod: - moductypes: validate that uctypes.struct addr argument is an int - asyncio: emit errors to stderr, not stdout - modframebuf: validate FrameBuffer bounds against input buffer - modframebuf: fix FrameBuffer get-buffer implementation - modframebuf: remove FrameBuffer1 from natmod build - vfs_posix: fix relative root path - vfs_posix: fix accidentally passing tests - vfs_posix: fix relative paths on non-root VFS - vfs_posix: fix getcwd() on non-root VFS - vfs_posix: additional tests for coverage of error cases - network_ninaw10: raise an error if nina_ioctl fails - machine_wdt: factor ports' WDT Python bindings to common code - machine_pwm: remove header file and move decls to .c file - machine_i2s: factor ports' I2S Python bindings to common code - machine_i2s: factor stream and ring-buf code - machine_i2s: factor I2S.shift method - machine_i2s: factor I2S.irq method - machine_i2s: factor print function - machine_i2s: factor init_helper argument parsing - machine_i2s: factor comments, some enums and macros - machine_adc: factor ports' ADC Python bindings to common code - machine_uart: factor ports' UART Python bindings to common code - modmachine: clean up decls of machine types to use common ones - modmachine: consolidate simple machine headers into modmachine.h - modmachine: consolidate mem, i2c and spi headers to modmachine.h - network_ninaw10: fix select flags handling in socket poll - remove empty utime_mphal.h file - machine_adc_block: factor esp32 ADCBlock bindings to common code - machine_i2c: do a fast poll during I2C.scan() - vfs_reader: add file ioctl to set read buffer size - modbluetooth: initialise nlr_jump_callback_top for IRQ handlers - vfs_posix_file: make standard file objects non-const - extmod.mk: allow enabling lwip loopback support - modmachine: make I2C/SPI defns available when soft impl enabled - modmachine: factor ports' machine module dict to common code - modmachine: provide common Python bindings for machine.idle() - modmachine: add MICROPY_PY_MACHINE_PIN_BASE option - modmachine: provide common Python bindings for bootloader() - modmachine: provide common bindings for 6 bare-metal functions - modmachine: provide common implementation of disable/enable_irq - mbedtls: enable certificate time/date validation by default - modnetwork: add deinit function to NIC protocol - network_ninaw10: switch to using soft-timer for polling - switch to use new event functions - add lists of libm/libm_dbl source files for ports to use - modssl_mbedtls: add SSLContext certificate methods - modssl_mbedtls: make SSLSocket.getpeercert() optional - modssl_mbedtls: fix parsing of ciphers in set_ciphers method - asyncio: add ssl support with SSLContext - modonewire: adopt Maxim recommended read timings - modonewire: improve write timings for better reliability - modos: factor os.dupterm_notify() function to common extmod code - os_dupterm: prevent recursive execution of mp_os_dupterm_rx_chr - asyncio: remove non-working Stream __aenter__/__aexit__ methods - modselect: handle growing the pollfds allocation correctly - modhashlib: support MD5 with mbedtls 3.x - os_dupterm: let mp_os_dupterm_tx_strn() return num bytes written - vfs_lfs: fix lfs cache_size calculation - nimble: do not set GAP device name after sync shared: - libc/string0: don't deref args for n==0 case - tinyusb: schedule TinyUSB task function from dcd_event_handler - tinyusb: expose mp_usbd_task as a public function - tinyusb: add a helper for hex string conversion - runtime/softtimer: generalise soft_timer to work without SysTick drivers: - ninaw10: add ioctl for reading analog pins - ninaw10: add support for external ADC channels mpy-cross: no changes specific to this component/port lib: - littlefs: update LittleFS to v2.8.1 - uzlib: for matches of the same length, take the closest one - mbedtls_errors: update error list for latest esp32 mbedtls - micropython-lib: update submodule to latest Support components ================== docs: - reference/mpyfiles: document change in .mpy sub-version - library/io: remove io.FileIO and io.TextIOWrapper - reference/micropython2_migration: add migration guide - mimxrt: change the examples which denote a Pin with a number - samd: fix the pinout for SAMD21 Itsy Bitsy Express M0 - library/esp: correct the description of esp.osdebug() - esp32/quickref: add DAC example - library: document SSLContext cert methods and asyncio support examples: - pins.py: remove this pins printing example tests: - basics/boundmeth1.py: add tests for bound method equality/hash - perf_bench: add string/qstr/map tests - extmod/asyncio_as_uasyncio.py: fix qstr order dependency - net_hosted/asyncio_loopback.py: add loopback test - extmod/deflate_compress.py: add a test for optimal compression - float/inf_nan_arith.py: include -inf in argument combos - run-tests.py: skip Thumb2 tests if target doesn't support them - update SSL network tests to use SSLContext, and work on CPython tools: - ci.sh: ensure enough commits are fetched for a common ancestor - boardgen.py: add initial implementation of a common make-pins.py - tinytest-codegen.py: externalise tests list - mpremote: add ioctl to specify large read buffer size - ci.sh: build ESP32_GENERIC-SPIRAM as part of esp32 CI - ci.sh: set `ulimit -n` for unix CI - manifestfile.py: add support for external libraries CI: - workflows: pin ruff to 0.1.0 and change flags for new version - workflows: cache ESP-IDF checkout and installation - workflows: use build matrix for esp32 port - workflows: enable build matrix for stm32 port - workflows: enable ccache for esp32 build - workflows: bump actions/github-script from 6 to 7 - workflows: bump actions/setup-python from 4 to 5 - workflows: bump actions/upload-artifact from 3 to 4 The ports ========= all ports: - make all ports skip execution of main.py if boot.py fails - remove SRC_QSTR_AUTO_DEPS from all ports' Makefiles - standardise arguments and output for make-pins.py script - fix incorrect identifiers on Arduino boards - move definitions of ATOMIC_SECTION macros to mphalport.h - switch build to use common lib/libm list of source files - fix sys.stdout.buffer.write() return value bare-arm port: - lib: add minimal strncmp implementation cc3200 port: - boards/make-pins.py: don't generate qstrs - boards/make-pins.py: add a note about tools/boardgen.py - mods/modmachine: use common implementation of disable/enable_irq - convert dupterm to use common extmod implementation - convert os module to use extmod version - eliminate dependency on stm32's irq.h - application.mk: don't add stm32 to build include path embed port: no changes specific to this component/port esp8266 port: - rename MICROPY_ESPNOW to MICROPY_PY_ESPNOW - machine_spi: rename machine_hspi to machine_spi - esp_mphal: make atomic section more atomic - modmachine: use common implementation of disable/enable_irq - avoid including ep_mphal.h directly - update port to use new event functions esp32 port: - rename MICROPY_ESPNOW to MICROPY_PY_ESPNOW - boards: update UM board image names for consistency - boards/UM_TINYWATCHS3: add new UM TinyWATCH S3 board - boards: update UM board settings to use custom PID/VID - network_ppp: allow building with IPv6 disabled - poll serial/JTAG for unread data to prevent blocking - mpconfigport: remove port-specific GAP name - machine_uart: add error checking for IDF API's - network_ppp: reduce PPP thread CPU usage - boards: disable ALPN support - network_lan: fix and simplify the code for ETH-SPI devices - network_lan: fix LAN.isconnected() - network_lan: register the hostname setting for Ethernet - modmachine: fix deepsleep() when previous sleep delay was set - boards: reduce size of D2WD and OTA firmware - use better build settings for ESP32-C3 - mphalport: add function to wake main from separate FreeRTOS task - usb: wake main thread when USB receives data - machine_pin: make irq object a sub-field of pin object - esp32_rmt: change RMT.source_freq() to class method - esp32_rmt: add RMT.PULSE_MAX constant - modsocket: implement socket connect timeout - modsocket: try garbage collection if the socket limit is reached - boards/sdkconfig.base: fix increasing log level via esp.osdebug() - boards/ESP32_GENERIC: reduce size of D2WD variant to fit in flash - uart: preserve console UART clock, fix UART console with DFS - uart: make compatible with sclk type change in ESP-IDF 5.3 - network_wlan: reduce RAM usage if SPIRAM fails to initialise - network_wlan: fix network.WLAN.status() to return better info - esp32_rmt: fix RMT looping - enable mbedtls cert time validation - boards: enable further IRAM saving opts to fit ESP32-SPIRAM fw - machine_rtc: preserve RTC user memory over most reset causes - boards/UM_TINYPICO: fix typo in baudrate instructions - boards/sdkconfig.base: disable unused mbedtls options - machine_i2c: use APB_CLK_FREQ instead of I2C_APB_CLK_FREQ - modnetwork: add WiFi AUTH_WPA3_ENT_192 authenticate mode - machine_dac: support one-shot mode of driver - modmachine: fix deprecated esp_pm_config_XXX_t - machine_i2s: fix deprecated fields and constants - mpconfigport: keep some funcs out of IRAM for ESP32-SPIRAM builds - boards/ESP32_GENERIC: disable network.LAN and VM-opt on D2WD - change minimum supported IDF version to v5.0.4 - re-enable custom mbedtls error string tables - add MICROPY_GC_INITIAL_HEAP_SIZE option and tune it - mpconfigport: enable MICROPY_PY_HASHLIB_MD5 mimxrt port: - led: fix LED init call from main, and simplify led_init - boards: define missing SNVS pins for all processors - machine_rtc: add RTC alarm/wakeup functionality - modmachine: add support for machine.deepsleep - boards/make-pins.py: update to use tools/boardgen.py - modmachine: fix settings for the MIMXRT1170 board - boards/OLIMEX_RT1010: adjust the UART pin assignment minimal port: no changes specific to this component/port nrf port: - boards: automatically configure MICROPY_PY_MACHINE_PWM - modules/machine: use SPI Python bindings provided by extmod - boards/make-pins.py: don't generate qstrs - boards/make-pins.py: add a note about tools/boardgen.py - use MICROPY_PY_MACHINE_SPI instead of MICROPY_PY_MACHINE_HW_SPI - use dupterm_objs[0] instead of board_stdio_uart - convert os module to use extmod version - mpconfigport: enable MICROPY_PY_MACHINE_BOOTLOADER - boards/ARDUINO_NANO_33_BLE_SENSE: don't enable MICROPY_MBFS - main: add /flash and /flash/lib to sys.path pic16bit port: no changes specific to this component/port powerpc port: no changes specific to this component/port qemu-arm port: no changes specific to this component/port renesas-ra port: - boards/make-pins.py: don't generate qstrs - boards/make-pins.py: update to use tools/boardgen.py - boards/ARDUINO_PORTENTA_C33: fix incorrect I2C pins - consolidate MICROPY_PY_MACHINE_I2C option rp2 port: - cyw43_configport: use m_tracked_calloc and m_tracked_free - machine_adc: add support for external ADC channels - boards/ARDUINO_NANO_RP2040_CONNECT: add external analog pins - boards/make-pins.py: don't generate qstrs - machine_uart: fix handling of serial break condition - machine_adc: refactor channel/pin validation code - remove 1ms timeout to make idle waiting tickless - change to use TinyUSB dcd_event_handler hook - mphalport: run TinyUSB stack while waiting for CDC input/output - integrate soft_timer using the alarm pool - mpbthciport: rework HCI polling timer to use soft_timer - mpnetworkport: rework lwIP polling to use soft_timer - mphalport: optimise exit of mp_hal_delay_ms loop - cyw43_configport: implement cyw43_delay_ms as mp_hal_delay_ms - switch to locally provided math library - switch rp2 and drivers to use new event functions - main: enable SEVONPEND CPU interrupt bit - mpconfigport: enable MICROPY_PY_OS_DUPTERM_NOTIFY - add new NO_DEFAULT_PINS config options for i2c, spi, and uart - boards: add support for Pololu 3pi+ and Zumo robots - boards: add SIL_RP2040_SHIM board by Silicognition LLC - rp2_dma: introduce a new rp2.DMA class for control over DMA xfers samd port: - boards/make-pins.py: update to use tools/boardgen.py - mpconfigport: set MICROPY_USE_INTERNAL_ERRNO to 1 - machine_uart: add machine_uart_set_baudrate() function - pin_af: fix a typo in a conditional compile - switch TinyUSB to run via a scheduled task - mphalport: run TinyUSB stack while waiting for CDC input/output - switch to shared TinyUSB implementation - use unique id for USB serial number stm32 port: - boards/ARDUINO_GIGA: add QSPI fix/workaround to early init - boards/ARDUINO_GIGA: fix name of pins in board init - eth: add Ethernet support for H5 MCUs - boards/STM32H573I_DK: enable ETH and DAC peripherals - powerctrl: add support for frequency scaling with HSI on H5 MCUs - boards/make-pins.py: don't generate qstrs - boards: fix errors in pins.csv and af.csv - boards: format stm32 alternate function csv files - rename pin_obj_t to machine_pin_obj_t - boards/make-pins.py: update to use tools/boardgen.py - boards/make-pins.py: add initial support for H7 dual-pad pins - add configuration options for analog switches - add STM32H5 support for sleep mode - boards/make-pins.py: fix H7 ADC generation - boards/stm32f4x9_af.csv: fix DCMI_VSYNC - boards/stm32g474_af.csv: fix final row ADC column - boards/make-pins.py: only support ADC1-3 - boards/NUCLEO_WL55: freeze LoRa driver - mpu: enable STM32WB mpu use to support qspi flash - add optional lwip loopback support - boards/NUCLEO_F446RE: add UARTs 1, 3 and 4 - boards/NUCLEO_H563ZI: add new NUCLEO-H563ZI board definition - sdcard: add SD card support for H5 MCUs - boards/STM32H573I_DK: enable the SD card - add missing header include for debug builds - modmachine: only enable machine.I2C if hardware I2C is enabled - usbd_cdc_interface: include header to get machine_bootloader decl - machine_i2s: improve accuracy of SCK frequency - usbdev: optionally pass through vendor requests to Setup function - mboot: guard use of tx_pending with USE_USB_POLLING option - mboot: expand device descriptor to make it easier to understand - mboot: add support for Microsoft WCID teensy port: - remove the teensy port unix port: - mbedtls: enable mbedtls cert time validation - update port to use the new event functions webassembly port: no changes specific to this component/port windows port: - use the MicroPython logo as application icon - implement MICROPY_INTERNAL_WFE() macro zephyr port: no changes specific to this component/port
- Loading branch information