Merge branch 'main' into feat/platform-view-for-ios

CHANGELOG.md

@@ -1,5 +1,24 @@
 # CHANGELOG
 
+## 2.2.1
+
+* fix: fix crash for windows
+* feat: bump version for flutter-webrtc.
+* fix: ratchet on a single frame until ratchetWindowSize (#544)
+* fix: fix mediaStreamTrack.getSettings() on Flutter Web.
+
+## 2.2.0
+
+* feat: add Transcription Event. (#531)
+* feat: Expose Participant.Kind. (#532)
+* fix: ignore unable to parse frames completely (#530)
+
+## 2.1.6
+
+* Framecryptor decrypting fixes (#520)
+* feat: add voiceIsolation support (#523)
+* fix: audio session control for ios.
+
 ## 2.1.5
 
 fix: audio devcie ids consistency (#513)

README.md

@@ -13,11 +13,8 @@
 # LiveKit Flutter SDK
 
 <!--BEGIN_DESCRIPTION-->
-Use this SDK to add real-time video, audio and data features to your Flutter app. By connecting to a self- or cloud-hosted <a href="https://livekit.io/">LiveKit</a> server, you can quickly build applications like interactive live streaming or video calls with just a few lines of code.
-
-## NOTE
 
-Version 2 of the <platform> SDK contains a small set of breaking changes. Read the [migration guide](https://docs.livekit.io/guides/migrate-from-v1/) for a detailed overview of what has changed.
+Use this SDK to add real-time video, audio and data features to your Flutter app. By connecting to a self- or cloud-hosted <a href="https://livekit.io/">LiveKit</a> server, you can quickly build applications like interactive live streaming or video calls with just a few lines of code.
 
 <!--END_DESCRIPTION-->
 
@@ -27,25 +24,20 @@ This package is published to pub.dev as [livekit_client](https://pub.dev/package
 
 More Docs and guides are available at [https://docs.livekit.io](https://docs.livekit.io)
 
-## Current supported features
-
-| Feature | Subscribe/Publish | Simulcast | Background audio | Screen sharing | End to End Encryption | Multi Codec Simulcast |
-| :-----: | :---------------: | :-------: | :--------------: | :------------: | :-------------------: | :-------------------: |
-|   iOS   |        🟢         |    🟢     |        🟢        |       🟢       |       🟢               |          🟢          |
-| Android |        🟢         |    🟢     |        🟢        |       🟢       |       🟢               |          🟢          |
-|   Mac   |        🟢         |    🟢     |        🟢        |       🟢       |       🟢               |          🟢          |
-| Windows |        🟢         |    🟢     |        🟢        |       🟢       |       🟢               |          🟢          |
-| Linux   |        🟢         |    🟢     |        🟢        |       🟢       |       🟢               |          🟢          |
-
-🟢 = Available
+## Supported platforms
 
-🟡 = Coming soon (Work in progress)
+LiveKit client SDK for Flutter is designed to work across all platforms supported by Flutter:
 
-🔴 = Not currently available (Possibly in the future)
+- Android
+- iOS
+- Web
+- macOS
+- Windows
+- Linux
 
 ## Example app
 
-We built a multi-user conferencing app as an example in the [example/](example/) folder. You can join the same room from any supported LiveKit clients.
+We built a multi-user conferencing app as an example in the [example/](example/) folder. LiveKit is compatible cross-platform: you could join the same room using any of our supported realtime SDKs.
 
 Online demo: https://livekit.github.io/client-sdk-flutter/
 
@@ -160,7 +152,7 @@ void main() async {
 
 #### Audio Modes
 
-By default, we use the `communication` audio mode on Android which works best for two-way voice communication. 
+By default, we use the `communication` audio mode on Android which works best for two-way voice communication.
 
 If your app is media playback oriented and does not need the use of the device's microphone, you can use the `media`
 audio mode which will provide better audio quality.
@@ -226,7 +218,9 @@ room.localParticipant.setScreenShareEnabled(true);
 
 #### Android
 
-On Android, you would have to define a foreground service in your AndroidManifest.xml.
+On Android, you will have to use a [media projection foreground service](https://developer.android.com/develop/background-work/services/fg-service-types#media-projection).
+
+In our example, we use the `flutter_background` package to handle this. In the app's AndroidManifest.xml file, declare the service with the appropriate types and permissions as following:
 
 ```xml title="AndroidManifest.xml"
 <manifest xmlns:android="http://schemas.android.com/apk/res/android">
@@ -244,6 +238,8 @@ On Android, you would have to define a foreground service in your AndroidManifes
 </manifest>
 ```
 
+Before starting the background service and enabling screen share, you **must** call `Helper.requestCapturePermission()` from `flutter_webrtc`, and only proceed if it returns true.
+
 #### iOS
 
 On iOS, a broadcast extension is needed in order to capture screen content from
@@ -494,11 +490,13 @@ Apache License 2.0
 A huge thank you to [flutter-webrtc](https://github.com/flutter-webrtc/flutter-webrtc) for making it possible to use WebRTC in Flutter.
 
 <!--BEGIN_REPO_NAV-->
+
 <br/><table>
+
 <thead><tr><th colspan="2">LiveKit Ecosystem</th></tr></thead>
 <tbody>
-<tr><td>Real-time SDKs</td><td><a href="https://github.com/livekit/components-js">React Components</a> · <a href="https://github.com/livekit/client-sdk-js">JavaScript</a> · <a href="https://github.com/livekit/client-sdk-swift">iOS/macOS</a> · <a href="https://github.com/livekit/client-sdk-android">Android</a> · <b>Flutter</b> · <a href="https://github.com/livekit/client-sdk-react-native">React Native</a> · <a href="https://github.com/livekit/client-sdk-rust">Rust</a> · <a href="https://github.com/livekit/client-sdk-python">Python</a> · <a href="https://github.com/livekit/client-sdk-unity-web">Unity (web)</a> · <a href="https://github.com/livekit/client-sdk-unity">Unity (beta)</a></td></tr><tr></tr>
-<tr><td>Server APIs</td><td><a href="https://github.com/livekit/server-sdk-js">Node.js</a> · <a href="https://github.com/livekit/server-sdk-go">Golang</a> · <a href="https://github.com/livekit/server-sdk-ruby">Ruby</a> · <a href="https://github.com/livekit/server-sdk-kotlin">Java/Kotlin</a> · <a href="https://github.com/livekit/client-sdk-python">Python</a> · <a href="https://github.com/livekit/client-sdk-rust">Rust</a> · <a href="https://github.com/agence104/livekit-server-sdk-php">PHP (community)</a></td></tr><tr></tr>
+<tr><td>Real-time SDKs</td><td><a href="https://github.com/livekit/components-js">React Components</a> · <a href="https://github.com/livekit/client-sdk-js">Browser</a> · <a href="https://github.com/livekit/client-sdk-swift">iOS/macOS</a> · <a href="https://github.com/livekit/client-sdk-android">Android</a> · <b>Flutter</b> · <a href="https://github.com/livekit/client-sdk-react-native">React Native</a> · <a href="https://github.com/livekit/rust-sdks">Rust</a> · <a href="https://github.com/livekit/node-sdks">Node.js</a> · <a href="https://github.com/livekit/python-sdks">Python</a> · <a href="https://github.com/livekit/client-sdk-unity-web">Unity (web)</a> · <a href="https://github.com/livekit/client-sdk-unity">Unity (beta)</a></td></tr><tr></tr>
+<tr><td>Server APIs</td><td><a href="https://github.com/livekit/node-sdks">Node.js</a> · <a href="https://github.com/livekit/server-sdk-go">Golang</a> · <a href="https://github.com/livekit/server-sdk-ruby">Ruby</a> · <a href="https://github.com/livekit/server-sdk-kotlin">Java/Kotlin</a> · <a href="https://github.com/livekit/python-sdks">Python</a> · <a href="https://github.com/livekit/rust-sdks">Rust</a> · <a href="https://github.com/agence104/livekit-server-sdk-php">PHP (community)</a></td></tr><tr></tr>
 <tr><td>Agents Frameworks</td><td><a href="https://github.com/livekit/agents">Python</a> · <a href="https://github.com/livekit/agent-playground">Playground</a></td></tr><tr></tr>
 <tr><td>Services</td><td><a href="https://github.com/livekit/livekit">Livekit server</a> · <a href="https://github.com/livekit/egress">Egress</a> · <a href="https://github.com/livekit/ingress">Ingress</a> · <a href="https://github.com/livekit/sip">SIP</a></td></tr><tr></tr>
 <tr><td>Resources</td><td><a href="https://docs.livekit.io">Docs</a> · <a href="https://github.com/livekit-examples">Example apps</a> · <a href="https://livekit.io/cloud">Cloud</a> · <a href="https://docs.livekit.io/oss/deployment">Self-hosting</a> · <a href="https://github.com/livekit/livekit-cli">CLI</a></td></tr>

analysis_options.yaml

@@ -27,6 +27,8 @@
 include: package:flutter_lints/flutter.yaml
 
 analyzer:
+  errors:
+    constant_identifier_names: ignore
   #
   # Enforce stricter type-checking
   # https://dart.dev/guides/language/analysis-options#enabling-additional-type-checks
@@ -41,7 +43,7 @@ analyzer:
     - '**/*.pbenum.dart'
     - '**/*.pbjson.dart'
     - '**/*.pbserver.dart'
-    - 'web/*.dart'
+    # - 'web/*.dart'
 
 linter:
   rules:

example/android/app/build.gradle

@@ -41,7 +41,7 @@ android {
         // TODO: Specify your own unique Application ID (https://developer.android.com/studio/build/application-id.html).
         applicationId "io.livekit.example"
         minSdkVersion 21
-        targetSdkVersion 31
+        targetSdkVersion 34
         versionCode flutterVersionCode.toInteger()
         versionName flutterVersionName
     }

example/ios/Podfile

@@ -1,5 +1,5 @@
 # Uncomment this line to define a global platform for your project
-platform :ios, '12.1'
+platform :ios, '13.0'
 
 # CocoaPods analytics sends network stats synchronously affecting flutter build latency.
 ENV['COCOAPODS_DISABLE_STATS'] = 'true'

example/lib/pages/room.dart

@@ -45,7 +45,7 @@ class _RoomPageState extends State<RoomPage> {
       }
     });
 
-    if (lkPlatformIsMobile()) {
+    if (lkPlatformIs(PlatformType.android)) {
       Hardware.instance.setSpeakerphoneOn(true);
     }
 

example/lib/widgets/controls.dart

@@ -35,7 +35,7 @@ class _ControlsWidgetState extends State<ControlsWidget> {
 
   StreamSubscription? _subscription;
 
-  bool _speakerphoneOn = false;
+  bool _speakerphoneOn = Hardware.instance.preferSpeakerOutput;
 
   @override
   void initState() {
@@ -46,7 +46,6 @@ class _ControlsWidgetState extends State<ControlsWidget> {
       _loadDevices(devices);
     });
     Hardware.instance.enumerateDevices().then(_loadDevices);
-    _speakerphoneOn = Hardware.instance.speakerOn ?? false;
   }
 
   @override
@@ -157,6 +156,11 @@ class _ControlsWidgetState extends State<ControlsWidget> {
     }
     if (lkPlatformIs(PlatformType.android)) {
       // Android specific
+      bool hasCapturePermission = await Helper.requestCapturePermission();
+      if (!hasCapturePermission) {
+        return;
+      }
+
       requestBackgroundPermission([bool isRetry = false]) async {
         // Required for android screenshare.
         try {

example/lib/widgets/participant.dart

@@ -84,17 +84,26 @@ abstract class _ParticipantWidgetState<T extends ParticipantWidget>
   TrackPublication? get videoPublication;
   TrackPublication? get audioPublication;
   bool get isScreenShare => widget.type == ParticipantTrackType.kScreenShare;
+  EventsListener<ParticipantEvent>? _listener;
 
   @override
   void initState() {
     super.initState();
+    _listener = widget.participant.createListener();
+    _listener?.on<TranscriptionEvent>((e) {
+      for (var seg in e.segments) {
+        print('Transcription: ${seg.text} ${seg.isFinal}');
+      }
+    });
+
     widget.participant.addListener(_onParticipantChanged);
     _onParticipantChanged();
   }
 
   @override
   void dispose() {
     widget.participant.removeListener(_onParticipantChanged);
+    _listener?.dispose();
     super.dispose();
   }
 

ios/livekit_client.podspec

@@ -1,6 +1,6 @@
 Pod::Spec.new do |s|
   s.name                = 'livekit_client'
-  s.version             = '2.1.5'
+  s.version             = '2.2.1'
   s.summary             = 'Open source platform for real-time audio and video.'
   s.description         = 'Open source platform for real-time audio and video.'
   s.homepage            = 'https://livekit.io/'
@@ -9,12 +9,12 @@ Pod::Spec.new do |s|
   s.source              = { :path => '.' }
   s.source_files        = 'Classes/**/*'
   s.public_header_files = 'Classes/**/*.h'
-  s.platform            = :ios, '12.1'
+  s.platform            = :ios, '13.0'
   # Flutter.framework does not contain a i386 slice.
   s.pod_target_xcconfig = { 'DEFINES_MODULE' => 'YES', 'EXCLUDED_ARCHS[sdk=iphonesimulator*]' => 'i386' }
   s.swift_version       = '5.0'
   s.static_framework    = true
 
   s.dependency 'Flutter'
-  s.dependency 'WebRTC-SDK', '114.5735.10'
+  s.dependency 'WebRTC-SDK', '125.6422.04'
 end

lib/src/core/engine.dart

@@ -616,6 +616,18 @@ class Engine extends Disposable with EventsEmittable<EngineEvent> {
         packet: dp.user,
         kind: dp.kind,
       ));
+    } else if (dp.whichValue() == lk_models.DataPacket_Value.transcription) {
+      // Transcription packet
+      events.emit(EngineTranscriptionReceivedEvent(
+        transcription: dp.transcription,
+      ));
+    } else if (dp.whichValue() == lk_models.DataPacket_Value.sipDtmf) {
+      // SIP DTMF packet
+      events.emit(EngineSipDtmfReceivedEvent(
+        dtmf: dp.sipDtmf,
+      ));
+    } else {
+      logger.warning('Unknown data packet type: ${dp.whichValue()}');
     }
   }
 

lib/src/core/room.dart

@@ -436,6 +436,7 @@ class Room extends DisposableChangeNotifier with EventsEmittable<RoomEvent> {
     ..on<EngineActiveSpeakersUpdateEvent>(
         (event) => _onEngineActiveSpeakersUpdateEvent(event.speakers))
     ..on<EngineDataPacketReceivedEvent>(_onDataMessageEvent)
+    ..on<EngineTranscriptionReceivedEvent>(_onTranscriptionEvent)
     ..on<AudioPlaybackStarted>((event) {
       _handleAudioPlaybackStarted();
     })
@@ -683,6 +684,37 @@ class Room extends DisposableChangeNotifier with EventsEmittable<RoomEvent> {
     }
   }
 
+  void _onTranscriptionEvent(EngineTranscriptionReceivedEvent event) {
+    final participant = getParticipantByIdentity(
+        event.transcription.transcribedParticipantIdentity);
+    if (participant == null) {
+      return;
+    }
+
+    final publication =
+        participant.getTrackPublicationBySid(event.transcription.trackId);
+
+    var segments = event.transcription.segments.map((e) {
+      return TranscriptionSegment(
+        text: e.text,
+        id: e.id,
+        startTime: DateTime.fromMillisecondsSinceEpoch(e.startTime.toInt()),
+        endTime: DateTime.fromMillisecondsSinceEpoch(e.endTime.toInt()),
+        isFinal: e.final_5,
+        language: e.language,
+      );
+    }).toList();
+
+    final transcription = TranscriptionEvent(
+      participant: participant,
+      publication: publication,
+      segments: segments,
+    );
+
+    participant.events.emit(transcription);
+    events.emit(transcription);
+  }
+
   void _onDataMessageEvent(EngineDataPacketReceivedEvent dataPacketEvent) {
     // participant may be null if data is sent from Server-API
     final senderSid = dataPacketEvent.packet.participantSid;

lib/src/core/signal_client.dart

@@ -494,18 +494,6 @@ extension SignalClientRequests on SignalClient {
         subscription: subscription,
       ));
 
-  @internal
-  void sendUpdateVideoLayers(
-    String trackSid,
-    Iterable<lk_models.VideoLayer> layers,
-  ) =>
-      _sendRequest(lk_rtc.SignalRequest(
-        updateLayers: lk_rtc.UpdateVideoLayers(
-          trackSid: trackSid,
-          layers: layers,
-        ),
-      ));
-
   @internal
   void sendUpdateSubscriptionPermissions({
     required bool allParticipants,

lib/src/e2ee/key_provider.dart

@@ -50,10 +50,11 @@ abstract class KeyProvider {
 }
 
 class BaseKeyProvider implements KeyProvider {
+  final Map<String, int> _latestSetIndex = {};
   final Map<String, Map<int, Uint8List>> _keys = {};
 
   int getLatestIndex(String participantId) {
-    return _keys[participantId]?.keys.last ?? 0;
+    return _latestSetIndex[participantId] ?? 0;
   }
 
   Uint8List? _sharedKey;
@@ -152,6 +153,7 @@ class BaseKeyProvider implements KeyProvider {
     logger.info(
         '_setKey for ${keyInfo.participantId}, idx: ${keyInfo.keyIndex}, key: ${base64Encode(keyInfo.key)}');
     _keys[keyInfo.participantId]![keyInfo.keyIndex] = keyInfo.key;
+    _latestSetIndex[keyInfo.participantId] = keyInfo.keyIndex;
     await _keyProvider.setKey(
       participantId: keyInfo.participantId,
       index: keyInfo.keyIndex,

lib/src/events.dart

@@ -433,6 +433,35 @@ class ParticipantPermissionsUpdatedEvent with RoomEvent, ParticipantEvent {
       '(participant: ${participant}, permissions: ${permissions})';
 }
 
+class TranscriptionSegment {
+  final String id;
+  final String text;
+  final DateTime startTime;
+  final DateTime endTime;
+  final bool isFinal;
+  final String language;
+  const TranscriptionSegment({
+    required this.id,
+    required this.text,
+    required this.startTime,
+    required this.endTime,
+    required this.isFinal,
+    required this.language,
+  });
+}
+
+/// Transcription event received from the server.
+class TranscriptionEvent with RoomEvent, ParticipantEvent {
+  final Participant participant;
+  final TrackPublication<Track>? publication;
+  final List<TranscriptionSegment> segments;
+  const TranscriptionEvent({
+    required this.participant,
+    required this.publication,
+    required this.segments,
+  });
+}
+
 class ParticipantNameUpdatedEvent with RoomEvent, ParticipantEvent {
   final Participant participant;
   final String name;

lib/src/extensions.dart

@@ -251,3 +251,13 @@ extension DisconnectReasonExt on lk_models.DisconnectReason {
         lk_models.DisconnectReason.JOIN_FAILURE: DisconnectReason.joinFailure,
       }[this]!;
 }
+
+extension ParticipantTypeExt on lk_models.ParticipantInfo_Kind {
+  ParticipantKind toLKType() => {
+        lk_models.ParticipantInfo_Kind.STANDARD: ParticipantKind.STANDARD,
+        lk_models.ParticipantInfo_Kind.INGRESS: ParticipantKind.INGRESS,
+        lk_models.ParticipantInfo_Kind.EGRESS: ParticipantKind.EGRESS,
+        lk_models.ParticipantInfo_Kind.SIP: ParticipantKind.SIP,
+        lk_models.ParticipantInfo_Kind.AGENT: ParticipantKind.AGENT,
+      }[this]!;
+}