[question] Getting Actual Video Frame Size in LiveKit (Flutter)

[siri-sinthawat]

Good day, everyone!

I'm new to LiveKit and video conferencing in Flutter, and I’m encountering an issue regarding video frame sizes for local or remote participants. I would greatly appreciate any guidance on how to resolve this.

When attempting to retrieve the actual size of a video frame, the dimensions consistently return a width of 1280 and a height of 720, regardless of the screen orientation or the actual video resolution.

Here’s the situation:

1. I’ve set up a VideoTrackRenderer to display a participant's video:

   VideoTrack? videoTrack = participantTrack.participant.videoTrackPublications
             .where((element) => element.source == widget.type.lkVideoSourceType)
             .firstOrNull;

   VideoTrackRenderer(
       videoTrack!,
       renderMode: VideoRenderMode.auto,
       fit: RTCVideoViewObjectFit.RTCVideoViewObjectFitContain,
     )

2. I then use the following code to retrieve the video frame size:

   Map<String, dynamic>? settings = videoTrack?.mediaStreamTrack.getSettings();
   int width = settings?['width'];
   int height = settings?['height'];

3. The returned dimensions are always width: 1280 and height: 720, regardless of any changes.
4. When changing the screen orientation or resizing the window, the dimensions remain unchanged.

Additional context:

I’ve tried exploring another approach using TrackEvent to get the frame size, as it updates the frame size dynamically every second. This method is better than my previous approach because it ensures that the frame size is updated in real time. However, it introduces a few challenges:

• It updates every second, which is too frequent for my needs. I only require two scenarios:
  1. Getting the current frame size.
  2. Detecting when the frame size changes.

Here is the code I’ve used in my current approach:

for (var track in participantTrack.participant.videoTrackPublications) {
  if (track.track == null) return;
  var listener = track.track!.createListener();
  listener.on<VideoSenderStatsEvent>((event) {
    Map<String, Size> frameList = {};
    setState(() {
      event.stats.forEach((key, value) {
        if (value.frameWidth == null || value.frameHeight == null) return;
        frameList[key] = Size(
            value.frameWidth!.toDouble(), value.frameHeight!.toDouble());
      });
      frame = frameList['f'] ?? frameList['h'] ?? frameList['q'];
    });
  });
}

Desired behavior:
I expect the retrieved dimensions to accurately reflect the actual size of the video frame, dynamically updating based on screen orientation or resolution changes, but only when necessary (e.g., when a size change occurs).

Does anyone know how to handle these scenarios effectively? Is there an event or callback that allows for more controlled updates, or a way to ensure accurate frame size retrieval without excessive updates?

Thank you in advance for your help!