CPU / Memory recommendations for BigBlueButton-liveStreaming

[glunardi]

Describe the bug
Running xvfb/chrome + pulseaudio + ffmpeg is quite resource intensive.

This ticket is partially related to #27

To Reproduce
Stream a session for 5 hours from BBB to RTMP/YouTubeLive
With a 4 cpu / 8 gb VM, the BigBlueButton-liveStreaming service (ffmpeg really) runs out of memory under one hour

Observed behavior
When running normally, the ffmpeg process seems to consume ~300MB of res mem. When the thread message queue blocks, it shoots up to 2.5GB+ res mem. Sometimes it recovers and sometimes not.

Expected behavior
The BigBlueButton-liveStreaming service (on a dedicated VM/server) should be able to live stream for the whole duration of a meeting. In our case, about 5 hours.

Tested with the latest BigBlueButton-liveStreaming on Ubuntu 20.04, project's docker image, and project's docker image (upgraded to 20.04).

Question for the team

• What do you recommend is a system configuration (virtual) to live stream using BigBlueButton-liveStreaming to YouTube Live for 5 hours?

[mtsonline]

...we had no issues with 8cpu and users reported 6cpu are working too.
do you think removing alsa will lower the requirements? ...
for now I would not reccommend 4 cpu as to many users had problems on such systems.

[glunardi]

...we had no issues with 8cpu and users reported 6cpu are working too.
do you think removing alsa will lower the requirements? ...
for now I would not reccommend 4 cpu as to many users had problems on such systems.

Thank you for sharing this. Much appreciated. It seems ffmpeg and chrome are actively competing for CPU cycle. Whenever ffmpeg is starved it really struggles to recover.

With some additional "tweaks" we were able to live RTMP stream overnight for 6 hours+ on a 4CPU VM to YouTube. It could have been just a fluke. Ran a very quick test to Facebook Live with the same pipeline.

This is the ffmpeg pipeline we ran:

audio_options = '-f pulse -i default -ac 2 -c:a aac -b:a 160k -ar 48000'
video_options = '-c:v libx264 -x264-params "nal-hrd=vbr" -profile:v high -level:v 4.2 -vf format=yuv420p -b:v 4000k -maxrate 4000k -minrate 2000k -bufsize 8000k -g 60 -preset ultrafast -tune zerolatency'
ffmpeg_stream = 'ffmpeg -loglevel verbose -thread_queue_size 4096 -f x11grab -probesize 10M -draw_mouse 0 -framerate 30 -vsync 1 -s 1920x1080  -i :%d -thread_queue_size 4096 %s -threads 0 %s -f flv -flvflags no_duration_filesize "%s"' % ( 122, audio_options, video_options, args.target)

Changes are:

• Audio rate to 48K was to prevent resampling (still needs to convert back from pcm to aac)
• Enabling VBR across the pipeline (with a vbr key used on YouTube Live)
• Added framerate to eliminate loads of frame duration error messages
• Added vsync to help with frame management
• Added probesize to try to address x11grab rate estimation error on startup

Test scenario is:

• 5 real clients connected
• 5 webcams (480p)
• 5 two-way audio stream
• 1 external video playback

Some notes:

• YouTube is happy to accept 48K audio (even though not supposed to be supported in flv/aac)
• Have not managed to just send PCM to YouTube (would be nice)
• Video quality is "nice" with these parameters

Question:
Tempted to reduce streaming resolution from 1920x1080p to 1280x720. Would prefer staying with fullHD but it is a lot more processing for xvfb, chrome, and ffmpeg. Is that something you ever had to do?

Will be running more tests today and will be happy to report back.

[mtsonline]

hi and thanks for your contribution, which is highly appreciated.

• YouTube is happy to accept 48K audio (even though not supposed to be supported in flv/aac)
what would be the expected rate though?

please always test facebook streaming too, as users need both of them ... best would be to additionally check restream.io streaming as well. If all three are working everyone should be fine in the end :-)

as always, if you make changes - it would be good to have the options as configurable settings with your best practice values as default.

same for the video resolution ... there where requests to have full hd and hd (720p) as configurable options. I'ld even add at least one more.

we'll be happy to merge your prs if the streaming platforms are working and the streaming is working stable ... the more user configuration options the better ;-)

cheers and thanks!

[markvdb]

Testing this right now. On bare metal Intel hardware, one can usually reliably reduce the cpu usage using vaapi/hw acceleration.

https://github.com/FOSDEM/infrastructure/blob/master/ansible/playbooks/roles/video-voctop/files/scripts/sink-output.sh is an example of an encoding pipeline we use at https://fosdem.org . The hardware accelleraton lowers cpu usage very substantially.

[mtsonline]

would it work on any other hardware too with just loosing the exceleration or would it give an error on other hardware?
Could you provide a pullrequest for this if you think it is working stable within liveStreaming?
If it does only work on selected hardware, it would be great to make it optional so one can turn on hardware exceleration in the compose.yml ...
cheers

[markvdb]

Vaapi hardware acceleration as an option is the only realistic target. We have only got experience with Intel's on-die hw accelleration core. From a quick look, it doesn't seem as if amd processors have such a thing... Op di 10 nov. 2020 om 09:36 schreef mtsonline <[email protected]>:

…

would it work on any other hardware too with just loosing the exceleration or would it give an error on other hardware? Could you provide a pullrequest for this if you think it is working stable within liveStreaming? If it does only work on selected hardware, it would be great to make it optional so one can turn on hardware exceleration in the compose.yml ... cheers — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

-- Mark Van den Borre Hogestraat 16 3000 Leuven, België +32 486 961726