Upon experimenting with h264_nvenc on a RTX2080super with vMix and OBS for high-quality streaming I ended up with much better quality streams and records of OBS. CPU-driven x264 is not my primary goal but it needs some adjustmens too, but first to Hardware Encoder "NVENC".
General perceptions of vMix MP4/x264 with the same bitrates:
MP4 H264High nvenc has much better quality than FFMPEG nvenc, this follows quality-wise MP4 H264High software.
This is not my goal for producing a good stream or recording. OBS does a better job on encoding, so it is possible in a live scenario. Sure, for postproduction a "slower" denoised encode can yield far better results.
I analyzed the files and MP4 software doesn't use CABAC, this costs quality and FFMPEG software is very conservative with main or even baseline profiles, FFMPEG nvenc isn't utilising high-profile or the slow preset.
Don't get me wrong, this are great settings for older or slower computers. But a high-end machine can do smaller files or yield better quality and utilise more processing power.
Digging into vMix I exchanged both ffmpeg-files inside
C:\Program Files (x86)\vMix\streaming to the most recent version of
ffmpeg.zeranoe (static-type). This works without a flaw, but I'm still testing and NO guarantee, that it might work for everyone.
Afterwards I analyzed which settings vMix uses for recording. The ffmpegformats.xml of
C:\Program Files (x86)\vMix\streaming can be user-edited - thats great!
More to the software-encoding later, first to
hardware.
I'm testing a long time with nvenc to produce good quality and searched around forums and nvidia-docs and sadly not every possible setting is explained. My commands gain much more quality than stock-nvenc, without sacrificing too much computing power. Keep in mind, nvenc is a dedicated encoding-processor (some call it streaming-processor) on the nvidia cards, it is not linked to CUDA-performance.
I changed the video codec to h264_nvenc and audio to aac, as these are the common names used of the present ffmpegs.
My commands are: -vcodec {video_codec} -pix_fmt:v nv12 -preset:v slow -profile:v high -rc-lookahead:v 60 -spatial-aq:v 1 -temporal-aq:v 1 -b_ref_mode:v 2 -coder:v cabac -bf:v 3 -b:v {video_bitrate} -acodec {audio_codec} -b:a {audio_bitrate} -f {extension_type}
It uses preset slow of the encoder with a lookahead of 60 frames. This should match your framerate, but I'm still testing. High profile with CABAC (maybe not neccessary to mention). Spatial and temporal AQ rearange bits inside the frame for better percieved quality (psyo-visual tuning). I use 3 bframes that can be used as a reference (b_ref_mode 2).
In contrast: OBS has a preset for highquality (a bit less than slow, but it doesn't really matter), lookahead and psyo-visual tuning. My settings are more advanced. (And some guys on the internet suggest to set concurring surfaces to 64, but I haven't found any info what this really does.)
My RTX can encode 2 high-quality streams or files in 1080p60 with these settings - as consumer cards only allow 2 streams, I don't know if more are possible.
For the
software part i've created a
MP4 x264 medium high AAC Profile with libx264 video-codec and aac for audio.
Command-Settings: -pix_fmt yuv420p -vcodec {video_codec} -preset:v medium -profile:v high -b:v {video_bitrate} -acodec {audio_codec} -b:a {audio_bitrate} -f {extension_type}
This is a profile with my PC is capable of and can be used as a small giveaway for clients.
Next I added high-profile to the MP4 x264 fast option -works with less CPU usage.
Now the fun part. x264 and h264_nvenc have a quality-based encoding! You don't specify a bitrate, but a quality to reach. That way you save bits on easy and slow moving party and spend them on higher motion parts of the video. This is only usefull for recording, as the bitrate fluctuates upredictably.
Instead of a bitrate define a QP for nvenc. Start your tests with 19 or 21 for achival and much higher values for small files.
For x264 define CRF and start with 16 or 19 for HQ archival and 20 to 23 for small client-files.
I've produced much input for you and the vMix Team to understand. But this works for recoding so far!
My problem is I cannot enter custom values or edit a XML-File for the streaming part of vMix.
By the way, streaming with ffmpeg2 and nvenc is not that bad, compared to recording. It uses a high profile, if selected and uses CABAC. Half-way there. But I'd like to use Bframes and maybe add more of my settings.
How can I edit these settings? Is there a file I can edit? (just for nerdy-testing-stuff)
In the future it would be really cool to have a custom-field inside the recording and streaming-window to enter custom values.
I understand, that you want an easy-to-use and reliable product and I don't want an easy field for every possible option. Just a custom field, like handbrake shows us or OBS on some parameters.
Keep up the good work for vMix!
