Quickly generating high quality scientific videos with Python
Recently I have had to pre-generate some videos or animations for scientific purposes from Python, and I have been frustrated by how slow the video generation process is and often unexpectedly getting a poor quality video. And because of how much time it takes, I never quite had the time and patience to redo everything and try different magical options on FFmpeg. Therefore I decided to take my time off to delve deeper and find out what is the best way to generate these videos for my needs.
My needs are:
- The videos will be a combination of a “natural scene” and simple, sharp line plots, and must retain high fidelity for the line plot (noticeable JPEG artifacts look terrible and unprofessional on plots, in my opinion)
- The videos must be of high resolution (we will stick to 720p for now) and high frame rate (60 fps)
- File size must be reasonable for me to transfer it easily over the Internet — specifically I am looking at a video around one minute (3600 frames) would take less than 100 MB, or around 8 MB for 300 frames.
- I want the video to be generated quickly, at 1x speed or faster — I am impatient and I don’t really care about saving every byte, as long as it can be reasonably stored in a cloud storage.
Preparation
Generate 300 720p frames for benchmarking. On the left side of each frame we will take a “natural” scene from Big Buck Bunny, and the background will be a moving plot.
import moviepy.editor as mpy
with mpy.VideoFileClip("https://archive.org/download/BigBuckBunny_310/big_buck_bunny_640.mp4") as clip:
scene_frames = [clip.get_frame(i) for i in range(300)]from moviepy.video.io.bindings import mplfig_to_npimage
import matplotlib.pyplot as plt
import numpy as np
plot_frames = []
fig, ax = plt.subplots(figsize=(1280/288, 720/288), dpi=288)
for i in range(300):
ax.clear()
ax.plot(np.linspace(0, 300, 5000), np.cos(np.linspace(0, 300, 5000)))
ax.plot(i, np.cos(i), ".", markersize=20)
ax.set(xlim=[i-50, i+10], xlabel="X Label", ylabel="Y Label")
plot_frame = mplfig_to_npimage(fig)
plot_frames.append(plot_frame)
frames = [i.copy() for i in plot_frames]
for i in range(300):
inset = scene_frames[i]
frames[i][0:inset.shape[0], 0:inset.shape[1], ...] = inset
plt.imshow(frames[60])<matplotlib.image.AxesImage at 0x26a2eac43d0>
Benchmark
import moviepy.editor as mpy
import timeit
from pathlib import Path
from IPython.display import HTML, display
from base64 import b64encode
def show_video(filename):
with open(filename, "rb") as f:
video_bytes = f.read()
if filename.lower().endswith("webm"):
format = "webm"
else:
format = "mp4"
data_url = f"data:video/{format};base64," + b64encode(video_bytes).decode("ascii")
display(HTML("""
<p>%s</p>
<video width="800" controls>
<source src="%s" type="video/%s">
</video>
""" % (filename, data_url, format)))
clip = mpy.ImageSequenceClip(frames, fps=60)The first part of the benchmark is performed in Google Colab
!cat /proc/cpuinfoprocessor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU @ 2.20GHz
stepping : 0
microcode : 0x1
cpu MHz : 2199.998
cache size : 56320 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat md_clear arch_capabilities
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa
bogomips : 4399.99
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU @ 2.20GHz
stepping : 0
microcode : 0x1
cpu MHz : 2199.998
cache size : 56320 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat md_clear arch_capabilities
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa
bogomips : 4399.99
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
Software GIF
GIF retain full fidelity and is widely compatible with browsers, but has very terrible file sizes.
print("Duration:", timeit.timeit(lambda : clip.write_gif("movie.gif", progress_bar=False), number=1))
print("Size:", Path("movie.gif").lstat().st_size)[MoviePy] Building file movie.gif with imageio
Duration: 98.6067062279999
Size: 101060951
It produced a whopping 101 MB while taking 99 seconds, so we will move right along.
Software H264 with libx264
Here we will test using the software CPU libx264 encoder at 3 quality settings using the CRF parameter. For now, we want the fastest speed so we use the ultrafast preset. Lower represents better quality. 0 is lossless, while 17 is "visually lossless".
for crf in (0, 1, 17, 23, 28):
filename = f"movie_libx264_ultrafast_crf{crf}.mp4"
print("Duration:", timeit.timeit(lambda : clip.write_videofile(
filename,
codec='libx264', preset="ultrafast",
progress_bar=False, ffmpeg_params=["-crf", str(crf)]
), number=1))
print("Size:", Path(filename).lstat().st_size)
print("-" * 80)[MoviePy] >>>> Building video movie_libx264_ultrafast_crf0.mp4
[MoviePy] Writing video movie_libx264_ultrafast_crf0.mp4
[MoviePy] Done.
[MoviePy] >>>> Video ready: movie_libx264_ultrafast_crf0.mp4
Duration: 4.072924522999529
Size: 46454755
--------------------------------------------------------------------------------
[MoviePy] >>>> Building video movie_libx264_ultrafast_crf1.mp4
[MoviePy] Writing video movie_libx264_ultrafast_crf1.mp4
[MoviePy] Done.
[MoviePy] >>>> Video ready: movie_libx264_ultrafast_crf1.mp4
Duration: 3.9026636369999324
Size: 33519732
--------------------------------------------------------------------------------
[MoviePy] >>>> Building video movie_libx264_ultrafast_crf17.mp4
[MoviePy] Writing video movie_libx264_ultrafast_crf17.mp4
[MoviePy] Done.
[MoviePy] >>>> Video ready: movie_libx264_ultrafast_crf17.mp4
Duration: 3.4450486129990168
Size: 8888630
--------------------------------------------------------------------------------
[MoviePy] >>>> Building video movie_libx264_ultrafast_crf23.mp4
[MoviePy] Writing video movie_libx264_ultrafast_crf23.mp4
[MoviePy] Done.
[MoviePy] >>>> Video ready: movie_libx264_ultrafast_crf23.mp4
Duration: 3.245059176999348
Size: 4837451
--------------------------------------------------------------------------------
[MoviePy] >>>> Building video movie_libx264_ultrafast_crf28.mp4
[MoviePy] Writing video movie_libx264_ultrafast_crf28.mp4
[MoviePy] Done.
[MoviePy] >>>> Video ready: movie_libx264_ultrafast_crf28.mp4
Duration: 3.172233902998414
Size: 2891056
--------------------------------------------------------------------------------
We can see that all CRF settings were able to encode at about 80 to 100 fps, with higher CRF being faster. How are the quality?
show_video("movie_libx264_ultrafast_crf17.mp4")show_video("movie_libx264_ultrafast_crf23.mp4")show_video("movie_libx264_ultrafast_crf28.mp4")
17 provides excellent quality to my eyes. 23 is okay if I need the extra saving in space, but I can see compression in the bunny. 28 noticeably compromise the quality of the bunny scene. With CRF at 17 the file size is about 9 MB for this 300 frame clip and encoding at 87 fps — sounds like a good choice here.
Software H265 with libx265
Now we can swap H264 for H265 and see its performance. For CRF we will here use 22, 28, and 32 which roughly corresponds to the H264 CRF above.
for crf in (22, 28, 32):
filename = f"movie_libx265_ultrafast_crf{crf}.mp4"
print("Duration:", timeit.timeit(lambda : clip.write_videofile(
filename,
codec='libx265', preset="ultrafast",
progress_bar=False, ffmpeg_params=["-crf", str(crf)]
), number=1))
print("Size:", Path(filename).lstat().st_size)
print("-" * 80)[MoviePy] >>>> Building video movie_libx265_ultrafast_crf22.mp4
[MoviePy] Writing video movie_libx265_ultrafast_crf22.mp4
[MoviePy] Done.
[MoviePy] >>>> Video ready: movie_libx265_ultrafast_crf22.mp4
Duration: 20.144980058999863
Size: 6320198
--------------------------------------------------------------------------------
[MoviePy] >>>> Building video movie_libx265_ultrafast_crf28.mp4
[MoviePy] Writing video movie_libx265_ultrafast_crf28.mp4
[MoviePy] Done.
[MoviePy] >>>> Video ready: movie_libx265_ultrafast_crf28.mp4
Duration: 16.899099792000925
Size: 2824476
--------------------------------------------------------------------------------
[MoviePy] >>>> Building video movie_libx265_ultrafast_crf32.mp4
[MoviePy] Writing video movie_libx265_ultrafast_crf32.mp4
[MoviePy] Done.
[MoviePy] >>>> Video ready: movie_libx265_ultrafast_crf32.mp4
Duration: 14.924015721000615
Size: 1580668
--------------------------------------------------------------------------------
So libx265 provides roughly a 25% to 50% reduction in file size compared to libx264 (both at ultrafast), but also takes 5 times as long to encode. Considering the potential compatibility issues with H265, for me I would say H264 is still the go to option here.
Software WebM with libvpx
This is also known as VP8. Here is the refernce. The -deadline realtime option comes from the VP9 guide and I am not sure if it has an effect here or not.
for crf in (10, ):
filename = f"movie_libvpx_realtime_crf{crf}.webm"
print("Duration:", timeit.timeit(lambda : clip.write_videofile(
filename,
codec='libvpx',
progress_bar=False, ffmpeg_params=["-crf", str(crf), "-deadline", "realtime"]
), number=1))
print("Size:", Path(filename).lstat().st_size)
print("-" * 80)[MoviePy] >>>> Building video movie_libvpx_realtime_crf10.webm
[MoviePy] Writing video movie_libvpx_realtime_crf10.webm
[MoviePy] Done.
[MoviePy] >>>> Video ready: movie_libvpx_realtime_crf10.webm
Duration: 6.72509285399974
Size: 1448422
--------------------------------------------------------------------------------
I only tried the default recommended CRF of 10 here. The quality is not as good as I wanted — I felt it was similar to CRF 28 on libx264. It is only half the size of CRF 28 on libx264 though, but takes twice the amount of time to encode. I am not sure if the realtime option is applied correctly here — but when I tried just --rt it gave me an error.
Software WebM with libvpx-vp9
Here is the encode guide for VP9 with FFmpeg.
for crf in (35, ):
filename = f"movie_libvpx-vp9_realtime_crf{crf}.webm"
print("Duration:", timeit.timeit(lambda : clip.write_videofile(
filename,
codec='libvpx-vp9',
progress_bar=False, ffmpeg_params=["-crf", str(crf), "-deadline", "realtime"]
), number=1))
print("Size:", Path(filename).lstat().st_size)
print("-" * 80)[MoviePy] >>>> Building video movie_libvpx-vp9_realtime_crf35.webm
[MoviePy] Writing video movie_libvpx-vp9_realtime_crf35.webm
[MoviePy] Done.
[MoviePy] >>>> Video ready: movie_libvpx-vp9_realtime_crf35.webm
Duration: 178.94476520099852
Size: 6949732
--------------------------------------------------------------------------------
This was obviously taking a long time even at the highest recommended CRF value of 35. The file size is also not competitive here, though the quality is quite good. I am not sure what is wrong here… Again, I am also not sure if the -deadline realtime option is correctly applied here.
Software encoders wrap up
I was a bit surprised that the good old H264 seems to be an excellent choice here. The more modern options do give me better file sizes, but H264 does not fare bad here at all. Meanwhile, encoding speed has always been a pain point in my workflow, so I was really hoping to explore an option that gives me 1x speed. And with that consideration, H264 is the only winner here.
The other more exotic options that I thought would be promising is the lossless APNG and MPNG, but the documentation on them with FFmpeg is sparse. I wasn’t able to get APNG working at all and with MPNG (codec="png") I was getting an obviously blurry output defeating the purpose of PNG in the first place. Plus I have discovered that CRF=0 on H264 works quite well too, so for true lossless that seems to be the better option. (though it seems CRF=0 doesn't open in browsers)
Hardware H264 with h264_qsv
But that’s not all of it yet. Hardware acceleration is all the rage right now and much of our chip has dedicated facilities to encode videos. And since we just found how good H264 can be which is also often supported by these hardware accelerators, I thought it would be worthwhile to test them as well. My impression was that they tend to give very good realtime encoding speed, but the file size can be huge (so streamers encode them again later I guess?). But may with some correct settings I can get them to fit my needs.
For these, I am using the hardware that I have access to on a desktop computer, since they are not available on colab.
First up, QSV is the hardware encoder on Intel CPUs. It is also known as QuickSync. The fastest preset is veryfast and quality is controlled through -global_quality from 1 to 51.
for crf in (10, 20, 30, 40):
filename = f"movie_h264_qsv_veryfast_{crf}.mp4"
print(filename)
print("Duration:", timeit.timeit(lambda : clip.write_videofile(
filename, preset="veryfast",
codec='h264_qsv', ffmpeg_params=["-global_quality", str(crf)],
logger=None
), number=1))
print("Size:", Path(filename).lstat().st_size)
print("-" * 80)movie_h264_qsv_veryfast_10.mp4
Duration: 2.7140514000000167
Size: 13985524
--------------------------------------------------------------------------------
movie_h264_qsv_veryfast_20.mp4
Duration: 2.359695600000009
Size: 5515647
--------------------------------------------------------------------------------
movie_h264_qsv_veryfast_30.mp4
Duration: 2.703982500000052
Size: 2199359
--------------------------------------------------------------------------------
movie_h264_qsv_veryfast_40.mp4
Duration: 2.801664599999981
Size: 1047328
--------------------------------------------------------------------------------
That was fast at about 110 fps of encoding. In terms of quality I found that 10 and 20 are both very good. Considering file size, I would say 20 is the sweet spot for me.
Hardware H264 with h264_nvenc
nvenc is the hardware encoder by NVIDIA. I am testing this on a 1070. The useful command here is ffmpeg -h encoder=h264_nvenc to see the different options.
filename = f"movie_h264_nvenc_fast.mp4"
print(filename)
print("Duration:", timeit.timeit(lambda : clip.write_videofile(
filename, preset="fast",
codec='h264_nvenc',
logger=None
), number=1))
print("Size:", Path(filename).lstat().st_size)
print("-" * 80)movie_h264_nvenc_fast.mp4
Duration: 1.272217499999897
Size: 1504943
--------------------------------------------------------------------------------
Very very fast, but low image quality. Umm it looks like the default bitrate is low? How do I improve the quality?
filename = f"movie_h264_nvenc_fast.mp4"
print(filename)
print("Duration:", timeit.timeit(lambda : clip.write_videofile(
filename, preset="fast", bitrate="8M",
codec='h264_nvenc',
logger=None
), number=1))
print("Size:", Path(filename).lstat().st_size)
print("-" * 80)movie_h264_nvenc_fast.mp4
Duration: 1.2489450000000488
Size: 5851164
--------------------------------------------------------------------------------show_video("movie_h264_nvenc_fast.mp4")
Now it looks a lot better and similar to the quality I am looking for, while being twice as fast as QSV. So the key with NVENC is to increase the bitrate. If I had more time I’d look futher into how to tune the quality.
Hardware H264 with h264_amf
Lastly AMF is by AMD. It’s got other legacy names like VCE. The support in FFmpeg is quite new; Windows seems to be well suppored. I am testing this on a 580. The documentation is scare but fortunately we have learned the trick of ffmpeg -h encoder=h264_amf.
filename = f"movie_h264_amf.mp4"
print(filename)
print("Duration:", timeit.timeit(lambda : clip.write_videofile(
filename,
codec='h264_amf', bitrate="8M",
# ffmpeg_params=["-quality", "speed"],
logger=None
), number=1))
print("Size:", Path(filename).lstat().st_size)
print("-" * 80)movie_h264_amf.mp4
Duration: 2.806763499999761
Size: 5307506
--------------------------------------------------------------------------------
Just borrowing the code above to apply to amf. Speed is similar to QSV and doesn’t seem to vary much after changing -usage and -quality.
Conclusion
- H264 is the format of choice.
- Use a hardware encoder (nvenc seems to be the fastest)
- For libx264, set CRF to 17 and set preset to
ultrafast. For QSV, set presetveryfastand-global_qualityto 20. For NVIDIA and AMD encoders, make sure to tune up bitrate to about8M; the other settings does not seem to affect much.