This is the supplementary material for the paper A Patchmatch-based Approach for Matte Propagation in Videos.
We show additional videos depicting examples of our interface and comparisons against other matting techniques.
Important: These materials are best viewed in a web browser that is not Internet Explorer (IE), since it cannot resize images properly.
This video presents a short demonstration of our user
interface for video matting (Section 4 of our paper).
The example shows the matte extraction for a short video sequence (20 frames).
The user starts by associating a keyframe to the first frame, defining a
trimap with a few scribbles. Shared matting [SM]
is used to dynamically compute the matte for the keyframe while the user draws the scribbles. Once one is
satisfied with the obtained matte, pressing the propagate button
generates mattes and trimaps for the rest of the video.
Finally, the generated trimap for the last frame is refined with additional scribbles. By pressing the propagate button
again, these changes are propagated backwards, improving the mattes for the entire sequence.
Quantitative Comparisons
We compare the performance of our technique (using Shared Matting and Closed-Form Matting)
with two state-of-the-art sparse-input video matting methods: Motion Aware KNN Matting, and Sparse Low-Rank Representation Matting
(see Section 5.1 of our paper).
For this experiment, we used three videos from the Video Matting
Benchmark [VM] with available ground
truth: Alex, castle and Dmitriy.
We applied all techniques to each video using 2, 3, 5 and 9
trimaps (equally spaced along the video). We used the error
metrics defined by Erofeev. et al. [VM]:
SSDA, dtSSD and MESSDdt.
The graphs show each error metric computed
on a per-frame basis for each of the videos. Next, we show the resulting mattes.
OURS+CF - Ours using Closed Form Matting [CF] to obtain the alpha mattes on keyframes
OURS+SM - Ours using Shared Matting [SM] to obtain the alpha mattes on keyframes
Dmitriy keyframe 37
Mattes computed for keyframe 37 using different image matting techniques (shown here for reference).
Dmitriy frame 41 - Propagated from keyframes 37 and 56
Mattes propagated for the unconstrained frame 41 from keyframes 37 and 56.
Dmitriy keyframe 56
Mattes computed for keyframe 56 using different image matting techniques (shown here for reference).
Qualitative Comparisons
We present additional data for the qualitative comparison presented
in Section 5.2 of our paper, using the ten
test videos from the Video Matting Benchmark [VM]: Artem, city, concert, flowers, juneau, rain,
Slava, snow, Vitaliy, and woods.
For each video we compare our method against dense-input
video matting techniques.
Below, we show, for each video, the resulting mattes.
OURS+SAM - Ours using SAM alpha matte on keyframes
OURS+LB - Ours using LB alpha matte on keyframes
OURS+SpSM - Ours using SpSM alpha matte on keyframes
Important: Note that in the results below,
the techniques DM, SAM, LB and SpSM require trimaps defined on every frame of the video.
Our technique
combined with previous ones OURS+DM, OURS+SAM,
OURS+LB and OURS+SpSM use only one trimap every fifteen frames.
Trimaps defined on frames: 0, 15, 30, 45, 60, 75, 90, 105, 120, 135 and 149
Video Info:
Video name: city
Source: videomatting.com
Number of frames: 80
Playback speed: 10 frames/s
For DM, SAM, LB, SpSM:
Number of trimaps: 80
Trimaps defined on frames: every frame
For OURS+DM, OURS+SAM, OURS+LB, OURS+SpSM:
Number of trimaps: 9
Trimaps defined on frames: 0, 15, 30, 45, 60, 75 and 79
Video Info:
Video name: concert
Source: videomatting.com
Number of frames: 200
Playback speed: 10 frames/s
For DM, SAM, LB, SpSM:
Number of trimaps: 200
Trimaps defined on frames: every frame
For OURS+DM, OURS+SAM, OURS+LB, OURS+SpSM:
Number of trimaps: 15
Trimaps defined on frames: 0, 15, 30, 45, 60, 75, 90, 105, 120, 135, 150, 165, 180, 195 and 199
Video Info:
Video name: flowers
Source: videomatting.com
Number of frames: 349
Playback speed: 10 frames/s
For DM, SAM, LB, SpSM:
Number of trimaps: 349
Trimaps defined on frames: every frame
For OURS+DM, OURS+SAM, OURS+LB, OURS+SpSM:
Number of trimaps: 25
Trimaps defined on frames: 0, 15, 30, 45, 60, 75, 90, 105, 120, 135, 150, 165, 180, 195, 210, 225, 240, 255, 270, 285, 300, 315, 330, 345 and 348
Video Info:
Video name: flowers
Source: videomatting.com
Number of frames: 151
Playback speed: 10 frames/s
For DM, SAM, LB, SpSM:
Number of trimaps: 151
Trimaps defined on frames: every frame
For OURS+DM, OURS+SAM, OURS+LB, OURS+SpSM:
Number of trimaps: 11
Trimaps defined on frames: 0, 15, 30, 45, 60, 75, 90, 105, 120, 135 and 150
Video Info:
Video name: rain
Source: videomatting.com
Number of frames: 50
Playback speed: 10 frames/s
For DM, SAM, LB, SpSM:
Number of trimaps: 50
Trimaps defined on frames: every frame
For OURS+DM, OURS+SAM, OURS+LB, OURS+SpSM:
Number of trimaps: 5
Trimaps defined on frames: 0, 15, 30, 45 and 49
Video Info:
Video name: Slava
Source: videomatting.com
Number of frames: 150
Playback speed: 10 frames/s
For DM, SAM, LB, SpSM:
Number of trimaps: 150
Trimaps defined on frames: every frame
For OURS+DM, OURS+SAM, OURS+LB, OURS+SpSM:
Number of trimaps: 11
Trimaps defined on frames: 0, 15, 30, 45, 60, 75, 90, 105, 120, 135 and 149
Video Info:
Video name: snow
Source: videomatting.com
Number of frames: 160
Playback speed: 10 frames/s
For DM, SAM, LB, SpSM:
Number of trimaps: 160
Trimaps defined on frames: every frame
For OURS+DM, OURS+SAM, OURS+LB, OURS+SpSM:
Number of trimaps: 12
Trimaps defined on frames: 0, 15, 30, 45, 60, 75, 90, 105, 120, 135, 150, 159
Video Info:
Video name: Vitaliy
Source: videomatting.com
Number of frames: 149
Playback speed: 10 frames/s
For DM, SAM, LB, SpSM:
Number of trimaps: 149
Trimaps defined on frames: every frame
For OURS+DM, OURS+SAM, OURS+LB, OURS+SpSM:
Number of trimaps: 11
Trimaps defined on frames: 0, 15, 30, 45, 60, 75, 90, 105, 120, 135, 148
Video Info:
Video name: woods
Source: videomatting.com
Number of frames: 151
Playback speed: 10 frames/s
For DM, SAM, LB, SpSM:
Number of trimaps: 151
Trimaps defined on frames: every frame
For OURS+DM, OURS+SAM, OURS+LB, OURS+SpSM:
Number of trimaps: 11
Trimaps defined on frames: 0, 15, 30, 45, 60, 75, 90, 105, 120, 135, 150
Qualitative Comparison - Adobe After Effects
We present a comparison against Adobe After Effects Rotobrush Tool [AE].
For AE we initialized the process with scribbles every 15 frames, trying to obtain
the best results for keyframes. We compare Rotobrush propagation with ours initializing
with DM every 15 frames.
Number of frames: 46 (interval 0-45 out of 150 frames)
Playback speed: 10 frames/s
Number of trimaps: 4
Trimaps defined on frames: 0, 15, 30, 45
Video Info:
Video name: city
Source: videomatting.com
Number of frames: 46 (interval 0-45 out of 80 frames)
Playback speed: 10 frames/s
Number of trimaps: 4
Trimaps defined on frames: 0, 15, 30, 45
Video Info:
Video name: concert
Source: videomatting.com
Number of frames: 46 (interval 90-135 out of 200 frames)
Playback speed: 10 frames/s
Number of trimaps: 4
Trimaps defined on frames: 0, 15, 30, 45
Video Info:
Video name: flowers
Source: videomatting.com
Number of frames: 46 (interval 0-45 out of 349 frames)
Playback speed: 10 frames/s
Number of trimaps: 4
Trimaps defined on frames: 0, 15, 30, 45
Video Info:
Video name: flowers
Source: videomatting.com
Number of frames: 46 (interval 0-45 out of 151 frames)
Playback speed: 10 frames/s
Number of trimaps: 4
Trimaps defined on frames: 0, 15, 30, 45
Video Info:
Video name: rain
Source: videomatting.com
Number of frames: 46 (interval 0-45 out of 50 frames)
Playback speed: 10 frames/s
Number of trimaps: 4
Trimaps defined on frames: 0, 15, 30, 45
Video Info:
Video name: Slava
Source: videomatting.com
Number of frames: 46 (interval 0-45 out of 150 frames)
Playback speed: 10 frames/s
Number of trimaps: 4
Trimaps defined on frames: 0, 15, 30, 45
Video Info:
Video name: snow
Source: videomatting.com
Number of frames: 46 (interval 0-45 out of 160 frames)
Playback speed: 10 frames/s
Number of trimaps: 4
Trimaps defined on frames: 0, 15, 30, 45
Video Info:
Video name: Vitaliy
Source: videomatting.com
Number of frames: 46 (interval 75-120 out of 149 frames)
Playback speed: 10 frames/s
Number of trimaps: 4
Trimaps defined on frames: 0, 15, 30, 45
Video Info:
Video name: woods
Source: videomatting.com
Number of frames: 46 (interval 75-120 out of 151 frames)
Playback speed: 10 frames/s
Number of trimaps: 4
Trimaps defined on frames: 0, 15, 30, 45
Handling Foreground Occlusions
We present an example of matte with foreground occlusion produced
by our interactive matting interface.
References
[AE]
Adobe Inc.: Adobe After Effects, 2019. adobe.com.
[CF]
A. Levin, D. Lischinski, and Y. Weiss. 2008. A Closed-Form
Solution to Natural Image Matting. IEEE Transactions on
Pattern Analysis and Machine Intelligence 30, 2 (Feb. 2008),
228-242.
[DM]
Ning Xu, Brian Price, Scott Cohen, and Thomas Huang. 2017.
Deep Image Matting. arXiv:1703.03872 [cs] (March 2017).
[LB]
Y. Zheng and C. Kambhamettu. 2009. Learning Based Digital
Matting. In 2009 IEEE 12th International Conference on
Computer Vision. 889-896.
[MAKNN]
D. Li, Q. Chen, and C. Tang. 2013. Motion-Aware KNN
Laplacian for Video Matting. In 2013 IEEE International
Conference on Computer Vision. 3599-3606.
[SAM]
Guangying Cao, Jianwei Li, Xiaowu Chen, and Zhiqiang He. 2019.
Patch-Based Self-Adaptive Matting for High-Resolution Image
and Video. Vis Comput 35, 1 (Jan. 2019), 133-147.
[SLRR]
D. Zou, X. Chen, G. Cao, and X. Wang. 2019. Unsupervised
Video Matting via Sparse and Low-Rank Representation.
IEEE Transac- tions on Pattern Analysis and Machine
Intelligence (2019), 1-1.
[SM]
Eduardo S. L. Gastal and Manuel M. Oliveira. 2010. Shared
Sampling for Real-Time Alpha Matting. Computer Graphics
Forum 29, 2 (May 2010), 575-584.
[SpSM]
L. Karacan, A. Erdem, and E. Erdem. 2017. Alpha Matting
With KL- Divergence-Based Sparse Sampling. IEEE
Transactions on Image Processing 26, 9 (Sept. 2017), 4523-4536.
[VM]
Mikhail Erofeev, Yury Gitman, Dmitriy Vatolin, Alexey Fedorov, and
Jue Wang. 2015. Perceptually Motivated Benchmark for Video
Matting. In BMVC.