open source shot detection implementation?

Question

I am trying to implement a video retrieval system and I need to first extract key frames from a video, ideally I want to have a library for automatically detecting those key frames, one key frame from each shot. Bonus if I can configure which key frame to extract(first, middle, or last of the continuous frames in a shot). Is there an open source implementation for this?

Answer 1

What you're after is called shot segmentation. While it's a pretty active research area, I think you're unlikely to find any complete libraries that solve the problem for you out-of-the-box. Your best option may be reading up on the topic, selecting an approach that best fits your requirements and coding it up yourself.

One approach is to calculate the chi-squared distance between the color histograms of adjacent frames. When this distance rises above a user-specified threshold, you are at a shot boundary. The approach is explained in this paper:

A. Nagasaka and Y. Tanaka, "Automatic video indexing and full-video search for object appearances", Journal of Information Processing archive, Volume 15, Issue 2 (1992), Page 316

I've played around with it, with some success. Notable failures are sudden light changes within a single shot (caused by camera flash, etc) and blending shot changes.

Once you know the shot boundaries, picking a keyframe from each shot will be trivial, as other people have pointed out.