OpenCV & Python Multithreading - Seeking within a VideoCapture Object

Based on the comments on the original question I've done some testing and thought it worth sharing the (interesting) results. Big savings potential for anyone using OpenCV's VideoCapture.set(CAP_PROP_POS_MSEC) or VideoCapture.set(CAP_PROP_POS_FRAMES).

I've done some profiling comparing three options:

1. GET FRAMES BY SEEKING TO TIME:

frames = {}def get_all_frames_by_ms(time):    while True:        video_capture.set(cv2.CAP_PROP_POS_MSEC, time)        capture_success, frames[time] = video_capture.read()        if not capture_success:            break        time += 1000

2. GET FRAMES BY SEEKING TO FRAME NUMBER:

frames = {}def get_all_frames_by_frame(time):    while True:        # Note my test video is 12.333 FPS, and time is in milliseconds        video_capture.set(cv2.CAP_PROP_POS_FRAMES, int(time/1000*12.333))        capture_success, frames[time] = video_capture.read()        if not capture_success:            break        time += 1000

3. GET FRAMES BY GRABBING ALL, BUT RETRIEVING ONLY ONES I WANT:

def get_all_frames_in_order():    prev_time = -1    while True:        grabbed = video_capture.grab()        if grabbed:            time_s = video_capture.get(cv2.CAP_PROP_POS_MSEC) / 1000            if int(time_s) > int(prev_time):                # Only retrieve and save the first frame in each new second                self.frames[int(time_s)] = video_capture.retrieve()            prev_time = time_s        else:            break

Running through those three approaches, the timings (from three runs of each) are as follows:

1. 33.78s 29.65s 29.24s
2. 31.95s 29.16s 28.35s
3. 11.81s 10.76s 11.73s

In each case it's saving 100 frames at 1sec intervals into a dictionary, where each frame is a 3072x1728 image, from a .mp4 video file. All on a 2015 MacBookPro with 2.9 GHz Intel Core i5 and 8GB RAM.

Conclusions so far... if you're interested in retrieving only some frames from a video, then very worth looking at running through all frames in order and grabbing them all, but only retrieving those you're interested in - as an alternative to reading (which grabs and retrieves in one go). Gave me an almost 3x speedup.

I've also re-looked at multi-threading on this basis. I've got two test processes - one that gets the frames, and another that processes them once they're available:

frames = {}def get_all_frames_in_order():    prev_time = -1    while True:        grabbed = video_capture.grab()        if grabbed:            time_s = video_capture.get(cv2.CAP_PROP_POS_MSEC) / 1000            if int(time_s) > int(prev_time):                # Only retrieve and save the first frame in each new second                frames[int(time_s)] = video_capture.retrieve()            prev_time = time_s        else:            breakdef process_all_frames_as_available(processing_time):    prev_time = 0    while True:        this_time = prev_time + 1000        if this_time in frames and prev_time in frames:            # Dummy processing loop - just sleeps for specified time            sleep(processing_time)            prev_time += self.time_increment            if prev_time + self.time_increment > video_duration:                break        else:            # If the frames aren't ready yet, wait a short time before trying again            sleep(0.02)

For this testing, I then called them either one after the other (sequentially, single threaded), or with the following muti-threaded code:

get_frames_thread = Thread(target=get_all_frames_in_order)get_frames_thread.start()process_frames_thread = Thread(target=process_all_frames_as_available, args=(0.02,))process_frames_thread.start()get_frames_thread.join()process_frames_thread.join()

Based on that, I'm now happy that multi-threading is working effectively and saving a significant amount of time. I generated timings for the two functions above separately, and then together in both single-threaded and multi-threaded modes. The results are below (number in bracket is the time in seconds that the 'processing' for each frame takes, which in this case is just a dummy / delay):

get_all_frames_in_order - 2.99sProcess time = 0.02s per frame:process_all_frames_as_available - 0.97ssingle-threaded - 3.99smulti-threaded - 3.28sProcess time = 0.1s per frame:process_all_frames_as_available - 4.31ssingle-threaded - 7.35smulti-threaded - 4.46sProcess time = 0.2s per frame:process_all_frames_as_available - 8.52ssingle-threaded - 11.58smulti-threaded - 8.62s

As you can hopefully see, the multi-threading results are very good. Essentially, it takes just ~0.2s longer to do both functions in parallel than the slower of the two functions running entirely separately.

Hope that helps someone!

Coincidentally, I've worked on a similar problem, and I have created a python library (more of a thin wrapper) for reading videos. The library is called mydia.

The library does not use OpenCV. It uses FFmpeg as the backend for reading and processing videos.

mydia supports custom frame selection, frame resizing, grayscale conversion and much more. The documentation can be viewed here

So, if you want to select N frames per second (where N = 1 in your case), the following code would do it:

import numpy as npfrom mydia import Videosvideo_path = "path/to/video"def select_frames(total_frames, num_frames, fps, *args):    """This function will return the indices of the frames to be captured"""    N = 1    t = np.arange(total_frames)    f = np.arange(num_frames)    mask = np.resize(f, total_frames)    return t[mask < N][:num_frames].tolist()# Let's assume that the duration of your video is 120 seconds# and you want 1 frame for each second # (therefore, setting `num_frames` to 120)reader = Videos(num_frames=120, mode=select_frames)video = reader.read(video_path)  # A video tensor/array

The best part is that internally, only those frames that are required are read, and therefore the process is much faster (which is what I believe you are looking for).

The installation of mydia is extremely simple and can be viewed here.

This might have a slight learning curve, but I believe that it is exactly what you are looking for.

Moreover, if you have multiple videos, you could use multiple workers for reading them in parallel. For instance:

from mydia import Videospath = "path/to/video"reader = Videos()video = reader.read(path, workers=4)

Depending on your CPU, this could give you a significant speed-up.

Hope this helps !!