lag in opencv videocapture when i use rtsp camera stream

I had similar issues and was able to resolve them by completely isolating the frame capturing from processing of the images. I also updated OpenCV to the latest (3.2.0) available, but I think this will also resolve problems with earlier versions.

void StreamLoop(String strCamera, LFQueue1P1C<Mat> *imageQueue, bool *shutdown) {    VideoCapture cap(strCamera, CV_CAP_FFMPEG);    Mat image;    while(!(*shutdown) && cap.isOpened()){        *cap >> image;        imageQueue->Produce(image);    }}int main(){    Mat aImage1;    bool shutdown(false);    LFQueue1P1C<Mat> imageQueue;    string rstp("rtsp://admin:password@ipaddress:554/live2.sdp?tcp");    thread streamThread(StreamLoop, rtsp, &imageQueue, &shutdown);...    while(!shutdownCondition){        if(imageQueue.Consume(aImage1)) {            // Process Image            resize(aImage1, aImage1, Size(640, 480));            detect(aImage1, rtsp);        }    }    shutdown = true;    if(streamThread.joinable()) streamThread.join();...    return 0;}

It seems that there is some issue with rtsp in OpenCV where it easily hangs up if there are even slight pauses while picking up the frames. As long as I pick up frames without much pause I have not seen a problem.

Also, I didn't have this issue when the video cameras where directly connected to my local network. It was not until we deployed them at a remote site that I started getting the hang ups. Separating frame retrieval and processing into separate threads resolved my issues, hopefully someone else might find this solution useful.

Note: The queue I used is a custom queue for passing items from one thread to another. The code I posted is modified from my original code to make it more readable and applicable to this problem.

i'm still a beginner in multi threading matters so any help would be appreciated

Having threads that have no way of exiting will cause you issues in the future. Even if it is test code, get in the habit of making sure the code has an exit path. As an example: You might copy and paste a section of code later on and forget there is an infinite loop in there and it will cause great grief later trying to track down why you have mysterious crashing or your resources are locked up.

I am not a C++ developer but I had the same problem in Java. I solved my issue by calling VideoCapture.grab() function before reading camera frame. According to OpenCV Doc, the use of the grab function is :

The primary use of the function is in multi-camera environments, especially when the cameras do not have hardware synchronization.

Besides that, in java application, you should release your frame's Mat objects every time you read new frames.