How to stream live video frames from client to flask server and back to the client?

So, what I was trying to do is to take the real time video stream captured by the client's webcam and process them at backend.

My backend code is written in Python and I am using SocketIo to send the frames from frontend to backend. You can have a look at this design to get a better idea about what's happening -image

1. My server(app.py) will be running in backend and client will be accessing index.html
2. SocketIo connection will get establish and video stream captured using webcam will be send to server frames by frames.
3. These frames will be then processed at the backend and emit back to the client.
4. Processed frames coming form the server can be shown in img tag.

Here is the working code -

app.py

@socketio.on('image')def image(data_image):    sbuf = StringIO()    sbuf.write(data_image)    # decode and convert into image    b = io.BytesIO(base64.b64decode(data_image))    pimg = Image.open(b)    ## converting RGB to BGR, as opencv standards    frame = cv2.cvtColor(np.array(pimg), cv2.COLOR_RGB2BGR)    # Process the image frame    frame = imutils.resize(frame, width=700)    frame = cv2.flip(frame, 1)    imgencode = cv2.imencode('.jpg', frame)[1]    # base64 encode    stringData = base64.b64encode(imgencode).decode('utf-8')    b64_src = 'data:image/jpg;base64,'    stringData = b64_src + stringData    # emit the frame back    emit('response_back', stringData)

index.html

<div id="container">    <canvas id="canvasOutput"></canvas>    <video autoplay="true" id="videoElement"></video></div><div class = 'video'>    <img id="image"></div><script>    var socket = io('http://localhost:5000');    socket.on('connect', function(){        console.log("Connected...!", socket.connected)    });    const video = document.querySelector("#videoElement");    video.width = 500;     video.height = 375; ;    if (navigator.mediaDevices.getUserMedia) {        navigator.mediaDevices.getUserMedia({ video: true })        .then(function (stream) {            video.srcObject = stream;            video.play();        })        .catch(function (err0r) {            console.log(err0r)            console.log("Something went wrong!");        });    }    let src = new cv.Mat(video.height, video.width, cv.CV_8UC4);    let dst = new cv.Mat(video.height, video.width, cv.CV_8UC1);    let cap = new cv.VideoCapture(video);    const FPS = 22;    setInterval(() => {        cap.read(src);        var type = "image/png"        var data = document.getElementById("canvasOutput").toDataURL(type);        data = data.replace('data:' + type + ';base64,', ''); //split off junk         at the beginning        socket.emit('image', data);    }, 10000/FPS);    socket.on('response_back', function(image){        const image_id = document.getElementById('image');        image_id.src = image;    });</script>

Also, websockets runs on secure origin.

I had to tweak your solution a bit :-

I commented the three cv variables and the cap.read(src) statement, modified the following line

var data = document.getElementById("canvasOutput").toDataURL(type);

to

        var video_element = document.getElementById("videoElement")        var frame = capture(video_element, 1)        var data = frame.toDataURL(type);

Using the capture function from here :- http://appcropolis.com/blog/web-technology/using-html5-canvas-to-capture-frames-from-a-video/

I'm not sure if this is the right way to do it but it happened to work for me.

Like I said I'm not super comfortable with javascript so instead of manipulating the base64 string in javascript, I'd much rather just send the whole data from javascript and parse it in python this way

# Important to only split onceheaders, image = base64_image.split(',', 1) 

My takeaway from this, at the risk of sounding circular, is that you can't directly pull an image string out of a canvas that is containing a video element, you need to create a new canvas onto which you draw a 2D image of the frame you capture from the video element.