【Qt从摄像头视频中获取数据】

有时候需要在视频上画图，所以需要能获取到每一帧视频数据。

以前从视频文件或视频流中得到帧，一般都是使用qt + ffmpeg或qt + vlc。

qt对显示处理视频大体有以下方法：

1. QMediaPlayer + QVideoWidget

这种方法只适合简单的显示视频功能，不适合对视频进行处理（比如画图）

2. QMediaPlayer + QGraphicsVideoItem + QGraphicsScene + QGraphicsView

这种方法功能强大，除了显示视频功能，还可以做复杂的图形处理(具体可以查看QGraphicsScene的使用)

3. QMediaPlayer + QAbstractVideoSurface

这种方法比较简单，是我下面要介给的。可以获取到每一帧视频数据，基本可以实现与qt + ffmpeg或qt + vlc相同的效果。

自定义VideoSurface：

class VideoSurface : public QAbstractVideoSurface
{Q_OBJECTpublic:VideoSurface(QObject *parent = Q_NULLPTR);~VideoSurface();QList<QVideoFrame::PixelFormat> supportedPixelFormats(QAbstractVideoBuffer::HandleType handleType = QAbstractVideoBuffer::NoHandle) const;bool present(const QVideoFrame &frame);signals:void frameAvailable(QVideoFrame &frame);};

实现如下：

VideoSurface::VideoSurface(QObject *parent): QAbstractVideoSurface(parent)
{
}VideoSurface::~VideoSurface()
{
}QList<QVideoFrame::PixelFormat> VideoSurface::supportedPixelFormats(QAbstractVideoBuffer::HandleType handleType) const
{QList<QVideoFrame::PixelFormat> listPixelFormats;listPixelFormats << QVideoFrame::Format_ARGB32<< QVideoFrame::Format_ARGB32_Premultiplied<< QVideoFrame::Format_RGB32<< QVideoFrame::Format_RGB24<< QVideoFrame::Format_RGB565<< QVideoFrame::Format_RGB555<< QVideoFrame::Format_ARGB8565_Premultiplied<< QVideoFrame::Format_BGRA32<< QVideoFrame::Format_BGRA32_Premultiplied<< QVideoFrame::Format_BGR32<< QVideoFrame::Format_BGR24<< QVideoFrame::Format_BGR565<< QVideoFrame::Format_BGR555<< QVideoFrame::Format_BGRA5658_Premultiplied<< QVideoFrame::Format_AYUV444<< QVideoFrame::Format_AYUV444_Premultiplied<< QVideoFrame::Format_YUV444<< QVideoFrame::Format_YUV420P<< QVideoFrame::Format_YV12<< QVideoFrame::Format_UYVY<< QVideoFrame::Format_YUYV<< QVideoFrame::Format_NV12<< QVideoFrame::Format_NV21<< QVideoFrame::Format_IMC1<< QVideoFrame::Format_IMC2<< QVideoFrame::Format_IMC3<< QVideoFrame::Format_IMC4<< QVideoFrame::Format_Y8<< QVideoFrame::Format_Y16<< QVideoFrame::Format_Jpeg<< QVideoFrame::Format_CameraRaw<< QVideoFrame::Format_AdobeDng;//qDebug() << listPixelFormats;// Return the formats you will supportreturn listPixelFormats;
}bool  VideoSurface::present(const QVideoFrame &frame)
{// Handle the frame and do your processingif (frame.isValid()){QVideoFrame cloneFrame(frame);emit frameAvailable(cloneFrame);return true;}return false;
}

看了上面的代码就知道，只需要外部连接frameAvailable信号就可以获取到每一帧数据。

具体使用：

QMediaPlayer *mediaPlayer = new QMediaPlayer;
VideoSurface *videoSurface = new VideoSurface;
mediaPlayer->setVideoOutput(videoSurface);
mediaPlayer->setMedia(QUrl("rtsp://admin:admin@192.168.1.112"));
mediaPlayer->play();

把frameAvailable信号与显示窗口的槽连接，比如：

connect(videoSurface, SIGNAL(frameAvailable(QVideoFrame &)), this, SLOT(ProcessFrame(QVideoFrame &)));
void QtVideoTest::ProcessFrame(QVideoFrame &frame)
{qDebug() << "=============ProcessFrame===============";qDebug() << "width : " << frame.width() << " height : " << frame.height();qDebug() << "start time : " << frame.startTime()/1000 << "ms";qDebug() << "end time : " << frame.endTime()/1000 << "ms";qDebug() << "pixelFormat :" << frame.pixelFormat();frame.map(QAbstractVideoBuffer::ReadOnly);QImage recvImage(frame.bits(), frame.width(), frame.height(), QVideoFrame::imageFormatFromPixelFormat(frame.pixelFormat()));qDebug() << "frame data size :" << frame.mappedBytes();frame.unmap();
}

如上，QVideoFrame转QImage，拿QImage进行画图操作就简单了。

Qt6.2.3获取摄像头录制YUV文件，逐帧获取录制，使用QVideoSink

Qt6.2.3获取摄像头录制YUV文件，逐帧获取录制，使用QVideoSink
必须要使用到QVideoSink，相应的依赖模块有：QMediaCaptureSession、QMediaDevices、QCamera、QVideoFrame

具体的使用方法是：
1、创建一个QMediaCaptureSession：
QMediaCaptureSession m_captureSession;
2、设置摄像头：
m_captureSession.setCamera(new QCamera(QMediaDevices::defaultVideoInput()));
3、设置YUV逐帧输出：
m_captureSession.setVideoSink(new QVideoSink m_videoSink);
4、设置YUV格式：
void QVideoSink::setVideoFrame(const QVideoFrame &frame)
5、主动获取一帧YUV数据：
QVideoFrame QVideoSink::videoFrame() const
6、被动接收每一帧：
connect(m_videoSink, &QVideoSink::videoFrameChanged, YourClass, yourParseFunction);
7、获取帧数据：
videoFrame.map(QVideoFrame::ReadOnly);
m_file.write((const char *)videoFrame.bits(0和1), videoFrame.mappedBytes(0和1));

mainwindow.cpp:

#include "mainwindow.h"
#include "ui_mainwindow.h"
#include <QCameraFormat>MainWindow::MainWindow(QWidget *parent): QMainWindow(parent), ui(new Ui::MainWindow)
{ui->setupUi(this);m_camera = new QCamera(QMediaDevices::defaultVideoInput());m_captureSession.setCamera(m_camera);m_captureSession.setVideoSink(&m_videoSink);connect(&m_videoSink, &QVideoSink::videoFrameChanged, this, &MainWindow::on_frame_changed);m_camera->start();m_file.setFileName("test.nv12");m_file.open(QIODevice::WriteOnly | QIODevice::Truncate);
}MainWindow::~MainWindow()
{m_camera->stop();m_file.close();delete ui;
}void MainWindow::on_frame_changed(const QVideoFrame &frame)
{QVideoFrame videoFrame = frame;/*** 我的电脑上返回的格式是Format_NV12：18，每个像素1.5字节，一帧1382400字节，* 如果你需要用别的YUV或者RGB格式，则需要自己设置QVideoSink或者QCamera格式*/QVideoFrameFormat::PixelFormat pixelFormat = videoFrame.pixelFormat();int width = videoFrame.width();int height = videoFrame.height();int planeCount = videoFrame.planeCount();uchar *pdata = nullptr;int len = 0;videoFrame.map(QVideoFrame::ReadOnly);for (int i = 0; i < planeCount; i++) {pdata = videoFrame.bits(i);len = videoFrame.mappedBytes(i);m_file.write((const char *)pdata, len);}frame_num++;qDebug("%d, %d, %d, %d, %d, %p, %d", frame_num,pixelFormat, width, height, planeCount, pdata, len);videoFrame.unmap();
}

mainwindow.h:

#ifndef MAINWINDOW_H
#define MAINWINDOW_H#include <QMainWindow>
#include <QCamera>
#include <QMediaCaptureSession>
#include <QMediaDevices>
#include <QVideoSink>
#include <QVideoFrame>
#include <QFile>QT_BEGIN_NAMESPACE
namespace Ui { class MainWindow; }
QT_END_NAMESPACEclass MainWindow : public QMainWindow
{Q_OBJECTpublic:MainWindow(QWidget *parent = nullptr);~MainWindow();private:Ui::MainWindow *ui;QMediaCaptureSession m_captureSession;QVideoSink m_videoSink;QCamera *m_camera;QFile m_file;int frame_num = 0;private slots:void on_frame_changed(const QVideoFrame &frame); // 一帧视频到来的信号
};
#endif // MAINWINDOW_H

main.cpp:

/*** \brief Qt6.2.3要逐帧获取YUV需要使用QVideoSink和QMediaCaptureSession* \details 编译运行后，窗口弹出即开始录制YUV文件，关闭窗口后文件保存* \note 保存的文件在本工程文件夹上级目录中的debug*目录下的test.nv12*       我的电脑摄像头默认输出是YUV NV12格式1280 * 720*       播放方式是安装ffmpeg软件后，在test.nv12文件目录下使用命令：*       ffplay -f rawvideo -pix_fmt nv12 -video_size 1280x720 test.nv12*/#include "mainwindow.h"
#include <QApplication>int main(int argc, char *argv[])
{QApplication a(argc, argv);MainWindow w;w.show();return a.exec();
}

005_yuv_input_save_file.pro:

QT       += core gui
QT       += multimediagreaterThan(QT_MAJOR_VERSION, 4): QT += widgetsCONFIG += c++17# You can make your code fail to compile if it uses deprecated APIs.
# In order to do so, uncomment the following line.
#DEFINES += QT_DISABLE_DEPRECATED_BEFORE=0x060000    # disables all the APIs deprecated before Qt 6.0.0SOURCES += \main.cpp \mainwindow.cppHEADERS += \mainwindow.hFORMS += \mainwindow.ui# Default rules for deployment.
qnx: target.path = /tmp/$${TARGET}/bin
else: unix:!android: target.path = /opt/$${TARGET}/bin
!isEmpty(target.path): INSTALLS += target

mainwindow.ui:

<?xml version="1.0" encoding="UTF-8"?>
<ui version="4.0"><class>MainWindow</class><widget class="QMainWindow" name="MainWindow"><property name="geometry"><rect><x>0</x><y>0</y><width>800</width><height>600</height></rect></property><property name="windowTitle"><string>MainWindow</string></property><widget class="QWidget" name="centralwidget"><layout class="QVBoxLayout" name="verticalLayout"><item><widget class="QPushButton" name="pushButton"><property name="text"><string>窗口不要开太久！开几秒就够了，否则存储的YUV文件会把你的硬盘塞满！</string></property></widget></item></layout></widget><widget class="QMenuBar" name="menubar"><property name="geometry"><rect><x>0</x><y>0</y><width>800</width><height>22</height></rect></property></widget><widget class="QStatusBar" name="statusbar"/></widget><resources/><connections/>
</ui>

报错：`QVideoFrame::unmap() was called more times then QVideoFrame::map()

原因：格式对不上QVideoFrame转QImage格式对照表`
QVideoFrame Class

参考：QT——QCamera摄像头的切换、分辨率切换、截图显示

cv::QVideoFrame转QImage失败原因及解决方案