v4l2, why need VIDIOC_DQBUF 2 times to get the latest frame

[dodoma]

hello, I have a raspberry pi 2 mode b+

I want to take a still image from camera through v4l2 api,
This is my code http://paste.ubuntu.com/15099128/,
my question is why do I need ioctl VIDIOC_QBUF and VIDIOC_DQBUF 2 times to get the latest frame?
Thanks.

[6by9]

The underlying GPU code is structured for efficient pipelines. If you starve it of buffers, it will have taken the capture(s) and is waiting for a buffer to write that image(s) into. As soon as it has managed that, it will take the next capture and wait for a buffer again.
In the case of H264, there's an ~2MB FIFO in the codec that you will be filling before you actually stall the camera itself.

You're not reducing latency anyway by waiting for the key press before asking for the frame as the sensor will have to deliver the "current" frame, and then capture the new one. You'd be better off constantly circulating buffers, and on key press store the next one to be delivered.

NB I believe the bug still exists where streaming multiple JPEGs like this doesn't allow AE to run, so the exposure settings may be suboptimal.

[dodoma]

en, you means try circulating buffers?
I am afraid it's too complex and I can't handle it correctly, would you point it more clear, or, some skeleton code for this will be very helpfull.

The demand is very clear, I want capture a still jpeg image as soon as possible when 'c' pressed(or when target ready notify by GPIO on produce system)

[6by9]

Totally hacked together pseudocode.

//Allocate buf
ioctl(QBUF, &buf);
while(1) {
  ioctl(DQBUF, &buf);
  if(kbhit()) {
    char c=getch();
    if (c== 'c')
      //do stuff with image
  }
  ioctl(QBUF, &buf);
}

So you're constantly asking for buffers, and when you actually want one, you grab the next one available.
(kbhit is a Windows function, but you'll find people asking for equivalent code all over the internet, eg http://cboard.cprogramming.com/c-programming/63166-kbhit-linux.html It's actually cleaner if you're polling the state of a GPIO instead)

If you want better efficiency, switch to using several MMAPed buffers from V4L2 instead of V4L2_MEMORY_USERPTR. It just saves a chunk of work under the bonnet.

[dodoma]

ok,very clear, I will try it tomorrow.

[dodoma]

hi, 6by9

I wrote the new test code on http://paste.ubuntu.com/15178179/
It works ok, but I also have a little worry about it.

It is more like the grabed buffer(_dqbuf() told me the buffer index) is stolen from a video stream,
I can't make sure it is the 1st frame AFTER the kbd hited(or, something else happend notify by GPIO). It maybe the frame before this event, in the following execute order:
1. photo target not ready
2. ioctol(DQBUF, &buf);
3. photo target get ready
4. return TRUE on GPIO check
5. use the buf (which is not I want)

event if this may be small probability, but could be happen. If you sleep 1 second in _dqbuf(), it will be more obviously.

[dodoma]

I think the still image app developer should control WHEN to shot, his mean is get an image after the instruction sent, not the one the device prepared anytime. It's more nature to call camera_capture_image() to do this, not prepare an alone thread to stream video, and grab one frame on wanted.

[dodoma]

http://jwhsmith.net/2014/12/capturing-a-webcam-stream-using-v4l2/

As a reference.

[6by9]

Feel free to go and patch the driver - the code's all in https://github.com/raspberrypi/linux/tree/rpi-4.1.y/drivers/media/platform/bcm2835.

The GPU underneath is running OpenMax IL components with multiple output ports. V4L2 doesn't match up to that brilliantly, so some things are compromised. Normally for JPEG captures under IL you'd have the preview port running, and only trigger captures on the stills port at point of request.

V4L2 (unless you jump through hoops for multiple devices) only has one output path. If you've asked to STREAMON (note the word stream - ie continuous!), then it asks for a new capture as soon as the first has been completed. If there is no buffer available then that capture will be taken, but blocks until it can output the data.
You can try adding an extra check in there to ensure a new buffer is available, and trigger a new capture if a buffer is available. You then also need to patch buffer_queue() to check whether the capture is awaiting a buffer, and trigger a new capture if not.

NB Your latency will be higher if you are requesting frames on being presented with a buffer - rough guess the higher of 120ms or (two exposure times + 30ms).

If you really want fine control, then V4L2 is the wrong API, and you should be using IL or MMAL from userspace. The demo app raspistill can take a capture from a signal trigger or keyboard press.

[dodoma]

ok, i'll try them all.

[dodoma]

hi, 6by9

After look around mmal and v4l2's source code, I found it's controlled by the format parameter to determine how to capture still image on v4l2. In the following situations:

1. with V4L2_PIX_FMT_YUV420 pixelformat

   • 1.1 larger than 1280*720 resulution

       port = camera_port = camera_component->output[2];
       concoder_component = NULL;

   • 1.2 small than 1280*720 resolution

       port = camera_port = camera_component->output[1];
       concoder_component = NULL;

2. with V4L2_PIX_FMT_JPEG pixelformat(which is the default)

       port = image_encoder_component->output[0];
       camera_port = camera_component->output[2];
       concoder_component = image_encoder_component;

I tested these 3 situation, In 1.1(large YUV capture), the bug don't exist, In 1.2, and 2,
I need to QBUF DQBUF 2time to get the latest frame.

I'm not familar with v4l2's code right now, so, doese these information useful for you to
fix this bug?

This is the 3 situation's v4l2 driver debug output, for reference:

1. YUV

   1.1 > 720p

   [ 8353.292590] bcm2835-v4l2: vidioc_g_fmt_vid_cap: w 1024 h 768 field 1 pfmt 0x4745504a bpl 1024 sz_img 786432 colorspace 0x7 priv 0
   [ 8353.305269] bcm2835-v4l2: Clipping/aligning 1920x1080 format 32315559
   [ 8353.311761] bcm2835-v4l2: Now 1920x1080 format 32315559
   [ 8353.317125] bcm2835-v4l2: vidioc_try_fmt_vid_cap: w 1920 h 1080 field 1 pfmt 0x32315559 bpl 1920 sz_img 3133440 colorspace 0x1 priv 0
   [ 8353.329194] bcm2835-v4l2: vid_cap - disconnect previous tunnel
   [ 8353.339833] bcm2835-v4l2: Set dev->capture.fmt 30323449, 1920x1080, stride 1920, size 3133440
   [ 8353.348368] bcm2835-v4l2: queue_setup: dev:a8edd800
   
   [ 8357.214689] bcm2835-v4l2: buffer_prepare: dev:a8edd800
   [ 8357.220048] bcm2835-v4l2: buffer_queue: dev:a8edd800 buf:a8d52c00
   [ 8357.226264] bcm2835-v4l2: start_streaming: dev:a8edd800
   [ 8357.372757] bcm2835-v4l2: enabled camera (refcount 1)
   [ 8357.685488] bcm2835-v4l2: Start time 8359808673 size 8
   [ 8358.002363] bcm2835-v4l2: buffer_cb: status:0, buf:a8d52c00, length:3110400, flags 4, pts 8359973041
   [ 8358.011718] bcm2835-v4l2: Convert start time 8357.415547 and 8359808673 with offset 8359973041 to 8356.579915
   
   [ 8361.581623] bcm2835-v4l2: buffer_prepare: dev:a8edd800
   [ 8361.586929] bcm2835-v4l2: buffer_queue: dev:a8edd800 buf:a8d52c00
   [ 8361.596179] bcm2835-v4l2: buffer_cb: status:0, buf:  (null), length:0, flags 0, pts 0
   [ 8361.720683] bcm2835-v4l2: buffer_cb: status:0, buf:a8d52c00, length:3110400, flags 4, pts 8363685356
   [ 8361.730024] bcm2835-v4l2: Convert start time 8357.415547 and 8359808673 with offset 8363685356 to 8360.292230
   
   [ 8365.741552] bcm2835-v4l2: stop_streaming: dev:a8edd800
   [ 8365.746808] bcm2835-v4l2: stopping capturing
   [ 8366.745343] bcm2835-v4l2: error 0 waiting for frame completion
   [ 8366.751277] bcm2835-v4l2: disabling connection
   [ 8366.759328] bcm2835-v4l2: Disabling camera
   [ 8366.816024] bcm2835-v4l2: Camera refcount now 0
   

   1.2 <= 720p

   [ 8435.853543] bcm2835-v4l2: vidioc_g_fmt_vid_cap: w 1920 h 1080 field 1 pfmt 0x32315559 bpl 1920 sz_img 3133440 colorspace 0x1 priv 0
   [ 8435.866060] bcm2835-v4l2: Clipping/aligning 640x480 format 32315559
   [ 8435.872428] bcm2835-v4l2: Now 640x480 format 32315559
   [ 8435.877657] bcm2835-v4l2: vidioc_try_fmt_vid_cap: w 640 h 480 field 1 pfmt 0x32315559 bpl 640 sz_img 460800 colorspace 0x1 priv 0
   [ 8435.890242] bcm2835-v4l2: Set dev->capture.fmt 30323449, 640x480, stride 640, size 460800
   [ 8435.902685] bcm2835-v4l2: queue_setup: dev:a8edd800
   
   [ 8438.936639] bcm2835-v4l2: buffer_prepare: dev:a8edd800
   [ 8438.942534] bcm2835-v4l2: buffer_queue: dev:a8edd800 buf:a8d70400
   [ 8438.948945] bcm2835-v4l2: start_streaming: dev:a8edd800
   [ 8439.095677] bcm2835-v4l2: enabled camera (refcount 1)
   [ 8439.406219] bcm2835-v4l2: Start time 8441529568 size 8
   [ 8439.450955] bcm2835-v4l2: buffer_cb: status:0, buf:a8d70400, length:460800, flags 4, pts 8441516020
   [ 8439.460112] bcm2835-v4l2: Convert start time 8439.135944 and 8441529568 with offset 8441516020 to -140454173.674012
   
   [ 8441.823820] bcm2835-v4l2: buffer_prepare: dev:a8edd800
   [ 8441.829172] bcm2835-v4l2: buffer_queue: dev:a8edd800 buf:a8d70400
   [ 8441.838006] bcm2835-v4l2: buffer_cb: status:0, buf:a8d70400, length:460800, flags 4, pts 8441549335
   [ 8441.847191] bcm2835-v4l2: Convert start time 8439.135944 and 8441529568 with offset 8441549335 to 8438.155711
   
   [ 8444.927456] bcm2835-v4l2: stop_streaming: dev:a8edd800
   [ 8444.932656] bcm2835-v4l2: stopping capturing
   [ 8445.935650] bcm2835-v4l2: error 0 waiting for frame completion
   [ 8445.941522] bcm2835-v4l2: disabling connection
   [ 8445.952265] bcm2835-v4l2: Disabling camera
   [ 8445.981834] bcm2835-v4l2: Camera refcount now 0
   

2. JPEG

   [ 8523.033235] bcm2835-v4l2: vidioc_g_fmt_vid_cap: w 1024 h 768 field 1 pfmt 0x4745504a bpl 1024 sz_img 786432 colorspace 0x7 priv 0
   [ 8523.045031] bcm2835-v4l2: Clipping/aligning 640x480 format 4745504A
   [ 8523.051464] bcm2835-v4l2: Now 640x480 format 4745504A
   [ 8523.056603] bcm2835-v4l2: vidioc_try_fmt_vid_cap: w 640 h 480 field 1 pfmt 0x4745504a bpl 0 sz_img 307200 colorspace 0x7 priv 0
   [ 8523.068162] bcm2835-v4l2: vid_cap - disconnect previous tunnel
   [ 8523.078725] bcm2835-v4l2: vid_cap - set up encode comp
   [ 8523.091043] bcm2835-v4l2: JPG - buf size now 307200 was 307200
   [ 8523.097073] bcm2835-v4l2: vid_cap - cur_buf.size set to 307200
   [ 8523.103026] bcm2835-v4l2: Set dev->capture.fmt 4745504A, 640x480, stride 0, size 307200
   [ 8523.111601] bcm2835-v4l2: queue_setup: dev:a8e92000
   
   [ 8532.154841] bcm2835-v4l2: buffer_prepare: dev:a8e92000
   [ 8532.160184] bcm2835-v4l2: buffer_queue: dev:a8e92000 buf:a8d71800
   [ 8532.166368] bcm2835-v4l2: start_streaming: dev:a8e92000
   [ 8532.313499] bcm2835-v4l2: enabled camera (refcount 1)
   [ 8532.635559] bcm2835-v4l2: Start time 8534759210 size 8
   [ 8532.901478] bcm2835-v4l2: buffer_cb: status:0, buf:a8d71800, length:129504, flags 5, pts 0
   
   [ 8532.910010] bcm2835-v4l2: Grab another frame as buffer has EOS
   [ 8537.313944] bcm2835-v4l2: buffer_prepare: dev:a8e92000
   [ 8537.319418] bcm2835-v4l2: buffer_queue: dev:a8e92000 buf:a8d71800
   [ 8537.331412] bcm2835-v4l2: buffer_cb: status:0, buf:a8d71800, length:129602, flags 5, pts 0
   
   [ 8537.339978] bcm2835-v4l2: Grab another frame as buffer has EOS
   [ 8541.866219] bcm2835-v4l2: stop_streaming: dev:a8e92000
   [ 8541.871643] bcm2835-v4l2: stopping capturing
   [ 8542.876095] bcm2835-v4l2: error 0 waiting for frame completion
   [ 8542.882028] bcm2835-v4l2: disabling connection
   [ 8542.893016] bcm2835-v4l2: disabling port
   [ 8542.903049] bcm2835-v4l2: Disabling camera
   [ 8542.925025] bcm2835-v4l2: Camera refcount now 0
   

[6by9]

Yes there is a deliberate change in behaviour between JPEG and raw images >720P, vs raw images <720P, H264, or MJPEG. The first uses the stills port, the second the video port.
The video port always streams. The stills port is a one-shot triggered by MMAL_PARAMETER_CAPTURE.

You've seen in the logs "bcm2835-v4l2: Grab another frame as buffer has EOS" which is the code path to immediately request another capture for JPEG.
As I put in my previous comment, additional logic could be added at lines 336, 400, and 564 to check for a buffer being available before requesting the capture. You would also have to alter buffer_queue() to make the request should we have found there was no buffer available at the above points.

[dodoma]

En, I'm a little busy now, I'll try it later.

[dodoma]

Fixed on JPEG capture, don't care about < 720P raw image capture currently.

[Ruffio]

@dodoma has your issue been resolved? If so, please close this issue. Thanks.