[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RFC: bus and data format negotiation

(I promised that I would analyze this. Sorry that it took so long, but I
had a lot of other things going on and this was one topic that I needed to
really sit down for and think about carefully.)

RFC: bus and data format negotiation

Version 1.0


As media boards become more complex it is getting harder to enumerate and
setup the busses that connect the various components on the board and to
map that to actual pixelformats (i.e. how the data will end up in memory).

This is a particular problem for sensors that can often be connected in
many different ways.

Several attempts have been made to find a good internal API for this, but
none were satisfactory.

In this RFC I'll analyze how such connections are made, what the core problems
are and I'll present a solution that hopefully solves these problems in a
well-defined and not too complex way. This RFC is directly related to open
issue #3 in the media controller RFC.


In general you have two components, A and B, and some sort of a bus between
them (i.e. the physical wires). This can be straightforward, e.g. 10 pins
on either side are directly connected, or there can be additional components
in between like level converters or perhaps even complex FPGAs that are not
under our control but that nevertheless can make changes in how the data is
presented. While information on such components may not be available at the
driver level, it is available at platform (board) level.

More complex boards have more than two components and multiple busses, but
many of the basics remain the same.

Bus configuration

In order to setup a component you will need to supply a bus configuration
that sets up the physical properties of the bus: signal polarities, how the
data should be sampled, etc.

In many cases there is only one possible bus configuration. But especially
sensors have more configurations. This configuration should come from the
board specification and not be autonegotiated. Depending on the board layout
a wrong bus configuration can have quite unpredictable results. Even though
both sides of the bus may seemingly support a specific configuration additional
board-specific factors may prevent that configuration from working reliably.

Data formats

For a given bus configuration a component can support one or more data
formats. A data format is similar, but not identical, to a pixel format. A
pixel format defines the format of the video in memory. A data format defines
the format of the video on a bus. The component's driver will know which data
formats it supports given the bus config.

Note that changing the bus config on the fly will also change the list of
supported data formats. Normally a bus config is setup once and not changed,
but this is not necessarily always the case.

Video timings

Once the bus is configured and the data format is set it is finally possible
to determine what resolutions and framerates are supported. Here we run into
a problem, though. The current V4L2 API is not clear on how that should be

We have three enumeration ioctls involved in this: either VIDIOC_ENUMSTD to
enumerate the supported video standards for analog TV video encoders and
decoders, and VIDIOC_ENUM_FRAMESIZES/FRAMEINTERVALS to enumerate native
sensor resolutions and framerates. Unfortunately the spec is not clear whether
ENUM_FRAMESIZES/INTERVALS really refers to the sensor (or more general, the
component that is the initial source of the video stream) or to the sizes
and framerates that you can capture on the video node.

What makes this worse is that there is an essential ioctl missing: you need
to have the equivalent of VIDIOC_S_STD to setup a sensor to a specific
resolution/framerate. Right now VIDIOC_S_FMT is used for that, but that is
really meant to take an input resolution and scale it up or down to the
desired resolution. It is not meant to setup a sensor (or video source or sink
in general) to a particular resolution.

To fix this we need to 1) specify that the framesize/interval enumeration
relates to the video source and not to the output of a possible scaler, and
2) add support for setting up a sensor to a specific size/framerate. Murali 
Karicheri from TI is working on something similar.

Using S_FMT to select a particular resolution never felt right, and now I
realize why. Luckily I don't think any of the webcam drivers we have currently
do any scaling, so using S_FMT will still work for those and applications do
not need to be modified.

To do the sensor setup in the new-style we can either introduce a new ioctl
to specify the size and use VIDIOC_S_PARM (yuck!) to setup the framerate, or
we use the new video timings ioctl that Murali is working on.

Data format negotiation

Depending on the bus configuration a component supports a list of data formats.
The next step is to somehow coordinate both sides of the bus to select
compatible data formats. In many cases you can just go through the supported
data formats and find matches. But this is not true in general. A generic
camera framework like soc-camera either needs an optional mapping table that
is supplied by the board configuration, or it should be given a callback
function that can be implemented at the board level to do the matching for
soc-camera (and if no function is supplied it can fallback to a default
matching function).

A simple example where matching would fail is if the sensor is sending on
8 pins and the SoC receives it on 12 pins (with the lowest 4 pins pulled
down). So the data format from the sensor is some 8 bit Bayer format, while
the SoC receives a 12 bit Bayer format. You can try all sorts of fancy
heuristics to solve such problems, but in my opinion that will never be able
to fully solve this data format negotiation. In the end the board designer
knows exactly how to match it up, and so it is best to place the matching code
at the level where that information is actually available: the platform code.

This type of data format matching works well in simple situations (e.g. a
sensor connected to a SoC), but when you have a network of components all
talking to one another it becomes impossible to manage that in a driver.

Using the media controller it should be possible to setup the data format
for a sub-device directly. Obviously, this only makes sense for SoCs.
In theory one could also setup the bus configuration that way, but I feel
uncomfortable doing that. This really belongs at the platform level.

Note that when dealing with a SoC that e.g. connects a previewer component
to a resizer component internally it is the responsibility of the SoC driver
to select the data formats. This will not come from platform data, since this
is a SoC-internal connection. Whether to allow an application to explicitly
set the data format in this case is something that the SoC driver developer
will have to decide.

Pixel formats

Until now we have been talking about data formats. But the end-user only sees
pixel formats. A pixel format defines uniquely how the video will be formatted
in memory. The translation from a data format to memory is generally done by
a DMA engine. A single data format can be DMAed into memory in potentially
several ways, depending on the setup of the DMA engine.

In many of our TV and webcam drivers there is only one data format (usually
the default format, and often only format, available by the various components).

So effectively the pixel enumeration maps that single data format to a set of
pixel formats. And selecting a pixel format just changes the DMA engine and
not the internal data format.

For a complex device it may become very hard to generate a full list of all
the possible pixel formats supported by all the possible data formats and DMA
engine settings. This can easily explode into an unmanagable list, not to
mention very difficult driver code. And does anyone really care about 200
possible pixel formats?

I would suggest that every driver should enumerate the available pixel formats
for the selected data format of the DMA engine. Some drivers might have
additional information that allows them to extend that to more pixel formats
and setup the data format based on the selected pixel format. For example,
soc-camera might achieve that if the platform code will provide it with a
table that maps pixel formats to data formats (a data format for the SoC and
one for the sensor).

A full list of pixel formats will typically be more important for consumer
market devices than for SoCs, so it is up to the driver to decide what is
best here.

Video nodes supporting input and output

There is a special case that is currently not implemented, but it will be
in the near future: stand-alone scalers and similar devices. These will have
a single device node that a driver can send data to, then the data is
transformed in some way and you get back the result.

How to enumerate pixel formats in this case? If you set the input pixel format
to X, then the set of possible pixel formats on the output might be different
compared to input pixel format Y. Ditto if you enumerate input pixel formats
based on a selected output pixel format.

In both cases you still want to enumerate all supported formats, but you also
want to know which match the chosen format 'on the other side'. I propose that
we add a new flag to the struct v4l2_fmtdesc flags field: V4L2_FMT_FLAG_INVALID.

This is only set if the enumerated pixel format is not currently valid compared
to the chosen opposite pixel format. In principle this flag is only set for
such input and output video nodes and not for others.


Based on this lengthy analysis I propose the following:

1) We add a v4l2-subdev API to setup bus configurations. This configuration
comes from the board specification.

2) Each sub-device will support a list of data formats depending on the bus
configuration. This is stored in the subdev driver.

3) A data format can either be setup explicitly through a media controller
or it is negotiated. You may need to do the matching via a callback to
platform code.

4) Restrict VIDIOC_ENUM_FRAMESIZES/INTERVALS to what is available on the
sensor (or video source/sink in general). It should have similar semantics to

5) We need a new API to setup a sensor directly and not rely on S_FMT to
do that for us since that is ambiguous when there is also a scaler in the
pipeline. A new API to setup digital video timings is being worked on and
can probably be used for this.

6) All drivers will enumerate those pixel formats supported for the currently
selected data format of the DMA engine. Optionally, some drivers may extend
that to other data formats as well as long as they know how to setup the board
to support the new pixel format.

7) For input/output video nodes the enumerated pixel formats will include a
new flag that tells the application whether the pixel format is invalid if
it does not match the pixel format on the other side. Obviously, the video
node will only work if both pixel formats are valid.



Hans Verkuil - video4linux developer - sponsored by TANDBERG Telecom
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[Linux Input]     [Video for Linux]     [Mplayer Users]     [Linux USB Devel]     [Linux Audio Users]     [Photos]     [Yosemite Photos]     [Linux Kernel]     [Linux SCSI]     [XFree86]     [Devices]     [Yosemite Backpacking]     [Linux Input]

Add to Google Powered by Linux