Re: [Gegl-developer] Path to the GIMP

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Ok. I had a response to this partially written, but it got eaten somehow. I am very upset about this (I lost hours of work!!)

Calvin Williamson wrote:
I think I just dont understand your notion of "image". Let me show you what I think they are with some specific
examples. Then maybe you can say if the way you are thinking
is different

ok. Yes, what I am thinking is different, it a subtle but very important point. An image is a bounded region, composed of pixels. The primary problem I see with your approach is that you are trying to treat images like ops. Very early on (at gimpcon) you and pippin both made it very clear that edges should be images and nodes should be ops. I didn't quite understand the power of that statement when you made it, but I went with it anyway, trusting that you two were right.

Now though, you seem to be doing something completely different. Let try and explain:

Ill just show how you might set up a projection stack for gimp using images and ops and gobject properties.

My viewpoint goes like this:

/ \ L3 combine2
       /  \
L2 combine1 / \
         L1  P

Each combine is an GeglOp, everything else is "data" (not a GeglOp). Other "data" besides the above image data (eg mask image data) is not shown but each combine op will have other kinds of data that feeds into it as well (opacities, modes, or whatever)

Images are edges, not nodes.

So combine ops will have two "image" input properties: These are two regular gobject style properties called "source1" and "source2".

(Think source2 over source1 here, so source2 is foreground
and source1 is background)

Also each combine op has one "image" output property. Again this is also a regular gobject style property and is called "dest". Now output properties are read-only gobject properties, and input properties are read-write gobject properties.

The reason for this is because output properties are only
to be set by the corresponding op as a result of the "evaluate" method. However inputs are obviously settable
properties of the op objects (and settable by g_object_sets)

Now you set up your ops like this:

GeglNode *combine1 = g_object_new(GEGL_TYPE_COMBINE_OP, "source1", P,
                                  "source2", L1,
GeglNode *combine2 = g_object_new(GEGL_TYPE_COMBINE_OP, "source2", L2,
GeglNode *combine3 = g_object_new(GEGL_TYPE_COMBINE_OP, "source2", L3,

So far all the "source2" properties are hooked up. Now hook up the output properties for each node to the appropriate input properties of the combines like this:

gegl_node_connect(combine2, "source1", combine1, "dest");
gegl_node_connect(combine3, "source1", combine2, "dest");

ok. The first problem I have with your example is that you are clearly treating images and nodes differently in your api, but your diagram makes no distinction.

Let me try to adjust your example to better fit what your api seems to be doing.

My format will be:
| connection


This distinction is necessary because clearly you are saying that images and ops are different beasts.

This example doesn't do your api justice however, because you have the problem that sometimes, your property is an image, and sometimes your property is an op, and they are treated differently

This tells the combine2 and combine3 ops that when they do their "evaluate" method, they should get the input property called "source1" from the output property of the corresponding node they are connected to along the source1 line (in this case they are connected to the "dest" output property of another
combine op)

Now you call
gegl_node_apply(combine3, "dest");

and pass it the name of the output property you want it to compute
(You should also set a ROI on this output property as well before
doing this, and that is left out here too)

There is no reason that an op has to be associated with a particular ROI. In fact, multiple users may want different ROI's computed from the op simultanously! Clearly, the ROI should be assocated with the resultant image.

Here, let me try and start my own example now.  Lets start with yours:

    /    \
   L3  combine2
        /  \
       L2  combine1
           / \
          L1  P

Only this time, let L3, L2, L1 and P are leaf nodes. (they could be files, frame buffers, networks, or a v4l or GStreamer source).

L3, L2, and L1 are allocated with (as an abstract example): g_buffered_node_new("inital_image",gegl_image1, "origin", point1);

and nodes are connected in a way similar to what you described. This time the diagram accuartly represents the connections between nodes. Now, every time someone wants a particular region as an image they call something like:

GeglImage* gegl_graph_get_image(roi_rect, other_rendering_parameters (antialising hints, etc)).

And a GeglImage object is immediately returned. This GeglImage object doesn't actually contain tiles, it just behaves like a BufferImage, and retrieve tiles from the underlying op as soon as you request a tiles using:


Where hints can contain information like, you are going to need a certain sized neighborhood, or the pattern you are planning to iterate over the image, which lets you mimimize latency and take advantage of parallel hardware. The roi is implicit in the size of the image that you requested. I am going to try to go over the advantages of this api one by one.

-Sytactic equivlence (Differences between ops and images)
First of, this method means there is no sytactic difference between files, network sources, and in-memory tiles. It makes the node construction api simplier.

Further, an op can (though it doesn't have to) do away with things like bounds. And image, when connected to an op, has bounds, a specific context in which it was rendered (say, with antilaising, a resampling algorithm, or with a particular resolution), to which the op should know nothing about.

-Ease of passing hints
There is a well defined way of passing hints to the rendering system. The hints are able to be passed exactly when they are needed (and thus, when they are known).

-multiple simultanous renderings
Using this method, a user could request several images from the graph (essentially queueing up multiple renderings) and the rendering system can take care of minimizing the amount of recaculations needed. This is especially useful, because it is trivial to extend this api to support resolution independence. (simply allow one to specify the rendering context to the get_image function).

-asyncronous ops (latency)
When you have operators that are acting asyncronously (say, they are in a different process, on a different thread, or over the network) it is important to minimize the latency to getting the tiles to the node. This is because the op can start processing input before the child nodes are entirely finished with the image. (using your model, the image needs to be totally calculated before the next op can being processing it).

-tile scheduling
Tile scheduling becomes easier because you can be sure that when a tile appears in the queue, with its hints, you know that someone specifically requested that tile and the direct prerequisites of the tile appear right next to it in the queue. If you are calculating entire images at once the prerequistes for a tile are not gaurenteed to be "close-to" its prerequisits, so even if you tried to bolt on some kind of tile scheduler, it would have to do a lot of work figuring out the prereqs of a tile to get those tiles out as quickly as possible.

Not really applicable to your specific example, but what happens when you try to do algebra? Using your example, you need to make special cases for images because images might be stored in properties to nodes.

-lower memory usage
When doing multiple renderings are needed your memory usage is much lower, because the multiple renderings can be scheduled simultanously.

Now your result is sitting in the op's "dest" output property when
you are done.

What happens if you want another rendering to be performed by the node?

So in this view images are just another kind of data moving through the graphs properties.

no, in this view, images are some kind of pseudo-node, and anything that deals with nodes is going to need to make all kinds of special cases in case its node is actully an image.

Now we certainly arrange for the set of combine ops to either operate in-place (so each dest is actually just the projection GeglImage P passed at the beginning) or you can have each op create dest images as it goes along the way and have those results cached,
so that if a layer changes, then the input property that corresponds
to that data is written as dirty and this propagates up the graph
to the root, and all the properties above that one are set dirty.

In-placeness is not really something that you need to worry about. Since the tiles are copy-on-write, just make every node do things "in-place" and if it needs to be copied (e.g. someone else is going to use the tile) it will be. And of course, property changes need to be propaged up the graph, but that is easy, because nodes know about their parents.

Anyway underneath the ops are asking images for their tiles
so and reading and writing to them, and placing pieces of
the images in caches and listening for changes to that same data so that they can mark properties dirty if they change in some way.

only, an op should not be caching it's inputs, only it's outputs. You are implying that the ops are caching inputs (and probably outputs too) since there is no way for the image it has to communicate with the underlying op. Caching inputs is inefficient. First off, the op doesn't have any real control over its inputs, so it shouldn't be in charge of caching them. Second, caching inputs is inefficent, since a single op may have multiple outputs and each of those outputs would be cahing the same data.

Is this very different from the way you are thinking?

Yeah, pretty different.

No, I think I meant what I said.  GimpImage is a stack of GimpLayers,
conceptually, and a GimpLayer is a combition of a GimpDrawable and some
associated blending info.  For all the reasons above, nothing should
keep around a GeglImage unless it is actually processing that region (as
this locks up some resources).  Thus, GimpImage maps to a tree of ops.
GimpLayer maps to a blend op and a leaf op.  A GimpDrawable maps to a
leaf node, which probably just contains a bunch of tiles (a la' a
gegl-buffered-image, but with all the op stuff connected to it).

I dont see why there is a need for leaf nodes. And how is the
drawable the output of a painthit if it is a gegl-buffered-image,
unless you additionally copy to and from it to get your paint hit on the drawable.

because a layer is not always a collection of pixels. Sometimes it is a text layer, or a vector layer, or an adjustment layer, or an effect layer. Ignore that thing about gegl-buffered-image. It was confusing and not what I meant.

There certainly is a need for there to be a certain kind of image that is capable of participating in operator caches, yes, but wont you want to have many different ops writing to a drawable?

Right, and the best way to handle that is with an op that can produce multiple, writable images, or that has procedural drawing calls. Requsting an image from an op is a way of specifing that you are interested in manipulating that region. The op can handle all the magic of locking minimal regions.

| I think most of the work in bringing in a new image type is
| related to replacing TileManagers and PixelRegions code all over
| gimp to use GeglImages and GeglOps instead.

Why do you think that?

I just mean the the typical place where an op in gimp
is set up looks like this usually:
PixelRegion src1PR;
PixelRegion src2PR;
PixelRegion destPR;

pixel_region_init(&src1PR, tile_manager1, x,y,w,h); pixel_region_init(&src2PR, tile_manager2, x,y,w,h); pixel_region_init(&destPR, dest_tile_manager, x,y,w,h);
blah_region(&src1PR, &src2PR, &destPR  ...);

Then you have the result in dest tile manager when you are done.
All these places in the code have to be changed to set up an op and then apply the op instead of what they do now.

yeah, but all the code that uses this stuff needs to change too.

And there is more to getting the gimp to use gegl than just getting gegl squeezed into the gimps processing model. To really take advantage of some of the cool stuff we can do with gegl, the gimps processing model needs to change (otherwise, what is the point of gegl?)

Beyond all this, the methods I am trying to outline here consider things like, networked (eg. beowulf) processing, multiple processors, real-time image sources, low latency processing, resolution independence. Now I don't expect all these features to be implemented right away, but when they are, I don't think the api will need to change all that much.


[Video For Linux]     [Photo]     [Yosemite News]    [Yosemite Photos]    [gtk]     [GIMP Users]     [KDE]     [Scanner]     [Gimp's Home]     [Gimp on Windows]     [Steve's Art]     [Webcams]