Probabilistic Joint Image Segmentation and Labeling by Figure-Ground Composition

We propose a layered statistical model for imagesegmentation and labeling obtained by combining independentlyextracted, possibly overlapping sets of figure-ground(FG) segmentations. The process of constructing consistentimage segmentations, called tilings, is cast as optimizationover sets of maximal cliques sampled from a graph connectingall non-overlapping figure-ground segment hypotheses.Potential functions over cliques combine unary, Gestaltbasedfigure qualities, and pairwise compatibilities amongspatially neighboring segments, constrained by T-junctionsand the boundary interface statistics of real scenes. Buildingon the segmentation layer, we further derive a joint imagesegmentation and labeling model (JSL) which, given a bagof FGs, constructs a joint probability distribution over boththe compatible image interpretations (tilings) composed fromthose segments, and over their labeling into categories. Theprocess of drawing samples from the joint distribution canbe interpreted as first sampling tilings, followed by samplinglabelings conditioned on the choice of a particular tiling.