Project
To be updated with developments in recent years...
A Deep Learning Driven Active Framework for Segmentation of Large 3D Shape Collections
High-level shape understanding and technique evaluation on large repositories of 3D shapes often benefit from additional information known about the shapes. One example of such information is the semantic segmentation of a shape into functional or meaningful parts. Generating accurate segmentations with meaningful segment boundaries is, however, a costly process, typically requiring large amounts of user time to achieve high-quality results. In this paper we propose an active learning framework for large dataset segmentation, which iteratively provides the user with new predictions by training new models based on already segmented shapes. Our proposed pipeline consists of three components. First, we propose a fast and accurate feature-based deep learning model to provide dataset-wide segmentation predictions. Second, we develop an information theory measure to estimate the prediction quality and for ordering subsequent fast and meaningful shape selection. Our experiments show that such suggestive ordering helps to reduce users' time and effort, produce high-quality predictions, and construct a model that generalizes well. Lastly, we provide interactive segmentation refinement tools, helping the user quickly correct any prediction errors. We show that our framework is more accurate and in general more efficient than the state-of-the-art for large dataset segmentation, while also providing consistent segment boundaries.
Deep Learning of Spatial Features on Irregular Domains
We propose a method for applying deep learning on problem domains which exhibit an irregular spatial topology. By utilizing a graph-based representation of the problem domain, we can define the spatial convolution operation in the spectral domain of the graph Laplacian. We also explore the graph construction methods for specific problem domains and the graph coarsening methods used as an analogy to the spatial pooling operators of conventional convolutional neural networks. We are exploring the application of such graph-based deep learning on a wide variety of problem domains, from sensor networks to human behavioral analysis.
Deformable Segmentation with Arbitrary Initializations
We propose a unified approach to deformable model-based segmentation. The fundamental force field of the proposed method is based on computing the divergence of a gradient convolution field (GCF). Then, using this proposed external force field, we derive a more salient representation for contour evolution that does not only lead to global minimum through convex relaxation but also can achieve the same result using the conventional gradient descent with an intrinsic regularization. Moreover, nonlinear diffusion can be conveniently applied to GCF to improve its performance in dealing with noise interference. Additionally, the derived deformable model can work well in 3D or higher dimensions because GCF can be naturally extended to higher dimensions.
Integrated Segmentation and Interpolation
We propose an integrated segmentation and interpolation framework, which can handle 3D and 4D sparse data based on a novel level set scheme. In this new framework, the level set implicit function is interpolated by radial basis functions (RBFs), and its interface can propagate in a sparse volume using data information where available and RBF-based interpolation in the gaps. The interpolation process uses segmentation information rather than pixel intensities for increased robustness and accuracy. The method supports any spatial configurations of sets of 2D slices having arbitrary positions and orientations.
Automatic Seizure Detection in EEG
We propose an automated method for the analysis of epileptic EEG data employing the random forests (RFs) algorithm. Feature extraction is performed using a discrete wavelet transform to give time-frequency representations, from which statistical features based on the wavelet decompositions are formed and used for training and classification. We show that RFs can be applied for the classification of ictal, inter-ictal and healthy EEG with a high level of accuracy, sensitivity, and specificity.
Conversational Interaction Recognition
We use both 3D pose and face features to learn and recognize different conversational interactions. In this work, we utilize the Kinect sensor to extract 3D pose features and a combined local and global model to extract face features from normal RGB cameras. We demonstrate that both features can still be applied to effectively train classifiers, although they are contaminated with noises. The differences in interaction among different scenarios in our data set are extremely subtle. We investigate both generative and discriminative methods, and employ a subject-specific supervised learning approach to categorize the testing sequences into seven different conversational scenarios.
3D Human Pose Interaction Classification
We present a subject specific human conversational interaction classification using 3D pose features, and show that it is indeed possible to understand human conversational activity from observing gesture cues alone. We adopt Kinect sensors to obtain 3D displacement and velocity measurements, followed by wavelet decomposition to extract low level temporal features. These features are then generalized to form a visual vocabulary that can be further generalized to a set of topics from temporal distributions of visual vocabulary. A subject specific supervised learning approach based on Random Forests is used to classify the testing sequences to seven different conversational scenarios.
Intravascular Ultrasound Image Segmentation
Intra-vascular Ultrasound (IVUS) imaging is a catheter-based technology, which shows 2D cross-sectional images of the coronary artery. In this work, we present a shape prior based graph cut method for IVUS image segmentation, which does not require user initialization. The shape prior is generalized from multiple training shapes, rather than using singular templates as priors. Weighted directed graph construction is used to impose geometrical and smooth constraints learned from priors. The proposed cost function is built upon combining selective feature extractors. A SVM classifier is used to determine an optimal combination of features in presence of calcification, fibrotic tissues, soft plaques, and metallic stent, each of which has its own characteristics in ultrasound images.
Left Ventricle Segmentation in SPECT
We propose a novel spatio-temporal constraint based on shape and appearance and combine it with a level set deformable model for Left Ventricle (LV) segmentation in 4D gated cardiac SPECT, particularly in the presence of perfusion defects. The model incorporates appearance and shape information into a soft-to-hard probabilistic constraint, and utilizes spatio-temporal regularization via a Maximum A Posteriori (MAP) framework. This constraint force allows more flexibility than the rigid forces of shape constraint-only schemes, as well as other state of the art joint shape and appearance constraints. The combined model can hypothesize defective LV borders based on prior knowledge.
3D Human Pose Estimation
A hierarchical approach is presented to efficiently estimate 3D pose from single images. To achieve this, the body is represented as a graphical model and optimized stochastically. The use of a graphical representation allows message passing to ensure individual parts are not optimized using only local image information, but from information gathered across the entire model. In contrast to existing methods, the posterior distribution is represented parametrically. A different model is used to approximate the conditional distribution between each connected part. This permits measurements of the Entropy, which allows an adaptive sampling scheme to be employed that ensures that parts with the largest uncertainty are allocated a greater proportion of the available resources. At each iteration, the estimated pose is updated dependent on the Kullback Leibler (KL) divergence measured between the posterior and the set of samples used to approximate it. This is shown to improve performance and prevent over fitting when small numbers of particles are being used.
3D Human Pose Tracking
Representing articulated objects as a graphical model has gained much popularity in recent years, often the root node of the graph describes the global position and orientation of the object. In this work, we present a method to robustly track 3D human pose by permitting larger uncertainty to be modeled over the root node than existing techniques allow. Significantly, this is achieved without increasing the uncertainty of remaining parts of the model. The benefit is that a greater volume of the posterior can be supported making the approach less vulnerable to tracking failure. Given a hypothesis of the root node state a novel method is presented to estimate the posterior over the remaining parts of the body conditioned on this value. All probability distributions are approximated using a single Gaussian allowing inference to be carried out in closed form. A set of deterministically selected sample points are used that allow the posterior to be updated for each part requiring only seven image likelihood evaluations making it extremely efficient. Multiple root node states are supported and propagated using standard sampling techniques. We believe this to be the first work devoted to efficient tracking of human pose whilst modeling large uncertainty in the root node. The proposed method is more robust to tracking failures than existing approaches.
Contour Based Object Tracking
Tracking of objects and simultaneously identifying an accurate outline of the tracked object is a complicated computer vision problem to solve because of the changing nature of the high-dimensional image information. Prior information is often included into models, such as probability distribution functions on a prior definition of shape to alleviate potential problems due to e.g. ambiguity as to what should actually be tracked in the image data. However supervised learning and or training is not always possible for new unseen objects or unforeseen configurations of shape, e.g. for silhouettes of 3D objects. We are therefore interested and are currently investigating ways to include high-level shape information into active contour based tracking frameworks without a supervised pre-processing stage.
Shortest Path Based Interactive Image Segmentation
A method for user assisted segmentation of objects of interest has been developed, which combines point-based soft constraint on object boundary and stroke-based regional constraint. This combined approach is formulated as an energy minimization problem on a multilayered graph, which is solved using a shortest path search algorithm. The proposed approach allows efficient and effective interactive segmentation, which can be used in open or closed curves to segment a variety of images in different ways.
Blood Flow Modelling
Vascular diseases, such as stenosis and aneurysms, are often associated with changes in blood flow patterns and the distribution of wall shear stress. Modelling and analysis of the hemodynamics in the human vascular system improve our understanding of vascular disease, and provide valuable insights, which can help in the development of efficient treatment methods. In this work, we present a robust and semi-automatic modelling pipeline for blood flow through subject-specific arterial geometries. The framework consists of image segmentation, domain discretization (meshing) and computational fluid dynamics. All the three subtopics of the pipeline are explained using an example of flow through a severely stenosed human carotid artery.
TEXEMS: Texture Exemplars
We present an approach to detecting and localizing defects in random color textures which requires only a few defect free samples for unsupervised training. It is assumed that each image is generated by a superposition of various-size image patches with added variations at each pixel position. These image patches and their corresponding variances are referred to here as textural exemplars or texems. Mixture models are applied to obtain the texems using multiscale analysis to reduce the computational costs. Novelty detection on color texture surfaces is performed by examining the same-source similarity based on the data likelihood in multiscale, followed by logical processes to combine the defect candidates to localize defects.
Fast Dynamic Texture Detection and Segmentation
Dynamic textures can be considered to be spatio-temporally varying visual patterns in image sequences with certain temporal regularity. We propose a novel and efficient approach to explore the violation of the brightness constancy assumption, as an indication of presence of dynamic texture, using simple optical flow techniques. We assume that dynamic texture regions are those that have poor spatio-temporal optical flow coherence. Further, we propose a second approach that uses robust global parametric motion estimators that effectively and efficiently detect motion outliers, and which we exploit as powerful cues to localize dynamic textures.
Surface Inspection and Novelty Detection
Automatic surface inspection for quality control has largely employed gray-level image processing techniques, for example in textile and wafer inspection. There are rising demands in the quality control industry for color analysis to fulfill its vital role in visual inspection, e.g. in ceramic tile manufacturing. In this work, we first systematically review recent advances in surface inspection using computer vision and image processing techniques, particularly those based on texture analysis methods. Following that, we present our work on developing texture analysis techniques in application to the detection of abnormalities in color texture surfaces, in particular ceramic tile surfaces on which patterns are regularly of a random nature.
Geometric Potential Force for 3D Deformable Model
A novel 3D deformable model based on a geometrically induced external force field is proposed, which can be conveniently generalized to arbitrary dimensions. The external force field is based on hypothesized interactions between the relative geometries of the deformable model and the object boundary characterized by image gradients. The evolution of the deformable model is solved using the level set method so that topological changes are handled automatically. The dynamic interaction forces between the geometries can greatly improve the deformable model performance in acquiring complex geometries and highly concave boundaries, and it gives the deformable model a high invariancy in initialization configurations. The bi-directionality of the external force field allows the new deformable model to handle arbitrary cross-boundary initializations, and facilitates the handling of weak edges and broken boundaries. In addition, the new deformable model can effectively overcome image noise, by enhancing the geometrical interaction field with a non-local edge-preserving algorithm.
Active Contouring with Level Set Regularization
We present an extension of our MAC model to deal with the initialization dependency problem that commonly appears in edge-based approaches. Its dynamic force field, unique bi-directionality, and constrained diffusion-based level set evolution provide great freedom in contour initialization and show significant improvements in initialization independency compared to other edge-based techniques. It can handle more sophisticated topological changes than splitting and merging. It provides new potentials for edge-based active contour methods, particularly when detecting and localizing objects with unknown location, geometry, and topology.
MAC: Magnetostatic Active Contour Model
MAC model is a result of our most recent effort in developing new active contour models. The proposed external force field that is based on magnetostatics and hypothesized magnetic interactions between the active contour and object boundaries. The major contribution of the method is that the interaction of its forces can greatly improve the active contour in capturing complex geometries and dealing with difficult initializations, weak edges and broken boundaries. The proposed method is shown to achieve significant improvements when compared against six well-known and state-of-the-art shape recovery methods, including the geodesic snake, the generalized version of GVF snake, the combined geodesic and GVF snake, and the charged particle model.
RBF Interpolated Level Sets
We introduce the radial basis function (RBF) into active models and present a new level set formulation able to handle more complex topological changes, in particular perturbation away from the evolving front. This allows the initial contour or surface to be placed arbitrarily in the image. The proposed level set updating scheme is efficient and does not suffer from self-flattening while evolving, hence it avoids large numerical errors. Unlike conventional level set based active models, periodic re-initialization is also no longer necessary and the computational grid can be much coarser, thus, it has great potential in modelling in high dimensional space.
CACE: Charged Active Contour based on Electrostatics
This new framework by introducing particle dynamics into active contour propagation, resulting in a Charged Active Contour based on Electrostatics (CACE) model incorporates the advantages of both contour and particle based models. CACE can detect object boundaries via contour propagation under the influence of a bi-directional force field that simulates the electrostatic interaction between an image-derived point charge field and a charged contour. In other words, the force consists of boundary attraction and competition terms that lead the contour towards object boundaries. CACE is much faster and more efficient in convergence than CPM. More importantly, it eliminates CPM's tendency to sometimes result in open contours. CACE also has significant advantages over the geodesic and GVF geodesic snakes in that it is more robust to initial placement and is able to handle objects of more complicated topology, e.g. those with narrow parts.
CCM: Charged Contour Model
CCM is a new deformable model based on electrostatics and geometric contour propagation. It is motivated by electrostatic interaction between charges, and simulates the movements of a set of connected charged particles in an attracting electrostatic force field, which plays the role of external force in this deformable model. CCM detects object boundaries with a charged active contour that propagates under the influence of Lorentz forces in an image-based electrostatic field. We make use of level set representation allowing topological changes to be handled naturally. Also, we build on the center of divergence concept towards automatic initialization. CCM improves on both traditional Geodesic snake and the recently introduced Charged Particle Model (CPM), which is also based on electrostatics, as well as charged particle dynamics.
Region-aided Geometric Snake
An enhanced, region-aided, geometric active contour that is more tolerant towards weak edges and noise in images is introduced. The proposed method integrates gradient flow forces with region constraints, composed of the image region vector flow forces obtained through the diffusion of the region segmentation map. We refer to this as the Region-aided Geometric Snake or RAGS. The diffused region forces can be generated from any reliable region segmentation technique, gray-level or color. This extra region force gives the snake a global view of the boundary information within the image which, along with the local gradient flow, helps detect fuzzy boundaries and overcome noisy regions. The partial differential equation (PDE) resulting from this integration of image gradient flow and diffused region flow is implemented using the level set approach. We present various examples and also evaluate and compare the performance of RAGS on weak boundaries and noisy images.
Mammographic Image Segmentation
Due to their natural handling of shape variation and independence of operation (once initialized), deformable contour models (snakes) are appropriate for the segmentation of mass lesions in digital mammographic images. The extracted shape and texture information through contour based segmentation are useful in determining benignancy or malignancy. In this work, we investigate four important edge-based active contour models for mass lesion segmentation. Two of them are classic active contour models and the other two are most recent advances in active contouring.
Supervised Protein Classification
We investigate different approaches for the classification of proteins based on sequence data, employing supervised machine learning techniques. Among many other methods for protein classification, multiple sequence alignment of proteins within a known family has been a popular way to extract statistical information about a protein family that would not otherwise be observed to the human eye. Profile hidden Markov models (HMMs) are then built from multiple alignment of raw sequence data and random forests (RFs) are used to discriminate between two sets of proteins based on features (e.g., functional amino acid groups properties extracted from the raw sequences). Compared to artificial neural networks (ANNs) and support vector machine (SVM), the machine learning based methods achieve a higher classification accuracy.
Video-based Graphics and Video Visualization
In recent years, a collection of new techniques which deal with video as input data, emerged in computer graphics and visualization. In this work, we report the state of the art in video-based graphics and video visualization. We provide a review of techniques for making photo-realistic or artistic computer-generated imagery from videos, as well as methods for creating summary and/or abstract visual representations to reveal important features and events in videos. We also provide a new taxonomy to categorize the concepts and techniques in this newly emerged body of knowledge. In addition, we give a concise overview of the major advances in automated video analysis, as some techniques in this field (e.g. feature extraction, detection, tracking and so on) have been featured in video-based modelling and rendering pipelines for graphics and visualization.