In recent years, a collection of new techniques which deal with video as input data, emerged in computer graphics and visualization. In this work, we report the state of the art in video-based graphics and video visualization. We provide a review of techniques for making photo-realistic or artistic computer-generated imagery from videos, as well as methods for creating summary and/or abstract visual representations to reveal important features and events in videos. We also provide a new taxonomy to categorize the concepts and techniques in this newly emerged body of knowledge. In addition, we give a concise overview of the major advances in automated video analysis, as some techniques in this field (e.g. feature extraction, detection, tracking and so on) have been featured in video-based modelling and rendering pipelines for graphics and visualization.
Video-based graphics is concerned with the manipulation and rendering of graphical models built from video data, instead of, or in addition to, traditional object representations. Its primary aim is to make creative computer-generated imagery from videos for artistic appreciation and entertainment. There are two main strands in this field, video refashioning and video-based scene modelling. The former typically involves manipulation of the geometrical entities (e.g. object
shape and distribution) and optical attributes (e.g. lighting, colour) of an input video, producing a new video that captures the essence of the input but in an expressive art form, such as by relighting the video scene with imaginary lights or
mimicking hand-drawn cartoon animation. The latter, meanwhile, involves reconstruction of a three-dimensional (3D) object or scene model captured by the input video, allowing such a model to be manipulated, combined with other models,
and rendered in the same way as conventional graphical models. The primary motivation for video-based graphics has been consumer multimedia applications, and the film and game industries.
Video visualization is concerned with the creation of a new visual representation from an input video to reveal important features and events in the video. It typically extracts meaningful information from a video and conveys the extracted information to users in abstract or summarized visual representations. Video visualization is not intended to provide fully automatic solutions to the problem of making decisions about the contents of a video. Instead, it aims at offering a tool
to assist users in their intelligent reasoning while removing the burden of viewing videos. This aim justifies deviation from the creation of realistic imagery (as found in videobased graphics), and allows simplifications and embellishments, to improve the understanding of the input video. In many ways, the subject of video visualization encompasses some aspects of video-based graphics. The development of the subject has been heavily influenced by many applications
in science, medicine, sport and security.
 |
| Three typical computational pipelines for video-based graphics, video visualization and video analysis. |
Categorization by Goals and Output Data Types
 |
| Two classifications by goals and type of output data. |
Categorization by Input Information and Levels of Automation
 |
| Another two classifications by input data and levels of automation. |
Video Analysis
- Optical flow estimation
- Image segmentation
- Feature extraction
- Recognition and detection
- Tracking
- 3D reconstruction
Video-based Graphics
 |
| Painterly rendering of a flower, from top-left in clockwise order: watercolour, Van Gogh, impressionism, abstract, pointillism and flower styles. |
- Mosaicing
- Panoramas
- Cut-outs
 |  |
| Dynamic still and clips. The transparency of additional poses is based on their importance. Most informative poses (i.e. motion extreme points) are completely opaque. | A frame from a video mosaic. |
 |  |
| Background and motion panoramas of a jumping athlete. | Dynamosaic of a waterfall video. |
- Video textures
- Video relighting
- Video retargeting
- Video-based modelling
 |
| Twelve photographs of themoon. (a)Unmodified photograph. (b) Control arrows to define a field of incident light direction. (c) Resulting composite photograph. |
 |  |
| Importance preserving image retargeting. The three important elements of the image are preserved as the image size is reduced. | Improved seam-carving accounts for frame changes over time, creating a consistent viewing experience. |
Video Visualization
 |
| Segments of a video. |
- Another video or an animation
- Video navigation
- Video montage and video synopsis
- Video skimming
 |  |
| Video browsing using interactive navigation summaries. | Video browsing by direct manipulation. |
- A large collection of images
- A single composite image
 |
| Reassembled depictions of key frames: (a) Salient stills compute the background from a number of frames and add local information about relevant events, (b) A similar approach is taken by dynamic video synopsis with the focus on the concurrent display (lower image) of events at different times (upper images), and (c) A video collage rearranges elements of different key frames in a new image of arbitrary shape. |
- Additional information and actions
- Enhanced stills
- Video abstraction
 |
| Enhanced stills: (a) Schematic storyboards enhance the displayed key frames with additional information on characters and camera movement, (b) Additional dynamics glyphs are used to enhance understanding, and (c) Trajectories extracted fromvideo are grouped together and depicted by schematic summaries. |
 |  |
| Interactive video cubism. | Video visualization - changes that remain for a period. |
 |
| Visual signatures in video visualization. |
Observations
- The developments in video-based graphics have been following a certain trend in parallel with that in the digital entertainment industry. It is driven primarily by the demand for novel and creative digital contents as well as the need for consumer multimedia applications. This trend is expected to continue, and hence provide new inspiration and stimulus for further research and development. However, the focus will likely change from one subgoal to another, although new subgoals will surely emerge.
- Video visualization can have applications in many disciplines including science, engineering, sports, medicine and security. However, most of these applications share a common goal: to reduce the time needed for watching videos and to assist the users in gaining insight and making decisions in a cost-effective manner. Different output data types reflect the diversity of means to achieve such a common goal. With the rapid increase of captured video data, there will be a continuous increase in demand for video visualization to address the shortcoming of automated video analysis. The research in this area also faces a huge challenge of scalability in terms of space, time and interaction required for viewing visualization.
- Like artificial intelligence, automated video analysis is an ultimate ambition in computer science. Although the realization of such an ambition will require a long-term effort, the research and development in video analysis has resulted in a large collection of low- and high-level techniques. Many techniques, such as optical flow estimation and 3D model reconstruction, have already been adopted for pre-processing data in video-based graphics and video visualization. Many more are yet to be integrated into systems for video-based graphics and video visualization. Hopefully, the brief overview of video analysis in Section 3 will enthuse researchers to explore various techniques originally developed for automated video analysis.
Fundings
This work was supported by the EPSRC under grants EP/G006555 on video visualization. It was also supported by the Wales Assembly Government
for Research Institute of Visual Computing (RIVIC) and the German Research Foundation (DFG) as part of the Priority Program 'Scalable Visual Analytics'
(SPP 1335).
Publications