Research
This page describes some of the projects I've worked on over these past few years. They are listed in reverse chronological order.
Interactive Retrieval of Activity Videos
In video retrieval of activities, there can be a very large number of queries users can make since there are so many activities in the world. Thus training classes a priori for retrieval is a daunting task. Furthermore, user's notions of what is relevant to their queries are subjective. (For example, "nice basketball shots.") Thus even if we could train all possible activity classes a priori, there would be no way to capture the subjective views of a user. Then it appears that having the user interact with search systems would be the only way to fully understand what it is the user wishes to query for. Relevance Feedback is a framework from Information Retrieval that has existed since at least the 1970s that can provide the needed user interaction. The idea is to first provide the user with a set of initial retrieval results and ask the user how relevant the results are. After getting this user feedback, the system would be able to better understand what the user wants and retrieval results could be refined. This process of learning from user feedback could be repeated as many times as the user desires to improve retrieval results.
The only problem is that when searching complex data such as activity videos, there may need to be a lot of user feedback for good performance. This is because without knowledge of the world, a computer would need more data from user feedback to effectively learn. Transfer Learning is a formulation from Machine Learning where knowledge from "source" classification task(s) can be used to supplement the learning of a "target" classification task when training data is scarce in the target task. For example, say we wanted to learn the classification of "basketball" videos but had only a handful of training examples, but had a lot of training data for "volleyball" classification. Then we could use the knowledge gained from volleyball classification to supplement the learning of basketball videos. In this case, the volleyball task would help in learning basketball because of the relatedness or similarity of the two activities relative to other classes of activities.
Still frames from volleyball and basketball videos showing their similarity. Thus Transfer Learning can be used between the two classes.
Then if system designers could build a set of source classification tasks for various activities (which account for only a small fraction of all activities that could be queried), Transfer Learning could be used to reduce the required amount of user feedback. The key problem in applying Transfer Learning to Relevance Feedback is the automated determination of which source tasks are related to that target task (user query). In our work, we explore strategies for determining whether a source task is related enough to the target task (user query) in the application of video retrieval. This is also to our knowledge, one of the first works to explore the combining of Transfer Learning in general to Relevance Feedback search. We are testing our methods on the YouTube Action Dataset as well as our own collected videos from the Internet. (All of these videos pose interesting challenges as they show a high amount of variation in quality, camera motion, and lighting.)
Some sample clips of videos for testing retrieval of activities.
Counting "Fly Spots"
For this project, Christian Shelton and I developed an algorithm for an Entomology study. The goal was to determine the trends in fly populations in farms. The method for measuring the frequency of flies was to leave index cards in places such as barns and have the flies that land on the cards leave their waste. This would result in brown or yellowish "fly spots" being marked on the cards. After leaving the cards out for some time, one would be able to count the number of spots on the card and infer that a lot of spots on the card would mean a lot of flies were at the farm.
The only problem was that the graduate students on the project would have to count the spots manually (which was labor intensive). We developed an algorithm for automatically counting the number of "fly spots" on scanned images of these cards. The system essentially works by considering the average RGB colors that the spots tend to have and then filtering out "non-fly-waste pixels", performing a connected components analysis on the image, and then considering the symmetry of the connected components. Despite the simplicity of this approach, the algorithm work quite well with a correlation of 0.97 to human expert counts in an extensive field test on a real dairy.
Example images of scanned cards with fly waste and annotation of spots found by our algorithm. (Yes, I scanned these in myself.)
Face Recognition Across Pose
Face recognition methods such as the well known Eigenfaces have been found to work very reliably. However, when the faces are not canonically alignment and/or 3D rotation is present, recognition performance drops significantly. To address this problem, a system for automatic alignment and learning of interpose appearance relations between same identity faces was developed.
Essentially, each face would be broken down into patches similar to the work of Kanade and Yamada and SVMs would be used to learn the appearance relation of each corresponding face patch. These SVMs would then be used to score different pose images of faces on how likely they are of the same person. Alignment of faces was done by first using SVMs to learn the typical appearances of face patches from canonically aligned images in the same pose. The learned SVMs for alignment then served as a scoring function for how aligned a given face image was and a coordinate ascent over the parameters of scale, translation, and in-plane rotation was used to align novel images. Evaluation of the system was done using the CMU PIE Database. For more details on this work, please see, "Face Recognition and Alignment using Support Vector Machines."
Example patches used in our system.
Face Poses in the CMU PIE Database
Fabric Drape Prediction using Neural Networks
This work was mainly an application for textiles research. The goal was to predict certain charateristics of how a piece of fabric given its mechanical properties would drape over a rigid surface. The particular characteristics were the "drape coefficient" and "circularity" which are measurements obtained from a device called the Cusick Drape meter. Two neural network architectures were compared in their ability to learn the drape coefficient and circularity given seven mechanical properties. Those interested may find more details of this work in, "Neural Network Models for Fabric Drape Prediction."
Cusick Drape Meter
Image of a piece of fabric being measured with a Cusick Drape meter.