(archive site)



One of the major current challenges in multimedia research is that of quickly and accurately finding events of interest in very large video collections. This requires efficient automated analysis of massive amounts of video that may vary dramatically in quality and composition: similar activities of interest may appear very different, while very different events may share many common elements.

AURORA: Content-Guided Search of Diverse Videos is a prototype system that includes components for analyzing, representing, and searching video content. ICSI is working on AURORA in collaboration with multiple institutions, led by SRI-Sarnoff. AURORA is funded by IARPA’s ALADDIN program (Automated Low-level Analysis and Description of Diverse INtelligence video), which aims to combine expertise in video extraction, audio extraction, knowledge representation, and search technologies in a revolutionary way, to create fast, accurate, robust, and extensible technology that supports the multimedia analytic needs of the future.

The AURORA team at ICSI focuses on the audio extraction aspect of the project; researchers at ICSI have been developing state-of-the-art techniques for using audio analysis to detect the occurrence of specified events in videos. Training a system to identify a particular event, such as a birthday party or a bike trick, that is defined semantically — i.e., training it to pick out something that would be conceptualized by a human being as a single, identifiable event that can be given a particular label — from a recording can be quite a difficult task, as the sounds (and visible objects) that make up recordings of different birthday parties may be quite different. In other words, there is a great deal of variability in the data.

To cope with this variability, we first train a system to recognize lower-level audio concepts, such as singing or clapping, by classifying them according to their acoustic features. We then train another neural network to identify the clusters of audio concepts that are associated with the events annotated on the videos in the training set, so the system can then in turn pick out those events in unannotated videos.


The test cases for projects in the ALADDIN program are centered around the TRECVid Multimedia Event Detection (TRECVID MED) Evaluations. The AURORA team participated in TRECVID MED in 2011, 2012, and 2013, and the AURORA system was among the top three each year.

Project Results

AURORA Publications


AURORA is a collaboration between ICSI’s Audio & Multimedia Group, Carnegie-Mellon University, Raytheon BBN, CyCorp, the University of Massachusetts, and the University of Central Florida, led by SRI-Sarnoff.

Researchers @ICSI (Current and Past):


Funding for AURORA is provided by the ALADDIN Video Program at the Intelligence Advanced Research Projects Activity (IARPA), under Contract D11PC20066 (awarded to SRI-Sarnoff). The opinions, findings, and conclusions described on this website are those of the researchers and do not necessarily reflect the views of the funders.