The Joke-O-Mat system demonstrates the applicability of ICSI’s state-of-the-art speech recognition, speaker diarization, and multimedia content analysis research to the domain of pop-culture participation.
Prefer to get your explanations on video? View a rehearsal of our three-minute 2009 ACM Multimedia presentation about Joke-O-Mat on YouTube.
When someone has seen a sitcom once (or twice), they might want to go back again to show a particularly funny scene to a friend, quote the sharpest punchline in a Facebook post, or perhaps create a YouTube montage of hilarious moments from a particular character. Joke-O-Mat enables quick, efficient browsing of a TV show, presenting the user with a video interface that includes navigation tools for the basic narrative elements, such as the scenes, punchlines, and individual dialog segments, along with a filter to choose only scenes with a particular protagonist, on top of a standard video-player interface. This allows users to selectively skip certain parts and to navigate directly to the desired elements or moments they remember from past viewings.
Joke-O-Mat explores the power of audio analysis for video processing, using acoustic event detections and speaker identification on the audio track to infer narrative elements based on genre-specific production rules.The system distinguishes laughter, music, and other noise as well as speech segments, then identifies the speech segments using pre-trained speaker models and identifies punchlines and scene breaks using a model of standard sitcom structure. The top punchlines are identified based on the length of the following laughter.
The first version of Joke-O-Mat won the 2009 ACM Multimedia Grand Challenge (the Yahoo! video segmentation task).
Fansourced Transcripts + Closed Captioning → Video Navigation
The Joke-O-Mat system was upgraded in 2010 to include keyword filtering and searching. Joke-O-Mat HD used both a speech- and speaker-recognition-based method to generate speech segmentations and a method that aligned the audio content with two forms of human-derived (HD) data, fan-sourced transcripts and closed captioning.
You can get the full Joke-O-Mat experience on our Joke-O-Mat Demo Page.
Joke-O-Mat in the News:
Our video-navigation interfaces, Joke-O-Mat and the Meeting Diarist, got coverage when they were demonstrated at an Intel R&D day: