How Your Smartphone Knows You’re Binge-Watching “Narcos”

January 26, 2016 • Blog Post • By Atlantic Re:Think


  • Audio content recognition (ACR) is an audio identification technology used in TV apps to cue related content on smartphones and tablets, such as quizzes, polls and games
  • The second-screen experience, combined with the Internet of Things, will provide companies with an enormous amount of data on consumer behavior that can reveal insights into what works, what sells and what doesn’t

If you’re one of the many people who watch TV and use your smartphone at the same time, you’re the target for “audio trigger” technology

Long ago in digital time, when Shazam debuted in Apple’s brand-new App Store in 2008, its uncanny ability seemed to match the magic in its name. Somehow, it could tell you what song you were listening to right now, along with the artist and composer. The underlying technology, a form of automatic content recognition (ACR), used the phone’s built-in microphone to record a clip, run the resulting “acoustic fingerprint” through its database for a match and then—Shazam!—flashed the answer back to your screen.

That was then. Unleashed by mobile connectivity and cloud technology, ACR companies, including Shazam, are now applying the technology not only to music but also to live events, retail stores, movies, programs and videos on cable, satellite and broadcast channels as well as content streamed from Internet services such as Netflix and Hulu. The 2016 version of ACR is capable of lighting up our digital devices with content relevant to just about wherever we are and whatever we’re watching, listening to or doing at virtually every moment of every day. The second-screen experience is meant to inspire more engagement and a richer experience for the consumer, but it also represents an important commercial platform for the ACR companies, for their media partners and for brands.

ACR-based TV apps use audio triggers—“digital watermarking” placed into content before it is distributed—to cue related content on smartphones and tablets, such as quizzes, polls and games. Such watermarked content “allows you to have another conversation with the consumer,” says Misha Greenberg, CTO of Signal360, a company that does on-site, proximity marketing at sporting events and retail outlets. More recently, the company has begun working with TV networks as well.

The Internet of Things is key

A Cincinnati-based startup called LISNR, which created the world's first and only data-over-audio solution for brands and content producers, uses proprietary ultrasonic Smart Tones to prompt related content on smartphones at live events and retail outlets. LISNR is also working with a major consumer packaged goods company to deliver second-screen experiences during the Rio 2016 Olympics this summer. “Today, broadcast is a minute part of our business,” says co-founder and partner Chris Ostoich. “But in the future, I think it—and stuff we’re doing with the Internet of Things—will probably be the largest drivers of the business.”

IoT technologies, which will eventually collect and analyze sensor-driven data from virtually every object and appliance in our lives—from car engines to industrial infrastructure to home thermostats and security systems—promise to multiply the ways people can engage, understand and interact with the world. And, not incidentally, IoT will provide ACR companies an enormous amount of data on consumer preferences and behavior that, with advanced analytics, can reveal critical insights into what works, what sells and what doesn’t.

The key to ACR’s growing utility lies in the combination of mobile connectivity and cloud technology. ACR-based apps rely on smooth, intuitive operation. Second screens would be impossible without robust Wi-Fi and cellular connections, and putting the apps’ processing power and content in the cloud not only makes fast, fluent streaming possible and virtually ubiquitous, it also eliminates the need for ACR companies and their content partners to buy and maintain all the hardware that would otherwise be required. The desire to make Disney Movies Anywhere accessible to Amazon, Microsoft and Apple devices was among the reasons Disney and its ABC Television Group decided last year to base all its content in the cloud.

The Syfy channel started using ACR technology back in 2012, when it built its Syfy Sync app, which allows viewers of one show, “Face-Off,” go behind the scenes, answer trivia questions and tweet directly from the app. The company has since applied Syfy Sync to more of its programs. “It’s a way to lengthen the run time of the show and widen the context,” says Matthew Chiavelli, Syfy’s SVP for digital. “Super-fans are able to dig in in a way they couldn’t before, even on the website. It definitely deepens engagement.” The app triggers second-screen content no matter how someone is watching, whether on-demand, via DVR or from an Internet stream. “That was key for us—we didn’t just want to limit it to live viewing.”

‘Sharknado’ as a tech innovator

Syfy first combined ACR and IoT technology in 2014 for its movie “Sharknado 2: The Second One.” In part simply to demonstrate the channel’s commitment to innovation, Syfy teamed up with Phillips Hue, maker of “smart” light bulbs. "Anytime someone was killed, all your lights went blood-red,” says Chiavelli, who adds that while the smart-light-bulb audience may still be confined to early adopters, the innovative experiment also “got sectors of the press, particularly in the tech world, talking about “Sharknado 2,” which wouldn't have happened otherwise."

The most practical and commercial promise of ACR is to create ads precisely targeted to receptive consumers. “I want second-screen apps to learn from my behavior so the information provided is more useful,” says Peter Minnium, president of Ipsos Connect, the brand-communications arm of market-research firm Ipsos. “Wouldn’t it be wonderful if I could press a button in the app and buy the soundtrack, or get relevant coupons, or buy or do other things I’m interested in?”

According to a 2014 Nielsen study, 84 percent of smartphone and tablet owners already use a second device while watching TV, which is why so many networks and other content providers have been launching second-screen apps. But it is still a long way from reaching its commercial potential.

The key to doing that, says Minnium, is in data analytics. “With second screens we’re adding an unbelievable creative canvas to work within to make even more engaging experiences,” he says, “but the challenge has always been the ability to act on the data and information in hand. That gap between brands and consumers can only be filled by working with smart technology partners.”

Big Data changes everything. HPE Software gives you the power to transform it into actionable intelligence so you can capitalize on new opportunities and solve real problems in the moments that matter.

Hewlett Packard Enterprise is pushing the Internet of Things forward by building a new class of systems that compute and analyze data where it lives—everywhere.


The Future Is Now, and Other Lessons From New Scientist Live

Blog Post

HPEs Spaceborne Computer Successfully Powers Up in Space and Achieves One TeraFLOP

Blog Post

Take Control of Large Public Venues Through Mobile Engagement

Blog Post