Macworld

With the Apple Watch Series 12 and Ultra 4, Apple is introducing some neat new audio-based AI features that seem genuinely useful and exciting.

The new S11 processor includes hardware that makes it possible to continually listen and (lightly) process audio without ruining battery life, and Apple is using that to provide four new AI-powered features. It calls these features “Audio Intelligence” because Apple can’t help but brand everything.

  • Full-time Shazam: The watch can detect music playing and automatically identify it using Shazam, displaying it on widget in the Smart Stack. Apple Watches already do this on demand, but the new thing here is that it does it continuously in the background like Pixel phones.
  • Sound recognition: The same feature you can get on HomePods and iPhone now comes to Apple Watch. You can get notified visually when your watch hears sounds such as doorbells, fire alarms, sirens, etc. You used to need your iPhone nearby to use this feature, but the Series 12 and Ultra 4 can do it independently.
  • Live Rewind: Double-tap the digital crown to see a text readout of the last 15 seconds of what was said around you. You can then ask Siri about it or save it to the Siri app.
  • Siri Recap: Get a high-level summary of an entire conversation in the Siri app, so you don’t forget specifics or need to take notes.

The first two of those features don’t really bother anyone. Those are existing features, and the only new thing is that the Apple Watch can do them in the background without iPhone intervention. The last two will be available later this year as a beta, and those are the ones that have people freaking out. Are people’s Apple Watches going to be recording you all the time?

Apple has issued a Audio Intelligence Security Overview white paper to help users understand how all this works and how it protects everyone’s privacy. It’s worth a read, but here are the highlights.

How Live Rewind works

On the Apple Watch Series 12 and Ultra 4, audio from the microphone is continuously flowing into a little buffer in the Secure Enclave. This actually happens on lots of Apple products; it’s how they know if you’ve said “Hey Siri” or not. This is a rolling buffer, where new data continuously replaces old data.

The Secure Enclave is a completely separate hardware block that is not accessible to the rest of the operating system, or any apps, or even Apple. It’s where your Face ID and Touch ID data is stored, for example.

Live Rewind will show you what you missed—privately.

Apple

When you double-press the Digital Crown, the last 15 seconds of that buffer is sent via an encrypted secure transfer to your iPhone, where an AI model transcribes it. This happens in the iPhone’s Secure Enclave, so the audio cannot be accessed by the rest of iOS, or any app, or the user, or even Apple. This is just a straight voice-to-text transcription; it doesn’t identify or label people who are speaking. The audio is then immediately discarded.

The text of the translation is then sent back to Apple Watch where it appears on the display. It is not saved unless you ask Siri about it or save it to the Siri app to revisit later. The Siri app’s data is end-to-end encrypted and synced between devices, and users can control whether conversations are automatically deleted after 30 days, 1 year, or saved forever. Apple cannot decrypt this data, even with a court order—only the user has the key.

And one more thing, an audible chime plays from the speaker on your Apple Watch when you implement Live Rewind, even if your Apple Watch is on silent or you have headphones connected.

So that’s how Live Rewind protects your privacy and those around you. Raw audio data can’t ever be accessed by the user, the system, Apple, or any apps. It’s just straight translated to text, which is then optionally saved in the end-to-end encrypted Siri app.

How Siri Recap works

Siri Recap works in a similar way, but there’s more going on. You’ll be able to leave it on all the time, only when you trigger it manually, or specify certain places or times when it can’t operate.

First, a tiny little lightweight model simply detects if a conversation is happening at all.

If it is, that encrypted and secure audio buffer is sent to the iPhone (via an encrypted data transfer) for processing. It is transcribed to condensed text, and the raw audio is then immediately discarded. It is never accessible to the OS, apps, the user, or Apple.

This transcribed text passes through a safety filter than will remove potentially harmful terms. Then it is encrypted and sent to Apple’s Private Cloud Compute servers to be summarized, along with relevent contextual data that can make the summary better (such as what music you might be listening to, a relevant calendar entry, or high-level place information like what city you’re in).

Apple

Private Cloud Compute generates a title and summary with key points. It removes any specific identifiable or sensitive info such as place names or financial data, while leaving generic places names like “park” or “grocery store.” That summary is then sent back to your devices, encrypted, where it is stored in the Siri app in a new Recaps tab. All data on Apple’s Private Cloud Compute servers is always discarded immediately after processing; Apple never saves any of it.

The summaries in the Recaps tab are deleted automatically after seven days unless you choose to save it.

Like Live Rewind, Siri Recap does not create a recording, does not identify speakers, and does not even create a verbatim transcript. No audio is ever accessible at any step of the process by the user, any app, or Apple.

Finally, Live Rewind and Siri Recap are not available for users under the age of 13. For users under 18 on a Child Account, access to Live Rewind and Siri Recap must be turned on by a parent or guardian.