Hands-On Apple 249: Explaining Audio Intelligence

0:00You're in a meeting and somebody rattles off a name or a room number, and it's gone before you've actually finished writing the first word. Or perhaps you're at a noisy diner, and you catch every third word of a story that you're supposed to react to. This Friday, the new Apple Watch ships with a fix for both of those problems, and a privacy conversation attached to it. Double press the digital crown in the last 15 seconds of what was said shows up as text on your wrist. Leave the other feature on all day, and well, Siri will write up your conversations for you. Sort of. Today, we are walking through audio intelligence, what it is, how it works, and where the lines are. Stay tuned. This episode is brought to you by Trusted Tech. Trusted Tech is reviewed more than $800 million in Microsoft 365 licensing spend and returned over $100 million to client budgets. On average, that's about 12% of every Microsoft 365 dollar going to waste.

1:03And licensing waste doesn't stay small. Left unmanaged licensing drift grows 3 to 7% a year, and Microsoft's newer contracts can lock you in for years. Trusted Tech's free Microsoft 365 licensing consultation at Trusted Tech.TeamSlashTwit shows you exactly where you're overspending and how to fix it before your next renewal locks it in. It's not just numbers on a page. Sensky services runs pest control and lawn care crews across the country. After a Trusted Tech review, they unified 1400 devices, right-sized 13% of their Microsoft 365 seats, and uncovered more than 1,000 licenses they didn't even know they were paying for. Trusted Tech doesn't stop at the cloud. The team also runs deep Microsoft 365 tenant assessments, delivers certified support services that save up to 52% versus Microsoft unified support. Handles on premise Microsoft licensing and offers Azure cloud consulting to cut infrastructure costs. Licensing, support, on premise, or Azure. Trusted Tech does it all when it comes to Microsoft.

2:13Avoid the licensing drift. Go to TrustedTech.TeamSlashTwit right now, submit the form and lock in your free Microsoft 365 licensing consultation before your next renewal costs you more. TrustedTech.TeamSlashTwit. Podcasts you love. From people you trust. This is Detroit. Hello and welcome to Hands-On Apple. I am Micah Sargenton. Today we are taking a look at the new audio intelligence features. We are taking a look at what Apple has said about the new audio intelligence features. We will take a deeper look when said audio intelligence features are available to us. But for now, I want to get ahead of the noise and the conversation so that we can understand what the heck is happening here and what you need to know.

3:14So first and foremost, we need to talk about audio intelligence because this is how Apple has defined these new terms. So let's figure it out because it's four features that are all tied to one chip and that one chip is on the Apple watch. So what is audio intelligence? First and foremost, it's important to understand that it does sound recognition. It is the umbrella term for four features on the Apple Watch series 12 and the Ultra Four that include sound recognition, automatic music recognition by way of Shazam, live rewind and Siri recap. So what why are these tied to these actual devices? What do we need to understand here when it comes to the reason that it is hardware locked? Well, it's because the new S11 chip has a secure X-Clave, not on-clave X-Clave.

4:25This is a walled off compartment that's in built into the chip where the microphone audio goes and then the operating system, your apps, you and even Apple cannot reach in and grab any of this. No audio recordings ever are ever stored. The X-Clave actually processes the audio and then immediately deletes it. So there's no file to share, there's no subpoena, there's no leak, there's nothing available to anyone to grab from the audio. It is just text transcription. And well, I'm no lawyer, but text transcription by way of an Apple Watch may or may not have context that's needed in a courtroom. Sound recognition is the first feature that we're talking about here. And with sound recognition before we get into the more privacy focused features, this means that on your Apple Watch on your wrist,

5:28the Apple Watch is listening for specific things. This was something that's always been available or it has long been available on the iPhone, but you can listen for sirens, for alarms, for doorbells, for a baby crying. And so it doesn't matter if your iPhone is across the house, people who are hard of hearing, people who are deaf, well, they are able to make use of a feature that will tell them more about what's going on in their environment. How do you actually go about switching it on? Well, as far as we understand, you do so by going into your settings on your iPhone, you got to accessibility, you go to sound and name recognition, you got a sound recognition, and then you turn on detect using Apple Watch. And then if it's already on for your iPhone, don't worry about it because the watch will pick it up automatically. What about shazam? Well, before you had to tap to get shazam to work, you no longer have to do that. That's pretty great, right? Settings on the watch, smart stack, music detection, and then the song title and the artist will just appear in that smart stack, which we've talked about before.

6:33So you don't have to tap on the widget to actually get it to listen and recognize what the song is. And each of these features, it's important to understand are opt in. Nothing is listening for anything until you say so. And so that's the first thing that you need to know before you get nervous and perhaps before other people around you get nervous, just by having a watch on your wrist does not mean that you're constantly listening. So let's break into I think the bigger features here, the features the people are a little worried about. And the first one is live rewind. It's a feature that gives you the last 15 seconds as text and it does so on purpose and it does so with a loud chime so that everyone knows that you're doing it. How does it work? Well, on your Apple Watch, you double press the digital crown and then what happens is the watch will give you a text snippet of the last 15 seconds of nearby speech.

7:38And it'll appear on the display. So it's just an instant replay of a conversation. And when you do this, you can actually ask Siri about it or save it to the Siri app. So what's going on behind the scenes? Well, the X-Clave is constantly keeping a rolling buffer, 15 second buffer that's constantly overriding itself. Remember, it's in the X-Clave. It's not accessible by anyone and it's just an audio buffer. Nothing, nothing in that buffer even gets transcribed until the moment that you press it. So it isn't listening and typing the whole time in the background, your transcript. There's just 15 seconds stored in this box. And when you double press that button, it goes in and gets it. And again, to access it, you need to turn it on. You do so with the watch app on the iPhone, you go into Siri, you go to live rewind, and then of course you double click the digital crown.

8:45One thing that you'll probably want to figure out on day one, something that I'll be looking into, is that right now we've talked about this. What are these sort of muscle memory moments of the Apple watch? You double press on the crown and it goes into the app switcher that button map is going to change for us. So the setting does change what that press does. So we'll need to see how it changes. And again, this snippet, temporary by design, about 30 seconds after the display dims, it's gone for good. Unless you tap Ask Siri to ask a question about it or you save it to the Siri app, then it will sink end to end encrypted across your devices. Everything until that tap stays on the watch and on the phone, ask Siri and save are the only moments when the text will actually leave and head into the private cloud compute. But up to that point, it doesn't, doesn't do anything. It just is buffering in the background.

9:53So again, bear in mind that there is kind of a catch here and I think it's a good catch. It's a catch that people can rely on. Live rewind announces itself. It's an audible chime from the watch speaker and it does so in silent mode. It does so with headphones on and it also plays a full screen animation. And it has a microphone indicator as part of that animation. So everyone around you knows you just did it. There's really not a lot you can do to hide it. And I'm sure that there will be people who will figure out different ways to try to reduce the sound. But for the most part, someone double click in that digital crown and using this feature is announcing themselves as someone who wasn't paying attention. Moving along, we've got Siri recap and I think this is the feature that has a lot of people freaked out. But you got to understand that this is a summary. It is not a transcript. Okay. It's notes that you didn't take. Summaries that you didn't write and it's a seven day clock. Okay. You leave it on during a meeting, for example, or a parent teacher conference.

11:06That gives you the ability to stay present. And then afterward what happens is Siri generates a title, a summary and key points all within the new Siri app in a special tab called recaps. And it's not a transcript that is so important to understand. It is a recap. Apple is explicit about saying that this is a high level bit of understanding of what's going on. It's like notes that you jacked down afterward as you're recalling it yourself. And by design, it will never label who said what? It's not interested in giving you that information. It also can be set up so that it's scheduled by time or location. So that's where it might be handy before you go to a parent teacher conference to have that information locked in. It lives in the watch app on the iPhone. So you can run it manually or you can set a schedule by time and location. You can set it to say, only run at work, never run at night.

12:16And then of course an on-off toggle and control center on the watch for the moment that you walk into something that you want to be private. As far as the kind of privacy plumbing goes, here's how everything works in plain terms and in my understanding of the sort of flow, right? We've talked about the X-Clave. The fact that there's this little area on the chip that is separate from everything else that other things cannot reach into and grab the audio. So the watch is X-Clave, the special listening place. First, and crypts the audio. And then it takes that and it hands it to the iPhone's X-Clave. The iPhone will further transcribe and in doing so, it will take, it is, it's, it's, Apple literally says it's designed to reduce the transcript by half. So it is making a summary of the actual conversation that's taking place and boiling it down to less than half its length.

13:26From there, that condensed text, so something that's already a summary of what actually took place, goes to private cloud compute. And then private cloud compute, aka, off in the servers, does the full summary. If you're curious kind of what goes along with this information, well, now playing, so whatever you're listening to at the time, calendar info, and then sort of, nebulos labels like home or work or your city without giving precise location. So this information is a little bit of context just enough for private cloud compute to do its work. But it's sort of like playing a game of telephone with a group who's playing a game of telephone with a group who's playing a game of telephone. And Jennifer Paterson, too, on this week in tech, argued that perhaps by the time this gets summarized by summarized by summarized, it may not be as useful as you would imagine.

14:35But we'll see. Now recaps these summaries, these, these sort of distillations of things that have happened will auto-delete after seven days. So they last a week, unless you choose to save them. At which point you can edit, delete or export any of them to notes or to the journal app, for example, all from the Siri app. I'm excited about being able to put them into my journal app, deleting one in one place removes it everywhere. So it's important to note, once again, that there is a limit. Apple says that summaries can be incomplete. They can be wrong and that it's filter for things like financial details, for example, for ID numbers may not catch everything. So of course, confirm anything important with an actual human. Energy of AHA, lack of concentration, nevlamental, you might be dehydrated.

15:38Gatorade Zero powders have been designed scientifically to help you improve the balance of hydration in your body. In fact, with the number one mix of electrolytes, five colors or artificial flavors, shop now online or at Retailers Nationwide. Gatorade, hydrates better than water, is it in you? Signos de hidratación leve y moderada, consulta tu médico si los síndomas persisten. Visible puts unlimited 5G data and hotspot in the palm of your hand, powered by Verizon's 5G network, with no contract holding you back. And for a limited time, you can get visible for just $19 a month for 12 months when you use promo code Save6. All the features of Big Wireless Service, for half the cost. Tap the banner to switch today. Terms apply, Standard Rate applies at month 13, see visible.com for plan features and network management details. Star Wars, the Mandalorian and Grogu is now streaming on Disney Plus. What are you waiting for? Hop on. The Must See Galactic Adventure comes home on Disney Plus.

16:41This is the way. Our only chance is to work together. I'm the lifeiest kid. Always wear your seatbelt. Star Wars, the Mandalorian and Grogu, now streaming on Disney Plus. Grade PG 13. Now before you decide to turn it on, let's talk about requirements, let's talk about timing, and then of course an understanding that Apple leaves the etiquette to all of us. In order to have it, you need an Apple Watch Series 12 or Ultra 4, and this is to be paired with an iPhone. It needs to be an iPhone 16 or later, not an iPhone 16E, that won't work. So it also needs to have Apple Intelligence and the new Siri AI Beta turned on. So again, Series 12 or Ultra 4, it needs that hardware with the Secure X-Clave,

17:42and then it pairs with an iPhone 16 or later because those are capable of doing the processing on device before they get sent off to Private Cloud Compute. When does this work? Well, once you've got that software set up, it is available when sound recognition and Shazam, those features, the features that will give you the simple understanding of sounds that are happening around you and also Shazam automatically working, that is as soon as the Watches arrive. So September 18th Friday, as we record this on Tuesday and later publish on Thursday, and then live rewind and Siri recap are according to Apple coming in beta later this year. It will be in English first, and it will not be in the EU to start, not surprised by that. Both of these server side features do count against Apple Intelligence's daily usage limits.

18:44So just be aware that a full day of recaps could hit a ceiling. There are age gates involved, so it is not available to people under 13 years of age and for teens who are on a child account, a parent or a guardian has to turn it on. But the etiquette question, that's one that Apple can't answer for us and Apple has not tried to answer for us. No recording exists, no one is supposed to be named, but the people who are sitting across the table from you may still reasonably object, right? And Apple's own document does go as far as to say. Be mindful of the people who may feel that way, which is kind of just another way of saying, hey, maybe ask. So while this features in beta later this year, maybe live rewind is for the moments that you missed something and Siri recap is only where you'd feel fine saying, hey, by the way, my watch is taking notes out loud.

19:54If you don't feel comfortable saying that, maybe it's not a place where you should have it turned on. So I like to end this with a little bit of homework. If a series 12 or an ultra four is lending on your wrist this week, as it is on mine, give those features that do work a go. Sound recognition with detect using Apple Watch and music detection in the smart stack with Shazam. See how those Shazam results feel when they show up on their own instead of requiring you to tap. And if you are on an older watch, well, this week's assignment is a conversation. Perhaps you ask a person how they'd feel about a friend using live rewind or Siri recap around them. And I would love to know what you and your friends think. So be sure to let me know in the club to it discord when the beta ships will do the hands on, of course. And there's a whole other layer to this story because the new Siri app itself is something we'll have to cover. And Siri AI which launched this week too.

20:57All of it deserves its own episode. So I'll be back next week with another episode of hands on Apple. But until then it is time to say goodbye to all of you. Thank you so much for being a subscriber. Thank you so much to those of you who are club twin members. And I'll talk to you again soon. Bye bye. At twit.tv slash I am our website or wherever you get your podcasts. I'll be looking for you.

22:02NFL football season is here which means when you switch to Verizon it feels like because you get NFL Sunday ticket from YouTube on us. When you buy an eligible 5G phone on select unlimited plans which means you can watch every out of market game every Sunday afternoon. And it's all on us. Now that's the Verizon way to kick off the NFL football season right. Switch and get NFL Sunday ticket from YouTube on us only with Verizon. Hello. Look what TJ Max dragged out. The Devil Wears product too is now streaming on Disney Plus and Hulu. We are digital. We are downloadable. We are streamable. The fashion event of the year is certified fresh. Pull yourself together we have work to do. Critics say it's smart and witty and the perfect sequel. That's all. Get runway ready for the Devil Wears product too on Disney Plus and Hulu. RIDD PG 13.