Apple Watch VP — It Has to Work in 2 Seconds!

Apple's Kevin Lynch, VP of Technology, and Deidre Caldbeck, Product Marketing, chat with us about their histories with Apple Watch and Health, new features like ID cards and keys, how they decide on new features, how they made Assistive Touch for Watch, bringing Mobility to iPhone, the visual representation and security of Health information, security vs sharing, and the future of Apple Health!

• I'm Kevin Lynch. I work on software across Apple watch and health.

• And I am Deidre Caldbeck, and I am on the Product Marketing team for Apple watch and health.

• And it's cool that Deidre had been off the paddleboard and came into the, yeah--

• No, it's underneath my feet. I'm just very stable.

• I know we didn't get to see you dance again this year. Kevin, we didn't get to see you and Jules do a duet. And I did file a radar, expect to behavior dancing.

• Off camera, that was all off camera this year.

• I'm actually really curious. If we go back to the beginning earliest days of the watch, when you were first read in or disclosed and you heard that Apple was making a watch, what went through your head? What was your immediate reaction?

• Well, for me, that was actually day one when I started at Apple, the morning I started, the story was we wanna build a watch, and we've got to get going. In fact, we wanna get going right now. We're feeling like we got to go fast. And actually I was advised to not even go to orientation. Like you don't even have time to go to orientation just get going, you know Apple well enough already and get started. And so it was a very rapid start. The thing that went through my mind, there's always, of course occasionally rumors about what Apple's may be working on. But when you look at Apple watch, I felt like that was inevitable. Like to have Apple make a watch, I felt like that was the first thing that went through my head was of course, oh we're gonna do that. 'Cause if you look, I love the history of computing. If you look at the history of computing, it's a series of evolution of smaller and smaller computing devices that are more and more powerful. And it goes from mainframes to mini computers, to desktop computers, laptops, portables, mobile phones. And so that evolution of power and administration leads you to, well, what's next, what's next after the phone? And that really leads you to something as small as a watch. And the watch is a natural location on your body. If you start thinking about something that's even smaller than a mobile phone, which is already pretty small, it's like, well, where would you put it? How would you carry it? How would you look at it? And so attaching it to your arm was a really really natural place to do that. So that was really interesting. And so of course, going through my head then was this is gonna be an incredibly challenging project. How do you even get a compelling interaction on a really tiny display? And battery life, of course is gonna be an issue, which is a major challenge. And so designing around that was from an engineering perspective, really interesting challenge, which I love. And then also this great opportunity, and really a sense of responsibility for working on a device that is actually on your body, like you wear with you, like that responsibility is really significant. How do we make something that people will feel really good about that supportive, and empowering for them, not bugging them. So that kind of philosophy around how we design all of our interactions is really sensitive to that.

• Unlike Kevin, I had no idea what was going on. Apple is very good at making sure that we keep our surprises, surprises for both employees and our users. And so I was working on iOS, and actually what I worked on initially was the health app because I worked on iOS, and Kevin's team was working on the health app, but I was on the iOS product marketing side. And so we were working out how to help tell the story of this whole new health app in iOS eight. And what I thought was amazing was we were gonna be able to talk about how you could bring all of your health data together from third-party apps and devices, and little did I know that one of those devices was soon to be Apple watch. So it was actually a really exciting thing for me, as you said, to be read in or sworn in on the secret of Apple watch much closer to when we actually announced it. So I really wasn't told about it until I think just a few weeks before the announcement. And I was very fortunate to be one of the people to show it off to our audience at the event, in the hands-on area. So I learned about it, honestly, just probably the same way you did Renee, and that was a really neat thing for me to work at Apple and hear about it like everyone else. So it was cool.

• My dad worked at IBM and he got an Apple two plus we wouldn't have to drive downtown to use the mainframes. And I had an iPhone. I wouldn't even have to go back to my Mac to do certain really important things. And then I got an Apple watch and it can't do as much as an iPhone, but it can do these brief frequent, but really critically important things without me even having to reach into my pocket or reach onto the desk for my iPhone. And it occurred to me that it took until like iOS five, and iCloud before the iPhone was really that independent from the Mac or the PC running iTunes, but systematically over the years with on-device logic and on-device app store, and bringing LTE over and just continuing now you have the family sharing where you can set up a watch for somebody, you are systematically just increasing the functionality. And I know that Apple is not shy about cannibalizing their own devices. You'd much rather do it than have somebody else do it, but it is that like a... Not a predatory process where you're looking at the iPhone and going, "What can we do next?" But is that like more of an organic thing where as time and technology allows, you'll just keep building up the functionality and the things that we're capable of doing from our wrist?

• The work we're doing there is really how can we empower people with Apple watch, and in different situations whether you're near your phone or away from your phone. And you've seen us doing that more and more over time, for sure. But really the main focus is how all of our devices work really well together. And it kind of, when you interact with each one of them what are you trying to accomplish? And each of the devices has its own kind of personality and way of going about things. So like you were saying with Apple watch, the little brief interactions is really core to how Apple watch works. And we started with that and that's how it's different than like you're using your Mac, you'll tend to use it for longer periods of time and phone. Like you were saying less than your Mac but more than your watch. So there are these kind of interaction times that are very different across the devices. And when we started working on Apple watch, we thought that time of interaction might be maybe something like 10 seconds. We were kind of estimating what those typical interactions might be like. And it was really interesting when we started actually living with the watch internally, we started learning that really, you wanna do those interactions even in shorter times than that. More like two seconds rather than 10 seconds. And that's a tall order to do some of these tasks in two seconds. We weren't sure how we could do that. So we paused what we were doing for a moment, we had everybody go and brainstorm for a bunch of different areas of the system. How could we possibly make those happen in two seconds? Like no rules. Like you can do whatever you want. You can change the interaction with the system, you can make shortcuts, you can take stuff out, whatever it is, make it two seconds. And we made a list of those things, and that was super productive. One of the examples of that is the messages app, when you receive a message, it used to be initially when you replied you had to kind of go into reply, and then go to a compose view and then press send. And that took more than two seconds. Now in messages, which has been this way for a long time now in the watch, when you receive a message, you can just scroll up with your digital crown, there's a bunch of suggested replies that we've generated for you, and there's some you can edit if you want, and you just tap them. So scroll and tap, and soon you receive a message that scroll and tap you can totally do in less than two seconds. And so we went across and did a bunch of that same for like wallet, doing Apple pay, like just a whole bunch of areas of the system, we went and tuned in. And that type of investment in time and engineering makes a huge difference. And it's sometimes like what are you removing more than what are you adding?

• That's one of the things I love most about Apple watch, because like the, the downside of focus is sometimes tunnel vision, but it really seems like over the last few years even though health and fitness have become so popular, have gotten so much attention, the original sort of promises of the Apple watch also included payments and authentication, and remote control and communication all these other things. And you've been systematically improving those as well whether it's walkie-talkie or now the Intercom system, the new home app, way more robust. And one of the things that interest me the most is keys and IDs, because this just takes... It both simplifies the process of authentication, identity and control, but also greatly empowers it. Can you talk just a little bit, about how those new technologies work?

• This is an area we've been working on for many years and it's really exciting. The infrastructure is really well understood now, and we're at the beginning of it kind of rolling out this generation of access to your home, your office, your car, hotel rooms. So each of these categories has its own way of implementing this and then we've designed it on Apple watch so you can just use tap to access, just like you do with Apple pay using NFC. You can make them so they have express mode turned on, we call it, so you can just tap and access, whatever it is you're using. And with the proximity of NFC, there's an intent you're waving your device in front of the lock to cause it to unlock. So there's not this guesswork about whether it's unlocking or not. You still have a gesture that's involved to do the unlock. So I think we've got really a great approach to this now and super excited for it to start rolling out.

• Previously, we talked a lot about how messages and photos, and things like that were really what kept you connected. And now we have this evolution of the watch where it's your identity. It allows you to control the things around you and access the places you love. And I mean, if you think about Apple watches, your identity we started with Apple pay, when it's on your wrist, that's your identity. And we've recently used it as your identity for when you wanna unlock your iPhone and you're wearing a mask. And now with these watch USAID updates, with things like state ID and the home app redesign and these other wallet features, it's really starting to be more of this identity access, and control where it's the safety and security of a device that's always on you. So it's really perfect for all of these capabilities.

• I'm always continuously impressed and inspired by the accessibility technologies that Apple brings to all of their products. But I think that demo of assistive touch on the Apple watch, it just next level blew people like straight up science fiction away. And I was wondering like, how do you sort of realize all of this potential and sometimes maybe even take it further than anybody imagined.

• Yeah, these are deep collaborations across multifunctional teams. So it's the design, accessibility, engineering storytelling, marketing, it's all the different teams work on projects like this. And it's the combination that really makes them awesome. And on that one, there were lots of explorations like more gross movements of your arm, shaking your arm rotating your arm, lots of things like that. Those require energy as well. So we thought about like how much energy does it take to do these different gestures? And if you're using assistive touch, you wanna be really thoughtful about how much energy every time and every interaction. And so we settled on a couple that work really well from a low energy human energy perspective, but are really hard to sense. And the two are there's four signals, clench, double clench and then we did pinch and double pinch. And the sensing of that is happening on your watch. And it's using the gyro, the accelerometers looking at micro movements of your arm when you do those gestures, it actually uses the heart rate sensor as well. It gets the imaging feedback from from your arm included with the gyro and the accelerometer. So those are all blended together with machine learning and we've created models to capture these gestures. This wouldn't have been possible a few years ago, and it's just, as we're starting to understand how we can use these approaches in terms of building software. And also the capability of the performance of Apple watch has gotten so incredible, we can run these things live now. And we've been working on accessibility in Apple watch since day one, we've first shipped Apple watch. We were thinking about how to enable people with this and it's been such a pleasure working across all the teams on this and the team members, and the assistive technology area have just really been brilliant and have been working as part of the integrated Apple watch team since the beginning.

• I love it, I love your dedication, like the plural your dedication to accessibility and also health, just the way you've been ramping up health. And please correct me if I'm wrong, but Apple watch has been a big focus of health, but now you have a mobility and a sort of tying in the iPhone in a way that's really Apple watch style like using the sensor fusion and the machine learning on device and taking all of that, and giving us really useful preventative information. And in my head, I'm thinking, well it's the Apple pencil one day gonna be able to tell me I have RSI or carpal tunnel, like is this the beginning of some great Apple ecosystem health thing, but I'm curious like when you see this escalating across multiple Apple devices now, Deidre, well, first, can you talk about bringing this kind of technology to the iPhone, and maybe how people will use it or get the most out of using it?

• Yeah, I mean, I think you said it perfectly when you said that we do kind of think about our entire ecosystem of devices, and the capabilities within each of those devices. And really our goal is always to empower people to better manage their health. And whether there are sensors on the watch, or the phone, or Apple pencil that can offer that empowerment for our users, that's actually where we'll focus the investment and the energy. And, you know, with walking steadiness, it's something that, as you said, we really wanted to think about how can we prevent these falls. We introduced fall detection with Apple watch series four. And of course it's been incredibly rewarding to hear the stories from our users who have benefited from that feature. But if you think about it, gosh wouldn't it be better if that fall had never occurred in the first place. And we know that Apple watch has amazing health sensors, the motion sensors, the heart rate sensor, even we're using the microphone for health features with the noise app, but the iPhone also has as you know, motion sensors. And in this particular case, the motion sensors on the phone and where you actually carry your iPhone, typically in your back pocket or your front pocket or maybe a shoulder bag or a cross body bag, those sensors actually can pick up those subtle signals from your hip movement that can actually give us the fidelity that we we need to provide you with those mobility metrics, that can then give you the signal for when you might be at an increased risk of falling. So of course the watch similarly has powerful motion sensors and some of those mobility metrics are from the watch things like stair ascent speed and stair descent speed. We can only get that from the watch, because of the way the watch is positioned, and you often may not have your phone with you when you're going up and down stairs. And so we're trying to think about the best place to get the best data, to provide the best experience. And in this case, the iPhone was really best positioned for this and that certainly gives us the opportunity to offer it to all of the iPhone users, and not just Apple watch users, but we know that combination of phone and watches is really powerful for a lot of these health features.

• Always felt to me like the phone was like the star ship and the watch was like the shuttle craft.

• Ooh.

• And you can just get so much done by going back and forth--

• I love it, I love it.

• Between those things.

• Walking steadiness actually came initially from fall detection. So we're doing fall detection in Apple watch, we're all working on it and it's incredible how that's been helping people, and we still get letters about that, but we were thinking as we're working on it, well how can we prevent people from falling? Like, rather than just detecting that they're falling can we actually stop them from falling? And that just led through this really interesting journey of discovery and brainstorming about while we may not be able to stop you in the moment from falling--

• very carefully, walked around disclosing any potential anti-gravity fall prevention technology, that you might have in play after you teased us so mercilessly with the teleportation, a couple of years ago, and then Craig escalated to portal technology, through his aperture science app just this week. So I appreciate that your whatever vibranium casing you're using is still in the future, but you're trying to give us tools we can use today.

• In the meantime exactly doing what we can, exactly.

• Yes.

• We actually did this really deep work that has taken years of sensing these different attributes that Deidre was mentioning. And then doing studies where we partly use the Apple heart movement study for this was about a hundred thousand people in that. And so we were able to use the insights that we've gathered from people's movements in that study, to actually see what would be predictive of falls. And so that's what this walking study this metric came from, was the modeling of those metrics. And then seeing which ones indicated most clearly that you had potential to fall. And that was supplemented by studies specifically on this topic too, where we had cohorts of people, where we were doing traditional walking studies with them, like observations and questionnaire filling out and things like that, kind of traditional characterization of potential falls. And then we tracked those users with, you know, they were part of the study, and as people did fall over the next kind of year or two, we were able to go back, and look at those authentic falls, and see what the indicators were earlier and what led to those. And you can't fake that you can't just do prep fake falls, you have to have actual falls. So it takes a long time to really get that pre-work done, see it actually happening and model it correctly.

• It feels like we're getting into like phase two of all of this technology, where in the beginning you were just building up more and more sensors and more and more data. And maybe labs is like the current ultimate expression of that. But now that you have all of this rich data you're providing things like trends, but not only are you looking at the trends, you have this Edward Tufty inability to take all of this medical information, and make it just so human digestible, like, just in terms of the interface and the graphing and all of that, how do you sort of deal with these ancient sometimes very inhuman data sets, and make them so people who are legitimately stressed like they have health anxiety can get so much benefit from just seeing it easily displayed maybe for the first time?

• Yeah, so trends is like, how do we draw insights across all kinds of different data types in a simple way. And again, we use a lot of modeling from the studies that we've been doing to look at patterns and the significance of those patterns for people. And so we basically have built systems that do statistical relevance of these insights. So not just showing you numbers that maybe bigger or smaller, but which ones might matter. So there's a lot of thoughtfulness by data type about the variances in which types of variances might matter for which data types. And then we look at comparing the past month, through the past six months, and then we look for the changes that might be notable in that time. And there was also a lot of work on like, how many of these trends might we show people? We don't wanna show people too many trends 'cause it's kind of overwhelming, but also you don't wanna like not show any trends. And so we did a lot of experimentation on tuning that in to get it to a point where it was a useful relevant trends without being too many.

• One of the things I'm curious about I get asked this question all the time and I'm sure you do as well now, especially that you have iPhone now involved in health. And I was joking about the Apple pencil before, but for example, the iPad, when you look across the Apple ecosystem and you start thinking about a future where, you know, maybe it doesn't make as much sense to have these particular features on an iPad because you don't carry it the same way. It's not with you the same amount of time, it doesn't have the same sensors or connectivity all the time, but maybe it's really great for reporting or for a big screen to see all these infographics that you're doing such a good job on, is health something that you really do look at in terms of the entire Apple ecosystem?

• Yeah, absolutely. I mean, if you think about the health features we have today, there are obviously several on Apple watch and iPhone, there's also some health features with Air Pods and some of our audio products, right? I mean, so there's absolutely an opportunity for us to leverage the ubiquity of our devices, to discover new ways that we can empower people to better manage their health. And we'll continue to investigate those areas, we'll continue to invest in those areas. I think with the new features that we introduced this year, we are excited to hear from more and more users who are going to take advantage of these great features, for example, with health sharing, we know that we hear from so many of our users that they're feeling this this burden really, or this overwhelming sense to care for both their children and their aging parents. And to be able to do that in a way that, just have it in one place in their health app where they can see their own health data, but then also the health data of their loved ones. And in a way that's private and secure, where not only do their family members feel safe when they're sharing their data, but also the people that they're sharing with know that it's a secure connection between the two of them.

• I love that so much. And I love your approach to privacy and security so much, because I think it's easy to be extremist like Info Sec extremist about these things. Yes, you wanna keep your health data private, but if you're an absolutist about it, you can't share it with your doctor or with your kids or your parents data, or maybe you're not physically capable of using that, but your caregiver is the one you want to, and I love that you're being so, like you're doing informed consent which to me is everything. But when people are willing to do that, you're empowering them to use that data, beyond just locking it up and making it actually functional for them.

• Exactly, and I think to your point about informed consent that's why the way we built the experience, it was with all of our features privacy is at the core, you are in complete control over the data you share and who you share it with. You can of course stop sharing at any time. You can choose very granularly which pieces of data you wanna share. If you've had a chance to go through the experience, you can even see a preview of what the person you're sharing with will be able to see, so that you can be sure that's what you wanna share. And it's not just control, right? It's also transparency. So you'll never share data that you can't see yourself, and all of those principles are at the core of these features. And we'll continue to ensure that that's the case in the future as well.

• One of the things we've done which is wonderful is we've added kind of this discipline to the mix of making products, which is the clinical side. So clinicians working with great health backgrounds, working on and that's really helped us make some great strides in the work that we're doing here around health, and have it be something that's super relevant medically to people, especially when we do the doctor sharing stuff, like how do we make a dashboard that a physician can look at and understand very quickly, 'cause these interactions tend not to be very long. And so you've got to really understand someone's information really fast. And so we put a lot of energy into the design of those views, so that that would be super relevant to a doctor. And on the privacy side of that, that was really interesting too, because the end point of that is not an Apple device usually it's a computer running electronic health record system, and they're viewing the patient's records, they're inside the health institution. And so how do we get this information from their Apple watch and their iPhone to show up in that other device? And yet, have it be completely end-to-end encrypted and secure in a way that Apple can't see the data. And we're actually serving that to that EHR through a web view inside of those systems. And so we generate a view of the data that comes from the person's phone into this web view and download it onto the EHR so they can see it right in place. And it's in context, it knows which person's data to bring up 'cause of the current person they're viewing the rest of the information for in the EHR. But the encryption of that data is handled with an encryption key that's shared between the user's phone and that health's institution that Apple doesn't have. And so the package of data is encrypted on the user's device, sent via the cloud down to the EHR, and you get this encrypted package of data with them. If you looked at it in transit, you couldn't tell what it was, it's this encrypted blob, but it shows up and it's decrypted at the last second inside the browser of the doctor's view. So that's where it kind of unpacks itself and shows it in the view. And then when the doctor stops looking at it it's not there anymore and it's encrypted.

• I have this beautiful dream where one day when the world stops ending, I'm back in South Bay with all of you and maybe I'm at bitter and sweet having a red velvet latte. And my Apple watch goes off and alerts me to a potential health concern, and I'm just a Canadian on US soil. So I go to the local clinic and I can just tap my Apple watch and it immediately authorizes like all my health records, all my allergies, my medication, everything that has to do with me, maybe it Apple pays my copayment or my insurance deductible. It does everything that I would otherwise have to sit there with endless clipboards, endless phone calls, in a panic having to go through. And it handles that as easily as it may be handled at transit interaction today. And so my question is not so much about future technologies but about when you're approaching all of these things, how much of it to you is just like yearly updates versus the stars that you see hanging above the horizon?

• We're at the beginning of this work in health I would say, even as we've done a number of things already, there's so much potential here. And then some of the things you just outlined are lots of great challenges that still exist in the world around health and sharing information. And we're at the beginning. So the work we do is long-term. So we do work that, you know, may show up kind of on an annual basis or so, but that work is really backed by in some cases, years of work, that just happened to align at that time. So there's a pattern of that. Some of it's short-term, some of it's long-term, but our thinking is definitely long-term about what we can do here for Apple watch and for health. And for both of those things, it's early still, like we've done a bunch of great stuff, but the ideas are still ahead for us to do a lot more. It's just really exciting.

• Deidre I mean, Tim Cook said that health is one of the things that Apple's gonna be remembered for. So zero pressure, right?

• Zero pressure. But if you think about it, there's the, you know, we talk about this a lot, but it really warrants continued emphasis. We are just so moved by these letters we hear from our users. And I think that's why Tim makes these comments, because we are all just so fortunate to be able to read these letters and you hear, oh gosh, I had no idea I had this condition and I I bought an Apple watch, because I wanted to get my messages when I'm on a run or I bought an Apple watch, because I wanted to be motivated to work out more. And I had no idea I had this, this issue, or I was out kite surfing, and I had an issue and I remembered I had a cellular Apple watch that's also water resistant. And so I could just make a phone call. And so I think the way I think about it is we in some ways have already made quite an impact on our users lives in a way that I don't think we would have imagined, when we were first building Apple watch. But I also think to exactly Kevin's point, this is just the beginning, because those stories are really what inspire us to do more and to invest in these areas where we think we can make a real difference in people's lives. And so we're so excited to continue to do this.

• I've said a few times that I think, and I don't want Cayenne or Craig to gift it to me, but I've said several times that I think that Apple watch is just the most important device that Apple has ever made, because it like, yes, a PC, a phone can save lives, but the Apple watch has so many features that are designed just for the purpose of improving, preventing, helping, contacting for life. And I think just thank you, you personally, and the teams for all the work and all the effort and all the thoughtfulness and compassion, and empathy that you've put into each year, every year, making it just a better and better device, not just for me, but like many people, I've given it to my parents, I've given it to my siblings, and it's made a tremendous, tremendous difference. So sincerely thank you.

• Thank you, Renee, it's like an honor. The whole team feels this excitement and dedication to working on this project for very much the reasons you're talking about, and it feels like responsibility as well. It comes back to the Apple watch being worn. I mean, it's on your body, it's there all the time. And that's a unique thing. And a lot of goodness can come from that and a lot of support for people. So I've never worked on something as meaningful and compelling as this project, I never have worked on something where we constantly get users writing to us about how this has affected their lives, I've worked on some really great things over time, but this one really is special in that regard. And it just keeps coming, Deidre, we get these stories all the time, and we share them across the team. So people write, Tim, Tim forwards them around or to us, like we collect all these letters up, it's the emails mostly and the whole team reads them, and that's super inspiring. And some of them are great letters, some of them are like, Hey, you guys can do better on XYZ. We share those too, which is really helpful if you do write those in, know that actually we read those, and it really affects our thinking about what we were working on in the product. So yeah, it's an amazing thing to be working on this.