Transcription
Hello everyone. It's Justin Ryan here with the Spatial Insider and in today's video, we're going to be diving into everything you need to know for Vision OS 27. There are a lot of great updates given today and maybe some hidden features that I want to make sure that you know about and so we've got an incredible guest that's going to break it all down for us. So, Steve, would you mind introducing yourself to our audience?
>> Sure. So, my name is Steve Sinclair and I'm senior director here at Apple in the worldwide product marketing organization. My team is responsible for product management and product marketing for Vision OS.
So, one of the big themes of this WWDC was all about Siri AI. What are your thoughts on that with spatial computing and Vision OS?
That really is the big news for today across all of our platforms that that support Siri is that the new Siri AI is more capable and conversational. Um, it's built to be that assistant that's ready for you for the questions that you're going to ask. Um, and it does that in a number of ways because it basically has more personalized context about you. it has uh the ability to reach out into the internet to pull information with broad world knowledge um and also to integrate with the apps that are on your device so that it can understand more about you and be able to help you in the moment when you want it. And so we're really excited about all the things that Siri AI is going to be able to do um built on top of that foundation of a new and improved Apple intelligence.
>> It was pretty awesome to see the visual yes of Siri in Vision OS. You know, we're all used to Siri on our iPhones, on our iPads, on our Mac, but to see it in Vision OS and the new version of it, it's pretty cool to me. And and one of the reasons that is is you've got these widgets in Vision OS that you can anchor onto a desk or on a surface or pin it onto your wall >> and the video looked like you' be able to pin the surb. Is that right?
>> That's right. So, what we were able to show and and what uh customers will be able to to to see and developers today will be able to see in the beta is that ability to pick up Siri and place it where they want. Maybe it's on the table in front of them and to pin it right there so that it's there for them when they need it. So, they can just look down at Siri, ask a question. You don't even have to say, "Hey, Siri." Um, and it'll respond to you. And you'll love the animations that it has because it can basically indicate to you that, oh, I heard you and I'm thinking and I'm going to go out and I'm going to get that information and then you get the substantiation of the the answer that it gives you in a small window that you can then, you know, grow if you need to and be able to uh get the information you want just by glancing at it.
>> That feels like science fiction, right? We're using our eyes to track. We're just looking at something. We're just speaking to it and it answers us >> well. And that's what we want uh Apple intelligence and Siri AI to be is that thing that's there when you need it um and helpful in the moment and it doesn't need to do more than that to be a big success.
>> So I'm curious for you when you're in the headset and you've got this new Siri do you just leave it pinned down there and then that way if you've got questions you just look down and ask it.
>> I have different uh I have different work setups depending on if I'm at home or if I'm uh at at at work in my office or in different places and so I do place it in different places. Sometimes they don't place it at all.
>> Is it persistent >> like widgets? It'll remember its place.
>> Yeah, it can remember its place.
>> That's awesome.
>> So, we also saw some big updates to visual intelligence.
>> Sounds like it works for both digital as well as physical objects being able to understand and see. Can you speak to that a little bit?
>> Yeah. So, again, that a lot of that is based on the fact that Siri AI is has on-screen awareness. And so, of course, for us, the screen is the world as well. And so it's it could be digital content that you have in a window. Say you have Safari open or you have your messages open or you have something else that you're you're working on. And so you can ask questions about that. Um but you can also look at physical objects in the world around you and query that. And again it takes advantage of the fact that it's aware of the things that you're looking at. Um and also able to uh take in information from your personal context. So, the example that we showed in in the keynote was um you know, someone that's going on a trip and they basically want to know is this bag, this backpack I'm looking at on this website um going to work well as a carry-on. And it knows, oh, you have a trip to Iceland later on this year. This is the flight you're on. This is the airline you're going on. And it can do the the homework behind the scenes to figure out, will this bag in its dimensions work as a carry-on? And then you immediately go into an example of, "Oh, but what about these boots I was planning to take and they're sitting on the table? Will those fit in that?" And it remembers, oh, you were talking about this backpack. I can recognize what those boots are. And it can do that um uh that correlation between those two to give you a helpful answer. And then it adds in some things. Oh, and you might think about, you know, hacking a little bit differently if you want to fit more things in there. We're excited about what people are going to do and learn about it as uh as we go through the the beta. Um, and we will continue to to to to see what possibilities are available with the new foundation models for Apple Intelligence and the new capabilities that Siri brings.
I love iOS. I've loved it since the beginning. And I tell people this all the time. >> Where where we are at right now with Vision OS is just as exciting to me as where we're at with iOS back in like 07, 08, 09 where every single year I'm just so excited because every year you bring a whole another level of advancement to the platform. Obviously being able to have Siri AI built into there, being able to speak to this persistent object that's just floating in your space, that to me just feels like magic. So, thank you for building that. Um, something else that I'd really like to speak about would be we saw a clip today, which is basically taking a photo from your phone, like a panorama, >> and turning that into an environment.
>> That's right. >> We constantly are getting feedback from from customers and our users about the things they'd like to see in Vision OS. you know, we may not speak too publicly about what we're working on, but we absolutely are listening to what to what folks ask for. And one of the things they loved was this idea of um being able to create spatial scenes from their photos in their library. And so spatial scenes being the ability to create with through generative AI depth and dimension to to the photos that that mean a lot to you and gives you that ability to lean in and look around. And so um one of the things that we got as feedback from from customers after Vision OS 26 was gosh I would really love to apply this to panoramas and could I use that as my own personal environment. So that's what we did this time around is that we took that same uh concept and that same technology and we applied it to your personal panoramas um that mean something to you and allow you to create again that depth and that dimension um and that ability of parallax to feel like you're in that scene and then to go into the environments view on Vision OS and be able to select those panoramas to become your backdrop for when you're working from when you're doing you So, uh, anything from an entertainment perspective to have that thing that means so much to you up and around you um is great. And of course, you can switch it out anytime you feel like it.
>> I'm really excited to try that, to take some of my old panoramas, add that depth to it, check that out. And I imagine that's going to work just like any other environment.
>> That's right. And so, um, those are amazing. Of course, we've also, um, made it possible for web developers to create environments, um, for their websites as well. Um, so we're excited to see what web developers are able to do to bring a personalized um bespoke experience to to to the visitors to their sites as well.
>> Now on these environments, my understanding is it's not the floor or the ceiling. It's going to be kind of that that larger panoramic in front of you is the environment, but this time now it's got depth.
>> This time it's got depth. And so you're right. It's it's very very similar to what you would expect from a panorama, but with the depth. Um, you still will see floor and ceiling. Um, no audio along with that. U, but basically give you that feeling of being surrounded. And of course, you can place apps in front of you. Um, and continue to work just just as you would in a normal environment.
>> And is that panorama? Is it 180 degrees? Can you go 360 with this?
>> You can go almost 360. Um, and you know, it's it's similar to um uh normal spatial scenes where it depends on the quality of the photo um how many pixels you have and how much AI has to work with to to make a convincing uh depth map of of of what that scene looks like. It will give you a very good experience of of of feeling like you're back in that spot. Um and so we're excited about that. We also added a new environment.
>> Yeah, I want to talk about that. from Apple. And so, um, this one is called Thor's Mo, >> and it's basically a, uh, a place in in Iceland that's just beautiful. It's got a beautiful river flowing through it during the day. And in the evening, of course, as you'd expect in Iceland, a beautiful aurora borealis that is visible um with the lights dancing above you as you as you sit and you work or you you enjoy your entertainment.
>> I'm super excited to try that one out. I'm I'm curious, how do you all decide which locations to build these environments for? I think our design team has a lot of fun trying to figure that out and I wouldn't want to put words in their mouth, but I'm sure there is a lot of competition for places that that they would love to see um put it put into Vision Pro and Vision OS in the future.
>> I'm that crazy person that actually took like a globe and put a pinpoint for every one of the environments that exists today to see if there's any sort of like system to it. But just seems like you're choosing beautiful locations and a good range of locations. Well, I can see myself now going and getting a lot of panoramas and going as wide with those as I possibly can so I can have that go around me. That's right. And experience that depth.
>> Can we speak to the quality of life improvements that are coming?
>> Um, certainly you saw in keynote today across platforms that there was a lot of emphasis on improvements at a platform level just to make the things you do every day feel snappier, feel better, work better. Um, and so a lot of emphasis was was was done there across every single platform and Vision OS was no exception. Even though we're a new platform, um, there's always things that we can be better at and and and make things better. And so, a number of areas that that we've gotten feedback and that we've built into to this release um will show up and we think customers will like it. A good example would be um notifications and being able to just glance up to see a notification that comes in and without having to tap it, just look at it. it will expand so you can get a quick view of of what uh might be in that notification. Um then you can tap and hold and say it's a messages notification. It will pop up a small messages app window so that you can deal with that that message right then and there in the moment um and then dismiss it and keep going about your way. No need to open up the messages app and to to manage that. So that's just one example. We've also um uh redesigned and reimagined what control center looks like. And so we've done some reorganization um so to speak to to make it easier to get to the controls that um that people use every day. And one of those controls is screen recording. And so I know you as a creator, many of our developers um and certainly others um that report on and are interested in capturing things in Vision OS um use the the screen recording capability today all the time, but it's fiated, meaning that it's blurry around the edges where you're not looking when you capture it. um and that does not translate as well onto a 2D screen. And so we wanted to give people um a way to capture high quality recordings that remove that fiation but still give them a 4K 30 frames per second UHD type of experience without having to connect to other devices. They don't have to connect to their Mac to do those captures. Um and so you can capture a scene um for up to three minutes, which is plenty of time for what most people need to do. Um, and so we expect to see a, you know, even better, um, captures coming from developers who are using it for marketing purposes and, um, certainly from creators who are reporting on all the great things we put in Vision OS this time.
>> Well, that's a big one for me. Like you said, whenever I'm doing a screen recording, I'm always trying to look at exactly what I'm wanting to show that it is in focus. >> Now that everything's going to be in focus for those three minutes, makes me really excited. Now, the first thing I want you to uh to to capture, not the first thing, but one of the first things I want you to capture is also um some of the improvements we've made to Safari. And so, what you'll see in Safari and a couple of our other apps is now we've taken that curvature that you expect and love in Mac virtual display of that curved window when you're in wide or ultra wide view. Um and now you're able to see that um evidence in Safari, in the Apple TV app, and in Free Form. So, in Safari, of course, as you stretch wider and wider, you want the ergonomics of it to wrap around you. And so, you can do that um in Safari. And so far, as you're in Safari, you can also open up the multi-tab view. And you can see all of your tabs lined up in kind of a carousel. If you remember the old cover flow, this is what it reminds me of, the old cover flow um from original iPhone. And the ability to to like flip through your tabs and just just rotate through them um is is is very satisfying. So anyway, you I think I think customers will will love the ability to see all their tabs up and curved around them. Um in Apple TV, um the curve display shows up and is best used when you're in multiv- view. Um so I don't know um who was watching F1 on on Apple Vision Pro, but I certainly was and I enjoy seeing it um five up and being able to see all the different feeds from the different drivers that I care about. Um, this last weekend I happened to be in Vision OS 27 and I had it curved around me and it was awesome.
>> That sounds amazing. Yeah. For those that maybe haven't experienced that, you know, Apple has this partnership with F1 where you can have five camera views going on at the same time. They've got the track, they've got the drivers, they've got all of these different shots and it's so easy to go from one to the next and get the audio source from that other video. If you haven't tried that out, you should definitely check that out. And now I really need to check it out. Vision OS 27. That's really exciting. Let's talk about developers, right? We're here at this developer conference.
>> It sounds like there was a lot of developer updates.
>> So, one of the things I'm really excited about is um a complete rewrite of Reality Composer Pro. We're now on version three. Um and the team has outdone themselves in building out an application that we think is going to be um really valuable to not only app developers, but also creators as well. We've actually pulled it out of Xcode and it's a standalone app within uh the developer uh portal. Um and so that makes it more accessible for creators who may not be hardcore app developers, but it builds in all sorts of capabilities for um character animation, um scene creation. It's got a great live preview capability, so you can bring scenes and characters into Vision Pro and be able to see and and interact with those and iterate quickly with it um uh while you're wearing while you're wearing your device. Um and it's got a great new mode called script graph, which is a nodebased development view um that allows you to pull together different nodes really quickly, so you can experiment on how you want your app and and your your animations to work. Uh we've made um uh big improvements to MR lighting and and being able to mix, you know, real light with artificial, you know, virtual light that that you create within your app and be able to mix those together. There's a lot of work that's being done around the realism in cloth um that you would find in character animations and such. We've made uh improvements across all sorts of different areas as far as the uh the creative flow for um Apple Immersive Video. and you can you can find out more about some of the great tools there. Um, team has been working very hard on that to to to deliver something that I think developers and creators are going to love.
It seems like Apple is very dedicated on making these experiences feel as authentic and as real as possible. You know, when you look at how light >> shines or casts onto an object. Uh, personas being another example. What's the perspective or maybe the the the thought back at Apple and and why this is so important to make these feel so real? Where we've seen maybe other platforms be a little more cartoonish in nature, Apple's gone the other direction of let's make this as real as possible.
>> Well, from day one, the definition of spatial computing is being able to bring this digital content in your space and feel like it belongs there. And so, great pains have gone into just what you're talking about and making things look and feel as realistic as possible. And that's not just the visual, it goes to the audio and it goes um to all the aspects that that make you feel like you transported yourself to a new place or um that you're experiencing something you otherwise couldn't experience. And so, you know, our our engineering and our design teams, you know, they're the best at trying to figure out those edge cases where maybe it could break and figuring out how to push it a little bit more so that you really do feel like um that you're really there or that object is really real or that you really can pick it up and adjust it. You want to have that not feel like an illusion. You want it to feel real. Um, and they really and and our team take it really to heart that that's what people are after in in this kind of uh paradigm.
One uh specific uh new framework that we're bringing is is is another one that we're very excited about because it's going to have I think broad um broad applicability across a lot of different enterprise and small business and and creator communities. And that's something we call spatial preview. And so spatial preview is a framework um that again allows you to take your Mac OS app Mac OS 27 um and basically bring content from your Mac OS app similar to what we do with Mac virtual display into your space while you're wearing Apple Vision Pro. Again, this idea that I can preview content um and I can make changes on the Mac OS side and have that reflected directly in my space so that I can do that quick iteration back and forth to make sure I like something. Now, it works with SharePlay, so you and I could be viewing something together either next to each other or somebody remote would be would be with us so that we could review it. That's great for your colleagues and co-workers or your boss to review something. Um, it's going to be great for for clients. Um, and we think it's going to be super valuable for designers in general. So, think architects, think um interior designers, jewelry designers, any type of product designer is going to have um the ability uh to pull this information in and and these these files into their space. So, it works with USD, but it can also work with Apple immersive video stills. It can work with PDFs. It can work with standard 2D content as well. Um, so we're really excited about that. One of the things um that we showed in our state of the union was an integration that we did recently with an app called SketchUp by Trimble. And that's a very uh very very popular um design app used by interior designers and architects um so that you can bring content into your space. And so it's really easy between the spatial um the spatial preview framework and something new called USD kit >> to create um the ability for birectional editing. And so I can bring those content into my space. I can edit it in Vision Pro um and those changes can be sent back to the Mac OS app as well. Um, so just makes for great workflows um and really extends that that great capability the Mac virtual display started um you know a year or so ago.
>> That's absolutely massive.
>> Truly, I mean I met with so many designers that have told me they would love to be able to take their designs, show it in their Vision Pro and then make modifications on their Mac and then have those show up. So how responsive is it? Is it almost real time? Is it streaming? How does that work?
>> Yeah. So, it's streaming that content and it's it's relatively quick. It depends on, you know, how big the object is and and how complex the object is. It does do some work behind the scenes to to make that go a little bit faster and to optimize what that stream um time takes. Um, but the idea is it needs to be performant. You can't be sitting waiting for it to update, you know, and that awkward pause of, oh, I hope it's no, you can't you can't have that happen. So, um, the team has been working very hard to do that and we hope to get a lot of feedback through the developer beta, uh, to make sure that we're fine-tuning that capability and also work with a number of other app developers to to to be able to incorporate it so that when it launches later this year, it's going to be available, you know, across apps that people want to use. We worked with SketchUp. We also worked with Cinema 4D. Um, and so they've got a version of this working as well. Um, and we look forward to having it work in, you know, many other design type apps. We actually built it into the preview app in Mac OS as well. So preview app and and I'm not the expert here on all the things that they did, but preview app has been supercharged. Um, and actually adds in new capabilities and so you're able to to to bring preview content into uh Vision OS as well.
So what I just heard in all of that is a lot of use cases being unlocked.
>> Yes.
>> For why you would want to go out and get a Vision Pro. And a lot of us already own Mac devices. So, it seems like a beautiful pairing. The Apple Design Awards were just announced. We had the nominees. There were the finalists were announced.
>> I'm curious what apps you use on your Vision Pro.
>> Well, I'm excited that um that two apps, two Vision OS apps were were awarded awards in the Apple Design Award. So, pretty pretty proud of those those developers and very excited for them. One of them, of course, is the NBA app, which, you know, has just knocked it out of the park release after release. You know, you know, starting with first multi- view in in the very first release that they had, going all the way through working on um you know, being able to deliver live NBA Laker games um in Apple Immersive, which is which is amazing, all the way to, you know, the latest release that allows you to basically stand anywhere on the court and be part of the play and and to view the game in that way. And so they welldeserved on the innovation award. Another app that that won that was Vision OS is called Primary. Um and that won for social impact. Um and this is a group of you know journalists that are taking the medium of spatial video and being able to tell stories in new ways that just really bring home what it feels like to be in these far away places. And so I use both of these apps all the time. Um you know certainly I spend a lot of time in Safari. I think that that's a amazing app and I've been spending a lot of time of course uh uh watching a lot of YouTube videos on my favorite subjects. Um and it's a great way to to to be able to feel immersed from from that standpoint. And every week in my job, I'm I'm lucky. I get to try out all of these things and get paid to to to to see these things. But every week I say our customers are going to love this one. They're going to love this one. they just it the the number of apps that that that we're seeing come in where creative developers are figuring out new ways to take advantage of the space around you um kind of blow me away.
>> I I do see that as well. I feel like the first year or so we had a lot of apps getting introduced to the app store and it almost felt like the beginning of watch OS when you had a lot of developers building apps maybe for what worked on iPhone but maybe it didn't work as well on the watch and I saw some similar things being done on Vision Pro and there's just been a lot of thought I' I've seen from developers over the last 12 months thinking through what makes it a spatial experience what adds value.
>> Yeah, that's that's right. And you know, I think that there's a lot of creativity that's being unlocked um to think about things in new ways. Um we recently announced before Vision OS 27 but we recently announced a a collaboration that we've done with uh Nvidia and cloud XR um which has enabled apps like iRacing and XPlane which are amazing simulation apps um that take advantage of fiated streaming um and being able to do that in a performant way um from an from a remote endpoint um to give you the feeling of you're really sitting there in the cockpit of the car or in the plane Um, and those just unlock these these new experiences that that people are blown away by.
>> Um, I've got one more question for you.
>> Okay.
>> And you know, the narrative out there in some press is that Apple has put Vision Pro on ice or that they don't care about Vision OS or they've broken the team up. >> I have my own feelings about it, but I'm not Apple and so I'd love to hear your thoughts if you're okay sharing on that topic.
>> Well, sure. I mean, it's still early innings for spatial computing and so there's a lot of things um that we're doing and we're learning and we're continue to improve on um and continue to invest in. And so I would hope that updates like Vision OS 27 and all the things that we're adding and all the capabilities that we're unlocking whether it's for consumer use or it's for you know business use um serves as evidence that you know we are you know invested in this platform and we're invested in this idea of spatial computing and building these capabilities that just can't be done on on any other device.
I saw many times today throughout the keynote Vision Pro mentioned Vision OS mentioned anytime a platform just showing the imagery of the different platforms Vision Pro is prominently there. Uh we saw Blackmagic Ursa and immersive cameras here um which gets me all thinking about what that could possibly mean for the future. But I just see so much evidence with that with with what with what's going on in Apple immersive video with Blackmagic and the NBA live games all of these things. Like there's just so many pieces to me that are coming together. Like truly, this is the most exciting product I own and it's been that way since day one when I first got it. Um, anyone that watches this channel knows that I'm a huge fan and I use it every single day. But truly, you're doing some innovative things that up until now only existed in sci-fi.
One of the things I do want to say is I have so much appreciation for our developer community and our creator community that have invested so much time and effort into making amazing apps um in making amazing spatial content. Um and we love the interactions that we're having with those those two groups and and what we're learning from them and what they're learning from us and how we're pushing their workflows forward. Um, and it results in just just better content and better things to do in spatial than than we could ever imagine. So, um, just really appreciate the effort that goes into into all the things people create.
I'll echo that and just, you know, I don't have any content to share unless developers are building or these filmmakers are creating videos. So, thank you for doing those as well. Um, Steve, thank you for meeting with me today on such a busy day of Sure. Dubdub and and being here at Apple Park with me and helping the audience understand what's going on in Vision OS 27. So, thank you.
>> Yeah, thank you. I appreciate it very much. Well, everyone, thank you so much for watching this video. If you enjoyed today's video, make sure to hit that like button and subscribe to the channel. And until next time, stay spatial.