Transcription
I've noticed a lot of people are getting pissed off about token cost right now. And while it's great that these new models that keep getting released, the cost of these models isn't falling. It's actually increasing significantly. And to be honest, if you're using AI in the cloud, that's something you're just going to have to get used to.
But there is another way, and that's one of the key messages that came out of the Apple Worldwide Developer Conference this week. And this is something I've been saying since the day I started this channel. And to prove that, I'm going to give everybody with a Mac the ability to write a local AI application and run models locally on their Apple device, even their iPhone, and it's going to cost absolutely nothing.
So in this video, I'm going to talk about Apple Core AI and then give you a skeleton app to help start making use of the amazing Apple AI technology. Now at WWDC, the headlines all went to Siri. But the announcement that actually mattered to me the most was tucked away in the developer sessions. And that's called Core AI, Apple's new framework for running AI models locally on device. It's the same inference engine that powers Apple Intelligence, but it's now been opened up for anyone to build on. It replaces Core ML, the the framework that Apple's had for nearly a decade. And it also runs across the whole chip, CPU, GPU, and the neural engine.
The name change tells you the whole story. Core ML was about machine learning, and it was built back in 2017 for the old world when we had small models such as image recognition, classification, with generative AI bolted on later as an afterthought. But Core AI has been rebuilt from the ground up for this world, large language models and generative AI, and it has the tooling to convert, optimize, and run them at at scale for modern applications. So it's the same idea, but with a much more powerful engine. So in other words, Apple just developers the infrastructure to keep intelligence local, on device, not in the cloud. So, so local AI is not a fringe idea anymore. It's actually the platform. And that means no token costs, no snooping, and no data retention.
But, Core AI is in beta. It's a developer thing, and it's a framework. So, it's not something that you're going to pick up and run with tonight. So, that's what's coming. But, in the meantime, I've built a local AI version that runs on Apple's existing on-device technology, which is proven fast, and it's working right now. And of course, as Core AI matures, we're going to put those into this project, as well.
So, now I'd like you to introduce you to Project Onyx. Why did we call it Onyx? Well, for two reasons. The first is because it's short for on your MLX, so it runs locally on your device. But, the second is that the Onyx is a precious stone, but it's opaque, which means you can't see into it. And And it's exactly the kind of privacy and security story about running AI on device.
Project Onyx is a small open-source application that runs a large language model entirely on your own device, your iPhone, your iPad, or your Mac. There's no cloud, no AI subscription, and there's no token cost from any AI company. The model loads on your hardware, and it runs right there, and all your conversations stay on the device. When you use it you with airplane mode, it's still going to work. And I was even doing this on a plane this week.
The great news is that I'm releasing this whole thing as open-source for free, because I believe a lot more people should be able to build private on-device AI without having to start from a blank page. So, let me tell you what Onyx is. It's a skeleton. It's a starting point. It's a scaffolding about loading a model and running inference on device with a clean, basic chat interface. It's not a finished app, it's but it's to get you off the ground. And that's because the hard fiddly plumbing with getting models to run has already been done for you. What you do from there is up to you. You can fork it, rip it apart, rebuild the interface, um bolt it into your own product if you like. It's yours to modify however you want. That's the whole point of putting it out in the open. But to be honest, I won't be able to answer your questions or provide support. It's It's up to you to test it thoroughly before you put any when you're a real-world scenario. Treat it as a learning tool or a launch pad, not as a finished product.
So, how do you actually get it running? Well, you need a Mac and you need Xcode, which is Apple's free development tool. You download Project Onyx from the repository in the link below and you open it in Xcode. You compile it, plug in your iPhone or iPad, and then deploy it to your own device.
Now, just to be clear, this is not using Core AI. It's still using Apple's existing MLX framework, which was built specifically for Apple silicon to take advantage of unified memory and the GPU that's sitting on the chip. Once I've had a chance to properly get my hands on Core AI, I'm going to build some of that into this project and make use of all the functionality. But right now, MLX is doing all the heavy lifting and it's more than good enough to prove the point and get you started.
So, this week Apple put its weight behind on-device AI and Project Onyx lets you actually run it locally today. It's open source, entirely private, and there's no token costs, and it's free. So, grab it, build it, and make something with it. That's because the era of private on-device AI is here. And all the links are in the description below. So, please go ahead, build something amazing, and do it on your own device, and never pay for another token again. I'm Dr. Earl Brandt, the founder of Kira, and if you enjoy these videos, please like [music] and subscribe. And I'd love to know what you're building with Project Onyx. Thanks again for watching and I'll see you in the next one.