Transcription
How can you build an iOS app 10x more effectively with AI tools like Cursor and Wind Surf?
AI coding platforms like Cursor and Wind Surf have become extremely popular in the past few months. We’ve seen people building all sorts of different applications, some of them very sophisticated. However, it has never been easy to build an iOS app with Cursor. One reason is that if you're building an iOS app, you have to use Xcode as a development environment. Even though you can still open the project file with Cursor, there’s still a lot of context that Cursor won’t have. For example, if there are error messages, it’s very difficult to bring all those error messages as context back to Cursor so that it can debug and fix issues.
As we know, clear and precise context is one of the keys to making an AI coding assistant effective. On the other hand, there are also frameworks like React Native and Expo, which help developers build a codebase once and transform it into Android, iOS, and web applications. The problem is that the best large language models out there, like GPT-3.5, don’t really have much knowledge about Expo and React Native. So often, the code it writes is just not good enough.
Both of these challenges make iOS development quite difficult. But the good thing is that over the past few days, I’ve learned some new workflows and tips that are going to make your iOS development 10x easier. You can almost develop an iOS app just like how you build a web application. I will show you step by step what my workflow looks like and also take you through an example of how we can bring a calorie analysis app idea to life.
What would you say if I told you there’s an app on the market that does just that?
Let’s start with a hot dog.
Holy, yes, it works!
Mother, do pizza!
Let’s do pizza!
Pizza!
Not hot dog!
Wait, what the heck? That’s it? It only does hot dogs? No, and a not hot dog?
Yeah, this app idea, where you can just get AI to tell you what food it is via the picture, just seems so magical a few years ago. But now, it’s actually very easy to build. And no, we are not going to build a hot dog/not hot dog app. I’m going to take you through an example of how you can rebuild the Cal AI app, which is an app that allows users to take pictures of food and automatically analyze the calorie intake using a multimodal large language model.
If you don’t know, Cal AI is an app built by a 17-year-old high school student. They’ve already made more than 1 million in monthly revenue. So without further ado, let’s get started!
But before I dive deep into this, one question I get asked a lot is: what does the roadmap look like for learning programming? That’s why I want to introduce you to this free ebook created by Google’s principal analytics lead and data scientist, Samantha Scalette. She wrote down all the secret tips and methodologies that she used to learn coding and designed a personalized learning roadmap that caters to your needs.
It covers all the fundamentals and basics of coding, like how to choose the right coding language to start with for your purpose, best practice prompts for different coding scenarios like debugging and optimizing code, as well as a detailed roadmap of how to master Python in just four months. It includes a purpose-built custom GPT that contains a lot of learning resources and detailed video tutorials showcasing step by step of her workflow.
So I definitely recommend you go take a look if you’re just getting started with your coding journey. I put a link in the description below for you to download it for free. And thanks, Samantha, for putting together this awesome material for free!
Now let me take you through my new iOS Cursor development workflow.
The first thing we’re going to do is transform your Cursor and Wind Surf into a proper iOS IDE. One of the key components you’re going to use is something called Sweet Pad. Sweet Pad is a VS Code extension that brings all sorts of Xcode functionality into VS Code directly. Because Cursor and Wind Surf are built based on VS Code, you can just install this extension directly.
To do that, the first step is to open Cursor and click on the extension button here and search for Sweet Pad. This should pop up, and you can click on the install button. After you install it, you should see Sweet Pad showing up here, and you can just pin it as well so it’ll be easier for you to access later. This is where we’re going to run the project and open the iPhone simulator later.
With Sweet Pad installed, your iOS development experience with Cursor is already 10x better. But there are also a few things you can do to bring it even further. One is that you can install the Xcode build server. The Xcode build server enables advanced code navigation for features like autocomplete, where it can show all the variables and things like error highlights.
You can just go to the terminal and do `brew install Xcode build server`. We also need to install iOS Deploy. This will allow you to run terminal commands to install mobile apps on your iOS device or simulator.
On the other hand, there’s also another package you can install called XCPretty. This will make Xcode output much easier to read in VS Code, and this is particularly useful for things like error logs. Again, you can install this by doing `brew install XCPretty`.
The last one is that you can also install `brew install swift-format`. This is a tool for automatically formatting your Swift code to follow consistent styles.
Last but not least, if you haven’t installed Swift language support in your Cursor, you can go to the extension and install it.
With all those setups, your Cursor is now becoming an iOS IDE. I will show you what I mean.
To get started, let’s open Xcode and create a new project. I choose "App." You can give it a name, and I’ll name it "SE Food." For the testing system, I’m just going to choose "None" for now and click "Next." Create.
After this is created, we can go to Cursor and open the folder. Now, the first step we’re going to do is do Command + Shift + P, and there’s one option called "Sweet Pad: Generate Build Server Config." This will create a build server.json file that allows the Xcode build server to work with your project directory.
You can choose a project here, and after finishing, you should see a build server.json file here. This is pretty much it! You can now start running the application directly from Cursor.
If you go to the Sweet Pad tab and click on this play button for the first time, it will ask you to choose the simulator. You can choose any simulator, but on the other hand, you can also connect your iPhone or iPad, then open the destination and refresh the destination iOS device here. You should see the device show up.
If you don’t see that, make sure you go to Xcode and set up the device properly. In this case, I’m going to open iPhone 16. Below that, you can see that the app has been running. You can see the simulator is open and running our test app automatically.
Now we can open GitHub. I’m using the GitHub desktop app, and I will click "Add Existing App" and choose the folder where we created this Xcode project.
Now let’s try to build and launch an iOS app. The one I want to showcase how to build is this app called Cal, which allows users to take a photo of food and automatically analyze and log calorie consumption. This app is generating more than 1 million in monthly revenue, and this is just a classic example of how you can bring any niche idea you have to the market.
Firstly, this is one setup we need to do. If we go back to Xcode, click on the project icon you have here, and go to "Info," we need to do something here to actually generate a file called info.plist. This is where we will write the device permission access to things like camera and location. Xcode, by default, didn’t really expose that file, so we just need to open anything like document types, add a new one, and leave it empty.
Now you will see that a new file called info.plist has been created. Now Cursor will be able to update this file to manage permissions.
On the other hand, before we just ask Cursor to build the application directly, there are some best practice setups that I often say people don’t really do. One is Cursor rules. All major AI IDEs like Cursor, Wind Surf, or VS Code all have this component: project-specific rules.
All you do is basically create a new file called .cursor-rules if you’re using Cursor. The information you put here basically becomes a system prompt for the Cursor agent. For example, you can put a random instruction here that always starts with "Yo," and you can open the side panel and just say "Hi," and you see that it starts with this random phrase that we put at the top.
But to make your Cursor rules actually effective, you want to make sure the rules you’re putting here are concise and necessary. I know there are platforms like Cursor.directory that aggregate different Cursor rules that people have been putting out there. But one mistake I see people making is that they just copy whatever Cursor rules other people have and use them for their own purposes.
This often doesn’t work very well for a couple of reasons. One is that you want to keep the Cursor rules as concise and necessary as possible because the more context you feed the AI agents, the more likely it will forget details and not follow the rules.
So by just copying other people's Cursor rules, you’re very likely going to include some stuff that is not necessary for you. The way I would structure my Cursor rules normally looks something like this:
Firstly, I give a one-line description about the context of the project. Here, I won’t dive deep into specifications about all the requirements of the project because I will create a new file called instruction.md later for detailed requirements. Here, I will just give a quick context.
There are two things I almost always put in: one is that I will ask the agent to always add debug logs and commands in the code for easier debugging and readability. The second one is that every time they choose to apply rules, explicitly state the rules in the output because quite often, it’s hard to tell whether the agent is actually adopting the rules that you list out here or not.
By doing that, it will make it clear. Next, I will have a section called existing project structure. This is where I would give the project structure so that the agent will know which files already exist and which files don’t.
To get the information again, just open the terminal and do `tree`. Okay, so `tree` is a package that can generate the file structure of your current file, and `-L 3` means it will go down three layers deep. So I can just copy the project structure and paste it in.
Next, I will also put the tech stack. This tech stack is particularly useful if you are using third-party libraries for certain functionalities, meaning it prevents Cursor from implementing a functionality with a different package, which might mess things up.
In the end, depending on the type of project you’re developing, I will also put in some Swift-specific rules. Again, in here, I normally just put in situations where I observe the agent often makes mistakes. If you’ve never seen the agent make certain mistakes, there’s no point in including the rules here.
With these Cursor rules, it should give us a pretty good starting point. The next one is that I will also create a folder called "instructions," and inside "instructions," I will create instruction.md. Here is where I create specific product requirement documents, normally including the scope and purpose of the project, and then list out the key features.
For each feature, I will number it so I can dive deep into the specifications. If you don’t know how to build it, I will also show you how I get this product requirement document ready. Typically, I will use a powerful model like GPT-4 to generate this product requirement document.
What I do is take a screenshot and then give instructions: "I’m trying to build an iOS app called Cal that auto-calculates and logs calories. It should have key functionalities like below: take a picture of food, analyze food, and view history. Help me think through how I should build this and how I should structure the PRD document for engineers."
The PRD should have sections for project overview, feature requirements for each feature, data model, and API contract. It should be explicit about dependency variable names and APIs to call, so there are no ambiguities.
It will generate the product requirement document. You should always review the document and see whether any part does really align with what you were thinking.
So it is breaking down into four different features: capture food image, analyze food, edit and confirm results, and view history. Looks good! The only thing is that I plan to use OpenAI’s multimodal model ability. Together, I normally need to provide some kind of code example of how to use this API point.
The easiest way to do that is to go to the OpenAI playground, choose the right model, and then the response format I will use is the JSON schema. This is where we’re going to use a structured output feature from OpenAI. Just click on this generate option.
Your AI calories calculator will be given an image of food and output what ingredients it contains and the calories. For example, the output should include the description of the image, ingredients, which I will use to indicate that it is an array, and total calories.
I’ll click enter. Then this will generate the JSON schema, and I can update it a little bit to my food calorie analysis. Here, I’ll change the title to be "image description," which describes what food you see in the image, and reasoning, then ingredients, which is an array. For each array, it has title, calories per gram, total grams, and total calories.
Since I changed the title here, I will need to also change it here and click save. I’m going to just take a picture of some food, like this pancake. So I will go back here and do a test.
Okay, so you can see that it outputs the information properly. It identifies it as French toast and all the ingredients it contains. So this looks pretty good.
What I need to do is choose a code option and change it to CURL. I’ll copy this and go back to our PRD ChatGPT session. I’m going to paste it in here for the large model API service we’ll be using. OpenAI’s latest model with structured output function. Below is the latest code example, and this is where I paste in this code example here and also copy this example output.
I’ll tell it this is an example response. Now help me update the document to reflect this, making sure you use the latest in taxing the code examples above.
It didn’t actually return me the full PRD, so I’m going to ask it to return me the full PRD. Cool! Now it returns me the full PRD. What I will do is copy this and paste it in the instruction.md.
If you want to get the full prompt as well as the Cursor rules I’m using here as a reference, you can click on the link in the description below to join the AI Builder Cloud, where I have the full PR pasted. You can just copy and paste, as well as tons of other practical AI coding examples that you can take as reference.
But more importantly, there’s a community of top AI builders who might have already experienced the problems that you have today. So you can just come here and post any questions and challenges you have, and both I and other community members will just come and provide advice.
So that’s pretty much it! We can get started. I will open the Cursor composer in the right side panel here and arrive at the problem: how we build an iOS app based on instructions.
Let’s first say, "Do step one: capture food image."
And I’ll accept all. For all code generated by Cursor, I think you should try to look into each file to understand what is implemented.
On one hand, it will help you to be better developers, but also it will allow you to quickly check and see if there are any issues.
You can see here it creates two files: one is CameraView, which is the code where it will define the UI. You can almost think of this as HTML. You can see that it shows a camera preview view, adds a space, and also adds a button.
It also creates a CameraViewModel. This is a common structure for iOS apps called Model-View-ViewModel. You basically use the code under views to render the UI, and the view models are where you will process the data that will be rendered on the UI. This will make the code easier to maintain.
Without further ado, let’s get started! I’ll go to Sweet Pad and click on this build/run button.
You can see we got some errors, and this is the best part of using Sweet Pad. We can just copy those errors from the build file to the end and then paste those errors in and accept this.
I will run again, and this time you can see that the build has been run successfully. If I open the simulator, it has this app with this button. Click on "Scan Food," and it will open the camera view.
But one thing is that in the simulator, you can’t really test out the camera feature. That’s why I will also try to install and test this app on my phone. I’ll just open QuickTime for you to be able to see on screen.
After you connect your phone to Xcode, you should see an option here, and we can try to run this. Here it says the OS version didn’t really match, and this is where we can go to "General" and change the iOS version to 18.
Now the error is gone, and I will go back to Cursor. In the destination, you should see an iOS device here. If you click refresh, this option should show up.
What I will do now is do Command + Shift + P and select "Set Active Destination." In here, you should see your device showing up. You can click on that and click "Build and Run," and that should reinstall the app on your mobile phone and run.
Now I can try to use it on my phone.
Okay, so here’s one problem. Even though I can see the camera view pop up, there’s no actual live view of the camera. What I do is I will copy those commands in and then open the Cursor.
I’ll prompt: "The app is installed and running. However, after I click 'Scan Food,' it brought up a sheet of camera with a camera button, but it didn’t show me the actual live camera view. Below is a log. Help me add more debug information and figure out what the issue is."
Here’s one thing I often do: every time there are errors, I always ask it to add in more debug information. Here we found the issue: the camera view isn’t showing because we need to properly set up the preview layer and its frame.
Okay, I’ll click accept and let’s try again.
Okay, so the issue didn’t seem to be fixed, but now we actually have more debug information. I will copy those things in.
The issue is not fixed; still the same behavior. Here are the logs, and now it says that the issue it found is that the preview layer frame is zero, which means the view isn’t getting properly bounced.
I will try it again. Here you can see this is the benefit of always asking to add more debug information because it will really help you to identify the root cause.
Okay, great! Now you can see this camera view is working properly. Now we can move to the next step.
But before we do that, I will just open my GitHub and give a commit: "Implement Camera View."
Then I’ll go back to instructions. Currently, after I click on the camera button, it didn’t really take me to the confirmation screen. So I will ask it to implement the rest.
The camera view is showing up properly, but it didn’t take me to confirmation after I take a photo. Help me finish the rest of the capture food image requirements.
This time it means creating a new view called ConfirmationView. Now, if I click on the camera button, it will take me to this confirmation screen where I can use a photo or retake.
Alright, so now let’s do step two.
Now we add a new commit: "Add Confirmation Screen."
Then go back to Cursor.
Great! Now let’s do step two.
Cool! This time it created a folder called "Service," where it created OpenAIService here. It also created an EndlessView to present the result from the OpenAI API.
It also updated CameraViewModel to include this Endless functionality. One thing I would do is check OpenAIService to see if it is implementing things properly and as expected.
You can see there are some parts that are not working well. It’s using the old model, which I changed to the new model.
The last thing is that I will also need to replace the API key, so I’ll just search and replace my API key here. I’ll click accept and let’s run the app again.
Okay, and this time we got some more errors. But the good thing is that we have those error messages here directly. So I’m going to paste this in and take the error cell to build.
Let me run again.
Okay, so this time we got some different build files. I’ll just again paste this in.
More build files.
Okay, great! So this time it succeeded. Let me try. I’m going to pick up an orange, and it’s showing the small text: "Analyzing food."
I can also see the log here, but it showed a failed message. So I’m going to copy those errors in.
The app built successfully but got this error after receiving the response from OpenAI.
So here I think the error is because of the model name, and I know for sure it is not the case. So I’m going to update the message here a little bit.
The app built successfully but got this error after receiving the response from OpenAI. The model name is not the issue; it should be GPT-4. Help me add more debug logs to understand the root cause.
Okay, so this time it gave me the proper error message. As you can see, it’s missing the required parameters: response format JSON schema.
If I check the original code example, you can see the key here should be JSON schema. So probably this is the issue. I will change this to be JSON schema here, and let’s run again.
Okay, and this time it gave me more errors about the JSON schema.
Let me just double-check.
So here we put JSON schema into the full analysis schema, and it has type properties. If we look at the proper schema, it should have name strict.
So I think the problem is that the schema here isn’t correct. What I would do is go to the instructions and copy this example code response and put it in here.
I will also mention OpenAIService.
I still got errors, so I think the JSON schema food and schema is not found properly. It has to have the same structure as the code example I pasted.
Please update the code.
Okay, so let me accept this and we can run again.
Okay, great! Now we get a response from OpenAI, and it’s showing an icon stream directly on the screen.
But you can see on my phone it is popping up a proper analysis like this.
Great! So I’m going to make a new commit: "OpenAI Integration."
Then let’s do the next step.
Now let’s do step three.
I finished it and created a new file called EditableFoodAnalysis, which is the data model.
As we mentioned, we are using this Model-View-ViewModel structure, and the model is where you’re going to store those data structures.
In this case, that means for each food analysis, we will have those data, which is title, image description, ingredients, total calories, and health.
This will help the system validate the data type, making sure all data is in the same structure.
I also created a view for editing, and now let’s try to run this.
Okay, and we got these errors. I’m going to copy those in.
Got failed boot errors, so I can run this.
Take a photo, and here one thing we can optimize further is the image compression.
Currently, it’s sending the original image to OpenAI, which is probably too big. If we compress it, it can make the image processing much faster from OpenAI.
So this is one optimization we will do after we finish the whole application.
Okay, cool! So I can see this adding button. If I click on that, I can start editing the view, but there are some changes I need to make.
I’ll give some feedback: a few things to fix.
One, in the editing view, the user should only be able to edit the grams of each ingredient and calories per gram. They should not be able to change total calories; that should be calculated based on the values above.
If they change grams or calories per gram, the total calorie calculation should be updated automatically.
The second issue is that the editing experience is not great. The UI looks crowded, and after I type in a character, the keyboard automatically closes.
Help me fix the above.
Okay, and I’ll click accept and try again.
So let me try again. I’ll take a photo of an orange and wait for the response.
I can click on edit.
Now, if I change the grams to be, let’s say, 80 instead, you can see the calorie calculation will be automatically updated.
I can add a new one. Let’s say I also had milk.
So this part is still not fixed, but in here, if I change the calories per milk to be 0.2 and amount is 100 grams, the total calories will be calculated.
But there are still some issues: the user should not be able to update the total calories and health score.
When updating ingredient names, after typing each character, the keyboard closes automatically, which makes the editing quite difficult.
Let me try again.
One thing up to this point is that I actually didn’t care too much about the actual UI because I’m going to come back and fix all the UI later.
Okay, let’s try again.
I’ll click on edit.
Now I can change the calories here, and you can see the total calories will be automatically updated.
If I add a new one, okay, so this typing experience is still not fixed, which we asked to fix.
But if I add milk as a new ingredient, the total calories will be automatically updated, so that’s pretty good.
I’m going to just ask to fix one last thing.
The calorie calculator is fixed now; however, the editing of ingredient names is not fixed. After I type in any character, it will automatically close the keyboard.
Help me fix this.
This time it added a temporary title to handle this and also unchanged the modifier to maintain the focus.
And it got an error again.
So we’re going to repeat the same process.
Try again.
Take a photo.
I click add.
If I try to change this, it’s still the same behavior.
This is very strange, so I’m going to give debug information.
Still the same behavior of auto-closing the keyboard while editing the ingredient title.
Help me add debug logs to understand the root cause.
Okay, so now we have some new logs. I can copy those logs in here and then say those behaviors are still the same.
Above is a log.
This time it found that the issue is because the view has been recreated every time the title changes, which is causing this focus loss.
Now if I try it again, I think this issue should be resolved.
Okay, great! Now you can see that this editing behavior is also implemented properly.
The last thing is I want to ask it to add the view history.
I will say, "Great! Now it worked. Let’s add features based on instructions."
Let’s try to run again.
Okay, again we’re going to copy those error messages in.
Okay, cool! So you can see now the home screen became the calendar view, and for each day, I can click on "Scan Food," take a picture, analyze it, and save this.
The problem now is that after I click save, the log is not showing in the history view.
So I’m going to give a prompt: "After I click save for food analysis, the log is not showing in the history list view. What could be the issue? Help me add debug logs to find the root cause."
Got build error again.
Okay, let’s try it.
This time I’m going to prompt: "Still in behavior, no meal found after clicking done in history view. Is it because data was not saved?"
Let’s try again.
Okay, great! So this time you can see that it is working with proper saving behavior.
So this is my workflow of how to use Cursor to do iOS development.
If you enjoyed this video, I will also make some content about how you can build Android apps with Cursor and Wind Surf, as well as how to make beautiful iOS app UIs using Cursor.
So please comment below if you want to see more of this type of content.
Meanwhile, if you want to get the full prompt as well as the Cursor rules that I am using for this type of iOS application, you can join the community that I’m building called AI Builder Club.
I am continuously posting in-depth content about how you can use AI tools like Cursor and Wind Surf to build production-ready applications, as well as large language model agents.
But more importantly, we have this community of top AI builders who might have already experienced the problems and challenges you are facing today.
So you can just come to the community, ask any questions and challenges you have, as well as learn from others’ tips and tricks.
I have put the link in the description below so you can click and join.
I continue sharing interests in learnings and projects I’m doing in AI.
If you enjoyed this content, please like and subscribe.
Thank you, and I’ll see you next time!