Transcription
Hi, I'm Meghan Athavale. I've been designing interactive walls for brands and events since 2010. In this video, I'll explain how interactive walls work and what you need to create your own. You can skip directly to the info you're looking for with the chapter links in the description.
The three main types of interactive wall are optical, touchscreen, and custom. You should choose the right one based on the type of interaction you want to achieve. In this video, I'll explain all three and what you need to build them. I'll also provide links in the description to all the products I mention.
The first type of interactive wall is the kind I specialize in, and it uses what's called optical imaging. This is the lowest cost and most flexible type of interactive wall. You can design one of these for almost any space, size, and budget. To make an optical imaging interactive wall, you need a display, a camera or sensor, and a computer. You can use any one of, or a combination of, projectors, LED panels, flexible LEDs, holographic displays, mist screens, and basic monitors. If you want to get really fancy, you can even use CRTs or DMX controlled lights and lasers. Any display that can be plugged into a computer can be used for this type of interactive wall. The computer will use input from the camera to make your wall interactive.
The camera you use will depend on the software and the type of interaction you're looking for. An optical imaging system will use either a 3D camera or an RGB camera or both. For LUMOplay, which is the software I make, you need a supported 3D camera. A good 3D camera costs around $200 to $500. LUMOplay responds to motion, so it's great for games and effects where the user is moving in front of a wall or throwing balls to play.
If you want to make an interactive wall that responds to motion, tracks poses, or detects objects, Orbbec's cameras are my top pick. They partnered with Microsoft to take up where the Kinect left off and they have a huge range of products. They're also compatible with all the industry standard interactive software out there, not just LUMOplay. If you plan to develop a custom application using Touch Designer or Notch or any one of the other interactive experience design programs out there, this is the 3D camera manufacturer that I recommend.
RGB cameras can be a great choice if you're using software that does object recognition, background extraction, image addition, or integration of a live video feed. Logitech is a good starting point for these types of projects, but if you really want a quality feed, a DSLR that can stream with very low latency is going to give the best results.
The simplest type of interactive wall is a touchscreen wall. You'll need a display with a touchscreen sensor, a computer, and software. This type of interactive wall is ideal for experiences that rely on the user actively touching the wall, and it's the only type of interactive wall that can reliably use touch specific gestures like pinch to zoom or click and drag.
The simplest type of touchscreen wall to install is a literal large scale touchscreen monitor like Viewsonic's Viewboard or some other all-in-one touchscreen. This type of touchscreen is often marketed as a smart board. All-in-one touchscreen monitors have a built-in computer that usually includes software. You can think of these as smart TVs that also happen to be a touchscreen. The Viewboard comes with Android built-in, so it's ideal for classroom applications that are Android compatible, like drawing programs and presentations, but it won't run programs that are outside Viewsonic's ecosystem, and it won't support true multi-touch. All-in-one touchscreen monitors like this are not designed for high traffic areas. They can't handle impacts, rough use, or installation in outside environments.
Ēlo and Planar make touchscreens that can be used in high traffic and less controlled environments. These touchscreens must be connected to a computer and they typically support both Android and Windows as well as multi-touch. Touchscreen monitors are extremely costly, so I don't recommend them unless you absolutely must have touch interactions with high accuracy. If that's what you need, it's important to understand the difference between types of touchscreens so you can choose the right technology for your project.
Resistive and capacitive touchscreens are what you find in mobile devices. They're fine for single users in a controlled environment, but if you've ever tried to use your phone when it's cold or wet, you know that this isn't the best solution for an outdoor installation. It also has a limit on the number of touches it can track, and the screens themselves are kind of fragile. Projected capacitive touchscreens support multi-touch, and they're much better for long-term interactive wall installations. If you want me to nerd out about how touchscreen technologies actually work, let me know in the comments. For the sake of keeping this video on track, all you need to know is that projective capacitive touchscreens can work through a protective barrier. So, it's easier to make this type of touchscreen really durable. Sometimes manufacturers get lazy and they just write P-CAP or PCAP. So, that's a good acronym to know if you're looking for a projective capacitive touchscreen.
Infrared or IR frames are also pretty common, but I don't recommend these for high accuracy touchscreen stuff. They can be super durable, and they also have the advantage of working without physical touch if you set them up right, but they're tricky to install, sensitive to certain kinds of light, and their accuracy and response time tends to get worse the larger they are.
Lidar is going to give you the closest thing to a touchscreen that you can get without a screen or a frame. OptiTUIO sensors from Interactive Asia are good for this, but they're also really expensive at around $2,000 and up, and their sensors only come with a handful of games. This is an okay choice if you're developing your own content, but it's not the best option if you want to use ready-made software with a large existing catalog, and it's not going to mimic touch gestures as accurately as a screen.
If you're installing a wall that responds to sound, weight, external data, weather, or anything that a camera can't see, then you're designing a custom sensor interactive wall, and I would like to be your friend. There are all kinds of custom sensors on the market, and any one of them can be used to make a wall interactive. The biggest challenge with these types of installation is the user or audience understanding the relationship between the wall and whatever it's interacting with. The best advice I can give is to keep this in mind. If the sensor is something the user is interacting with directly, place it somewhere where the wall will be the focal point during this interaction. If the wall is collecting data and displaying it interactively, find a way to highlight the incoming data so the audience knows they're witnessing a live or emergent display. I love hearing about these kinds of projects and I'm available to consult if you want advice. You can reach me at info@LUMOplay.com.
Lidar sensors will pretty much always be mounted directly to the wall above your interaction area and they will track the area right in front of the wall. Sometimes people put the lidar sensor underneath the interactive area, but I find that in most public installations, this isn't very safe.
3D cameras and RGB cameras can track a lot more stuff than lidar, and you can position them in all sorts of ways depending on the type of interaction you want. But the two most common ways to install one of these cameras is either pointing at the wall or pointing at the user. Pointing your camera at the wall or display allows you to track the area in front of the display. And it's ideal for things like wall ball games, but you can also use this setup to do pose detection. Pointing the camera away from the wall and at the user is better for pose detection or skeletal tracking or in the case of an RGB camera for detecting objects and people and streaming live video of players.
Choosing a display for an interactive wall will depend on a bunch of factors including the size and shape of the display, the lighting conditions, and your budget. It's very common to use projectors, and these are by far the most flexible and lowest cost option for large displays, custom video mapping, or multi-surface interactive experiences. I made a whole video about how to choose a projector, and I'll link that here now and in the description of this video. The main thing to consider when installing a projector for an interactive wall is shadows. No matter what, people interacting with your projected wall experience are going to cast shadows if they get between the wall and the projector, which makes ultra short throw projectors or rear projection the best choice if you want to minimize this. If you're installing a wall ball experience or an interactive experience where the player is not standing right next to the wall, a short throw projector mounted behind and at least a few feet above the player's heads will reduce the shadows to a very acceptable level. I only recommend long or fixed lens projectors for interactive wall installations when the person interacting is nowhere near the wall and it's a huge wall. These types of projectors are the least expensive and easiest to seem. So, if you're making an interactive Pac-Man game on the side of a building, this is the type of projector you should use.
You can also use LED walls or flexible LED displays or flat screens in the same way as you would use a projector. Some of these screens will work really well in daylight, which is not something you can do with a projector, and you won't have to worry about shadows. But something that I think is often overlooked when it comes to LED displays is that most of them don't show up well on video. If your installation is something you want people to take pictures of and videos of to share with others, a projection will almost always cost less and look better.
Whether it's a Raspberry Pi or a Green Hippo media server, you are going to need a computer to run the software for your interactive wall. What you choose will be entirely dependent on the type of experience you're designing. A Pi or Arduino can run simple sensor experiences like applying a filter to a video when someone touches a theremin. Android is a great option for Android apps, obviously, and to scale up single player experiences that already work on mobile devices. Linux or Windows are best for mid-range commercial and educational installations in places like retail stores and museums.
Software companies like LUMOplay and Touch Designer will provide you with the exact recommended specs for their software. LUMOplay even has a partner who sells a kit. If you're using Quartz Composer or Touch Designer, a MacBook M1 or higher will run your installation well. If you're coding something yourself using Unity or Unreal or straight up OpenGL or Python, I recommend picking a computer that can play Ark. If it can render the dinosaurs from the video game Ark, it'll run your installation like butter, unless you're really bad at coding. If your installation involves multiple displays or very, very high-end, high resolution content, I would consider a Green Hippo or Catalyst media server.
I hope this video helped answer some questions about how interactive walls work and what you need if you want to make one. I help people do this all the time. So, if you have a project in mind and you'd like to discuss it, you can reach me at info@LUMOplay.com or leave a comment below this video. If you're excited about making interactive experiences, you may want to check out these videos next. And please consider subscribing if you're an artistic nerd and you want to join the LUMOplay community. We'd love to have you. I'll see you next week with another tutorial.