NeRF and Gaussian Splatting - easily explained

So hello everyone, my name is Julia Federrico. I am a PhD student in computer engineering at the University of Pisa affiliated with the CNR exploring topics related to artificial intelligence and computer graphics. What we will see today is what I presented in a university lecture for the virtual worlds course. Specifically, we'll be looking at two very popular technologies in the field of AI and computer graphics. Nerf angle sense plotting. I will try to give you a general idea of how they work without going too much into all the details and varants focusing instead of giving you a clearer understanding of the main concepts. All right, let's get started.

Regardless of whether we are talking about Nerf or Gaussian splatting, the goal is the same. Let's imagine we are taking photos of something in our world. we may not have the opportunity, the time or even the resources to capture every possible angle. The common goal of Nerf and Gaussian plotting is to create a sort of representation of what we see from the angles we do have and then potentially extract a 3D model. But primarily the aim is novel view synthesis. Being able to imagine or generate a view from an angle we haven't actually captured.

Okay, let's start with Nerf. Often when someone explains what Nerf does, they hit us with a this big formula. Well, I'm doing the same thing. So, I'm definitely not being original. But what I'm going to do is go through each term in this formula and you will see that in the end this big formula actually makes sense. Imagine you have placed the object you want to model inside a box. What we are going to do is to discretise that box. Each tiny cube inside it is called boxer. Now we want each vauil to have a color and an opacity that accurately represent the part of the object it contains. For this reason, every vauil will have an associated RGB color and an opacity value. Of course, at the beginning, these values are random, but during training, we want them to gradually adjust so that they reflect the actual color and opacity of the region each boxer represents.

But how can individual boxers know what color and opacity they should take on? We will use the captured views as our guide. Each view is associated with a camera in the scene. And from the camera's origin, we can cast rays through every pixel of the captured image. Each ray can be described by a simple equation that involves its origin, its direction, and a parameter t which tell us exactly which point along the ray we are considering. And here's the key point. that equation is actually one of the components we see in the big formula. So you can already see that the formula is computing something related to the ray itself. Meaning we will need to apply it to every single ray.

Now we want to find the points where the ray intersects the box because within that interval is where we are going to sample points along the ray. We will call the distance between one sample and the next as delta. Usually delta is not fixed. It's chosen using a technique called important sampling. The idea is that it's more important to sample where the density is higher because that's where we will find meaningful information for rendering. Simply put, if a region is empty, there is no point at all in sampling it. Of course, at the beginning, every region needs to be sampled. But as training progresses and the boxers start learning to represent the correct densities, sampling becomes more focused on the relevant areas. But for simplicity, let's assume for now the delta is fixed.

Since we are doing all this on a computer, we need to adapt the mathematical formulation to a discrete version. However, while this formulation is correct, it has little physical interpretability and low numerical accuracy unless delta is very close to zero. That's why we usually use a slightly different formulation. All right. Now, let's focus only on the boxers that contain at least one sample taken along the ray. Remember, at the beginning, each vauel has a random color and a random density. In this example, there are seven voxels involved. So the formula simplify to a sum of just seven contributions, one from each voxel. Each contribution takes into account the delta distance and the boxer's color. But here's an important detail in the formula. The color does not just depend on the point t along the ray. It also depends on the viewing direction. As you can see in the example on the right, the same vauel can appear with different colors depending on the direction from which is viewed. This means that the vauel must learn not only what its color should be, but also how the color changes with direction.

Finally, we have a term that accounts for the density of the current vauel and another that accounts for the densities of all the previous boxers along the ray. This last term is called transmittance. To get an idea of what transmittance means, let's first explain the physical concept behind it. Imagine you have a free regions or three boxers each with a density equal to zero. Now some information let's say light travels from the left and passes through these three regions because their density is zero. The information comes out unchanged since it doesn't lose any energy along the way. But if one of those regions has a density greater than zero, then part of that information or energy gets absorbed. More precisely, if a ray passes through an object like this boat here and travels a distance d through a region with a density sigma, the resulting transmitted information is described by the Lambert beer law. This law tell us that the outgoing information depends not only on the density of the material it travels through but also on the distance it travels and this is exactly the concept captured in our formula.

In our case we interpret it a bit differently. The information reaching us is the color coming from each boxer. But that color, the energy must pass through several other voxels before it reaches our eyes. So if the first vauel has a high density, it will absorb most of the light coming from the others and therefore will contribute the most to the final color we see. That's why each transmittance act as a kind of weight determining how much of each boxer's contribution actually reaches us. So if we collect all the colors coming from each vauel and we weight them based on how far they have to travel to reach the camera and therefore how much energy they lose along the way, we get the final color predicted by the model. To guide the model in producing the correct color and density for each voxel, we simply compare this predicted color with a real color of the corresponding pixel in the captured view. Once the model has been trained, each voxel will have its own density and its own color which also depend on the viewing direction. At that point, we can take a new viewpoint, one that was never captured before, and ask the model to imagine what it would look like. And that's really the main goal of the models like this, to render views that we never actually seen before.

Okay, at this point, you're probably thinking, you didn't understand anything. Weren't we already talking about Nerf? Well, actually, no. But don't worry. What you have just seen is crucial to really understand what Nerf is about. The resolution of the newly rendering views strongly depends on how we choose to discretize the space. A coarse discretization will result in a low resolution generated views. To achieve higher quality results, we will need to increase the voxal resolution. But doing so has a huge impact on both memory consumption and the speed of training and inference. And this is where nerf comes into play. Instead of assigning a color and a density to each vauel and optimizing them directly, why not let a neural network to predict the color and the density of an epothetical vauel? So, Nerf is basically a neural network that given the red direction and unbox we want to query predicts these two values, the color and the density. Keep in mind nothing else you have seen changes. The formula for computing the final color along array stays exactly the same. The only difference is that instead of asking each vau cell for its current color and density, we now ask Nerf itself to predict the color and the density of the vauel.

The underlying neural network is simply two branch MLP. Given the vauer's position, the first branch predicts its density and the second branch predicts its color. Notice that the density prediction does not use the redirection. That's because again only the color depends on the viewing direction of the vauer. So nerf can be seen as a function modeled by a neural network whose parameters we want now to optimize. Given a boxel position and a view direction, the function predicts its color and density. However, this naive formulation comes with some problems. Imagine that the plot on the left represents a function that given a special position, for example, along a single axis, returns the true color that point should have. When the scene we want to model is really complex and full of details, this function can be quite quite complicated, meaning it contains a lot of highfrequency components. The issue is that MLPS aren't particularly good at modeling such high frequency functions. As a result, they tend to produce overly smooth outputs, which leds to renderings that lack fine details because the function learned by Nerf are mostly made up of low frequency components. But with a simple trick, we can actually solve this problem. Instead of representing the boxer's position and direction in cartisian form, we can switch to what are called fer features vectors made up of a pair of s and cosine functions applied to the original cartisian coordinates. The number of these pairs determined by the degree L is directly related to how complex the scene is. For a simple scene with a few details, a low L works just fine. For more detailed scenes, we'll need to increase L. As you can see in this example, using FIRE features shown in the second row allows the model to capture more fine details compared to the Cartisian representation. And this improvement is not limited to the classic inverse rendering use case of Nerf. It also applies to other tasks where Nerf is used differently. For example, to predict the occupancy of a mesh, meaning that given a point in a space, Nerf doesn't predict color or density, but rather a binary value indicating whether the mesh exists at that point or not, or even for single image fitting where given just a pixel, the model simply predict its color.

Even through nerf is mainly used for novel view synthesis. It can theoretically also be used to extract a mesh once the scene has been properly fitted. Just to give you an idea, don't take this as the correct way to do it. Here's the basic idea. First, we decide on the resolution which is closely tied to how Nerf was trained. Then, for every possible vauel, we use nerf to predict its density. Notice that in this step we can ignore the viewing direction since we only use the first branch of the MLP that one that doesn't depend on the direction. Once we have obtained this density field, we can apply an algorithm called it merching cubes to extract the mesh. This mesh will be made up of many triangles and each triangle will have its own normal, a vector that is perpendicular to the triangle surface. To color each triangle, we can take advantage of this normal since it gives us a direction that Nerf can use to predict the color. But what I have just explained is really just an idea. In practice, the meshes extracted using Nerf, even with the current state-of-the-art methods, are not particularly good. They tend to be noisy and not very useful from a practical standpoint. And this is where nails comes in. Today, if you want to extract a 3D model from captured views, nails is the go-to approach.

While nerf is mostly used for novel view synthesis, nails is almost identical to nerf but it changes the way we interpret the object we are trying to model. In computer graphics, there is a well-known concept called it senior distance function. Where's DF? To give you an intuition, if we focus on the surface of an object and for every point in the space, we compute its distance from that surface, that's the SDF. By convention, if a point is inside the surface, the distance is negative. If we apply this function to every point in space, we get what's called a senior distance field. a field that describes for each point in the space how far it is from the surface. This simple change of paradig means that instead of predicting the density of a point in space like nerf does, nails predicts its SDF and that's essentially what nails is. As you can see from these examples, this approach allows you to extract meshes that are less noisy, smoother, and richer in detail.

The philosophy of Nerf, as we have discussed so far, is mostly a proof of concept. Training a Nerf model on a single scene, even a relatively simple one, can take minutes, hours, or even days. That's why a much faster and more efficient variant was developed known as instant neural graphics primitives or EMGP. To understand the main idea, let's look at it in 2D through you can easily extend it to 3D. The core concept behind EMGP is to rely on multi-resolution encoding. We start by creating a 2.2 grid over the image we want to fit. The first level of the grid, the zero level has nine vertices. And to each of vertex, we assign a vector. This vector is a learnable parameter that will be updated during training. And we store all of these vectors in a table. Next, we subdivide the space again. This time using a slightly higher resolution grid. And once more, we store a learnable vector for each vertex in another table. We keep doing this up to a certain level L. For instance, L equal to 6 is often enough in practice.

Now we want to predict the color and in the for DK is also the density for a given point. Say this green point. We don't feed the nerf its row special coordinates or view direction anymore. Instead, we do something that fundamentally changed the way Nerf works. That's because with this approach, we only need a much smaller MLP to reach the goal. For each grid in the multi-resolution hierarchy, we find the four vertices of the cell that contains our point and we retrieve their associated vectors. We then perform a weighted interpolation based on the distance, meaning that the closer a vertex is to the point, the more it contributes. After that, we concatenate the interpolated features from all grid levers into a final vector and that is what we feed into nerf to predict color and in 3D the density as usual. This approach dramatically speed ups training while preserving quality. Just a quick note for clarity, I've talked about tables, but in practice we use hash tables for a feature storage and indexing. As you can see from these examples, ENGP achieves the desired results across different tasks in just a few seconds.

All right, we have reached the last topic of this lecture. We are now going to explain what Gaussian splatting is and look at the mathematics behind this algorithm. One that achieves realtime performance even on very complex scenes. Let's imagine we want to model this scene. Gian splatting starts with a point cloud which it can really have any shape. For instance, a sphere. However, to ensure fast convergence, it's common to initialize the point cloud using the one derived directly from the model itself obtained with methods that given the captured views predict the death for each view. The core idea of Gson plotting is to associate a gausian with each point in the point cloud and then optimize those gausian. Each gausian is defined by its center, its scale along each axis which tell us how wide or fat it is along the direction, a rotation and both a color and an opacity.

Each three-dimensional gausian has its own mathematical expression that given its center and the coariance matrix allow us to evaluate it at any point we want in the space. This gion exists in what we call world space that is a coordinate system where everything is expressed relative to the world itself. However, when we want to understand how that gashan appears from a particular viewpoint, we need to transform its coordinates into the reference frame of the camera. Each camera has its own rotation matrix and translation vector which define how objects in work space should be rotated and translated to be represented correctly in the camera's coordinate system. Therefore, the mathematical expression of the gausian changes slightly. Both its center and its coariance matrix are updated according to the new coordinate system using these two simple equations.

To understand how a gashion appears from a particular camera viewpoint, we can imagine sampling many points within its 3D volume and projecting them onto the camera plane using a standard perspective projection. In this example, we are ignoring things like focal length or lens distortion. This is just to give you an intuition of the idea. However, such a perspective projection does not produce a perfect 2D gion. Instead, the projected shape can look distorted, sometimes like an egg or even a banana. But for our purposes, we need the projected or splatted gausian to remain a proper 2D gausian since we want to describe it using a closed form mathematical expression that's easy to work with. To achieve this, if we denote the projection function by M, we can take its first order tailor expansion around the 3D gausian center. This give us an important matrix, the Jacobian one, which allow us not only to approximate the perspective projection locally, but also to deride the 2D projected gausian in a closed form directly from the original 3D one. So if we want to know the color of the pixel on which a gausian has been splatted, we simply assign it a color of the gausian weighted by its opacity a father weighted by the value of the 2D gausian at the specific pixel location. Of course a single pixel is usually affected by multiple gausians projected from the 3D world. Therefore, the final pixel color is obtained as a weighted sum of all those gausian contributions. Once the predicted color is assigned, we can compare it to the ground through color from the captured image. By minimizing this error, each 3D gausian will adjust its parameter. It will translate, rotate, rescale, and update its color and opacity, gradually improving the reconstruction.

However, we haven't yet considered the special hoardering of the gausians. That is their depth relative to the viewpoint. Remember when we talked about boxers? Boxers that are closer to the camera given the same density contribute more to the final color than those farther away because the light or the information from distant boxers gets partially absorbed along the way. The exact same principle applies here. A gausian with a high opacity will absorb much of the light or information coming from the gausian behind it and therefore its contribution should dominate in the final pixel color. To properly handle this, we once again use the concept of transmittance which accounts for how much light passes through the gausian in front. This process knows front to back compositing describes how we will blend the gausian in death orderer from the nearest to the farthest one waiting each contribution according to its transmittance.

Gausian splatting is well known for its realtime performance which certainly would not be possible if we had to evaluate every gausian for every pixel of the image especially when the image resolution is high and the number of pixels is enormous. So how we can figure out which pixels need to be evaluated for a given gion? Imagine we have a set of points. One way to measure how they are distributed along each axis is to compute their coariance matrix and then calculate its AM values. Large again values indicate high spread of points along the corresponding axis. We can apply the same idea here. Each splatted gausian also has a coarance matrix and therefore its values can tell us the extent of its influence. We can define the footprint region. The area where we actually evaluated the gausian as a bounding box centered at the gausian origin will side lengths proportional to the heagen values of its coariance matrix.

If we were to directly parameterize the coariance matrix using three variables that can change during training, we would likely beate two key mathematical properties of coariance matrices. namely there must be symmetric and positive semidefinite. I really hate this last term semidefinitess. I I don't know even know how to pronounce it properly. So I will just refer to this as the two properties from now on. Okay, sorry. Hope you will forgive me for that. To avoid this we can use a trick from classical analysis. The he decomposition. Non-square matrices can't be decomposed in this way and even square but asymmetric matrices may have complex again values or nonorthogonal a vectors. However, square and symmetric matrices which is exactly our case always have real non- negative a values and orthogonal a vectors. From the spectral theorem, we know that any symmetric matrix can be written in this way where R is an orthogonal matrix and L can be further expressed as a matrix multiplication where S is a diagonal matrix whose handles are the square roots of the A values meaning they must be all positive. That's easy to enforce during training because each diagonal element can be treated as a learnable parameter free to change and we simple ensure positivity by passing it through an appropriate activation function. These three parameters which correspond to the egg values and hints to the spread of the gausian along each axis are directly related to the scale of the gausian.

As for the orthogonality of matrix R, we can rely on a well-known concept in computer graphics. According to Aero, the rotation of a point around an axis can be perfectly described by four values. Aquitium on the other hand is nothing more than an expression of this kind. It has a real part and a complex part and it also can be uniquely described by these four coefficients. There is a closer relationship between the representation the quaternian one meaning that each coefficient of the quadian can be computed starting from the values. However, while the rotation of a point is quite intuitive in the representation is not as straightforward with quaternians. Nevertheless, each quatnian can be associated with a rotation matrix that physically expresses the rotation of a point or in this case of the gashion itself. So by using four learnable parameters for rotation and free for scale, we can construct a coariance matrix that exactly satisfy the famous two properties.

We have finally reached the end of our journey. And of course, we can't wrap things up without talking about how the color of each gausian is encoded. Just like in Nerf, the color of a gausian should depend on the view direction or more precisely on the camera on the image plane is being splatted. There is a really cool way to handle this using spherical harmonics. To understand what they are, let's start from a simple function. We all know one that given an x predicts its corresponding its law. Now a function is say to be defined on a circle if it takes as an input coordinates that lie only on the circumference of that circle. Generalizing this idea, a function is said to be defined on a sphere if it's evaluated only at points on the surface of a sphere. Well, that's exactly what the spherical harmonic functions are. Functions defined on a sphere. They have a specific mathematical formulation involving two summations over a set of coefficients that we will optimize and over a set of fixed terms called harmonic coefficients which depends on the direction at which the function is evaluated. The value of L we choose plays a role similar to what we saw earlier with fer features. A higher value of L allows the function to represent greater direction variation and finer detail. In the official implementation, the author use a degree of L equal to 4. In the formula below, I have expressed the Cartasian coordinates in their spherical form. I know it might look a bit complicated, but don't worry, the harmonic coefficients have already been premputed by someone who was done the hard work for us and they are readily available as a function of the direction. So, we can already interpret like this. A spherical armonic is simply a function that given a direction returns a value. Perfect. That's exactly what we needed. Let's dig a bit deeper. If we choose a degree of zero, we are essentially defining a function that is completely independent of the direction. In other word, if the constant value it predicts corresponds to say a red, then the function will always predict red no matter the direction. As we increase the degree of the harmonic function, we start to see a wider range of possible variations in the predicted value depending on the direction. So in this hand what we will do is to use spherical harmonics to determine each individual color channel of the gashion. In this way, the color of the gashion becomes dependent on the view direction. Exactly what we want.