What is Docker? How to install it? How to write your own Dockerfile? How to forward ports from Docker to the host machine, or, for example, transfer a file from the host machine to Docker? We will discuss all of this in our video today.

Hello everyone, dear friends! You are once again on the channel "Pavlin Sharit." I am Nikolai Pavlin, and in today's video, we will talk about what Docker is and how to use it. I will try to cover all the basic aspects, starting from installation and ending with deploying an application on Docker Hub. We will go through the entire process that you will likely need when working with this technology.

Why is it important to know this technology? Well, first of all, most employers require it. They require it because they have long been packaging their services, applications, and so on into Docker containers, which are deployed using some orchestrators, such as Docker Compose files, and so on. This technology is in demand; it significantly simplifies everything related to deploying applications, whether in a development environment or a production environment. Therefore, it is highly likely that your employers will expect you to be familiar with this technology.

Let me quickly outline how this course will be structured. I will try to break it down into logical blocks. Each logical block will include a small part of the presentation, and then we will look at how to work with it in practice. So, it will be a combined approach, not just separate practice and theory.

Now, let me share what I have prepared for our course. First, we will start with an overview of what Docker is, why it is needed, and how it differs from virtual machines. Next, we will look at how to install Docker. I should clarify that I will be demonstrating everything using macOS, but I hope that there should be no issues with installation. I will provide links to detailed guides in the official documentation on how to perform the same steps on other operating systems.

After we install Docker, we will run our first container and understand the difference between an image and a container. Then, we will practice forwarding ports from the Docker container to your host machine. We will also practice how to ensure that certain files within the Docker container are saved on your host machine. For example, if you have a database, you would want it to not start from scratch every time.

Next, we will discuss what a Dockerfile is and how we can create our own image, which we will then run as a Docker container. We will explore this in the context of a FastAPI application, which I periodically show you when we talk about FastAPI.

Once we have a solid understanding of the Docker utility, I will introduce you to another abstraction or utility, to be precise, Docker Compose, which allows you to manage a set of containers using a single configuration file. We will try to write such a file together and see how the commands we learned earlier are described within the configuration file.

But before we move on to the theoretical part, I would like to recommend that you subscribe to my Telegram channel because I often communicate there, and you have the opportunity to directly influence the content that is released on this channel. The topic of Docker was suggested by my subscribers, and they voted for it to be the topic we would cover.

So, we have finished the introductory part. I think we can now start mastering Docker. Let's move on to our first slide, where I would like to explain what Docker is, why it is used, and how it mainly differs from virtual machines.

But before we get into that, let's imagine how we used to deploy all applications. When I was just starting as a developer and didn't know how to write code, we also deployed our applications in a similar way. We had some virtual server, or it didn't matter if it was virtual. We had a server where we ran some service to support and start the application. One of the popular mechanisms was Supervisor.

So, we have a Python application with a requirements.txt file. We log into that server, install Python, making sure to install the correct version, and then we install all the dependencies. After that, we start our application using Gunicorn or some other server. Then we realize that we want to run another application on this virtual machine, but it requires a different version of Python. We install the other version of Python and realize that we have dependency conflicts, and everything goes awry. We then want to download another server and deploy everything separately.

Managing all dependencies, whether at the language level with libraries or at the system level, can lead to conflicts. Python solves this through virtual environments, but we can also have dependency issues at the system level. For example, we might need one version of a library, while another application requires a different version.

Thus, we understand that we want to keep all our applications sufficiently isolated to avoid any conflicts with dependencies or application functionality. This is precisely the problem that Docker aims to solve. It is a platform that includes everything necessary to create isolated applications, referred to as images and containers, which we will discuss shortly. It allows us to store these images in repositories so that we can download and run them when needed.

On the second slide, you can see this mechanism: we have some registry on the far right where applications are stored as images. We can download them and run them isolated from each other, and this entire story will not conflict.

If we talk about the difference between virtual machines and Docker, let's first clarify what a virtual machine is. A virtual machine essentially provides the ability to create a fully isolated computer where we allocate a certain amount of RAM, CPU cores, and we can also define network interfaces, emulate drives, and so on. To do all this, there is software like VirtualBox. If you haven't done this, you would choose an operating system, allocate resources, install the system from scratch, and so on.

So, it is presented as a completely isolated computer, but we understand that if it is a separate computer, there are quite a few overhead costs associated with its launch, and it takes up a lot of space. In our examples, we will have a simple case, which is quite resource-intensive for relatively simple tasks.

Docker, on the other hand, works within the host machine and allows us to run applications isolated from each other in terms of dependencies, but without the additional overhead costs associated with launching such applications. Therefore, Docker provides a much more efficient and scalable model for running and executing applications compared to virtual machines.

It seems great, but why do we even need virtual machines? Let's imagine a case where a company has a diverse environment, which is also our situation. Some applications need to run on Windows because they are written in .NET, and a web server is used, while other applications, like in our case, are written in Python, and so on. In this case, we would need to take one server, simply because it is usually more cost-effective to run multiple virtual machines on it—one for Windows and another for Ubuntu—where we can isolate all our applications using Docker.

So, we cannot completely abandon virtual machines, but Docker provides us with a convenient way to use and run applications, especially on these virtual machines.

To summarize the first introductory part: we cannot abandon virtual machines because they solve certain tasks. When we need to divide a powerful server, we want to install Windows, Ubuntu, and Arch Linux, or something else. Docker, on the other hand, is a technology that allows us to create isolated applications, store them in a special way, download them, and run them on the host machine. This provides opportunities for scaling and easy application transfer.

Now that we have talked a lot, let's figure out how to install everything. To install Docker, you can Google "Docker," and you will find its main website, which looks like this. Scroll down almost to the very end, and you can click the "Download Docker" button, which will take you to a page called "Get Docker." Here, you can download the application for your operating system. For Windows, it will download an installer, and for macOS, it will download a .dmg file that you will install.

If you go inside, there is a detailed mechanism for how to install everything. I won't reinstall it all, but I will show you the result of our installation. After we run this application, we will have a new application called Docker Desktop, which will represent a web interface. Let me show you that. In this interface, you will see all the images and containers, and we will talk about what they are later. This is a visual representation of the console utility that we will learn to use later. I personally prefer to use the console, but both approaches are valid.

Your task at the moment is to Google "Docker," go download it, and install it. I hope you won't have any issues.

Now, what we see in the left menu focuses on containers and images. These are important abstractions that we need to understand in order to move forward.

Let's first clarify what an image is. An image is essentially a packaged application ready to run. This is what distinguishes it from a container. If we draw an analogy with something more familiar, it's like having installed games on your computer. You have all the necessary files, but the game isn't running yet. You can double-click it, and the game starts. You can also run this game in multiple processes or windows. In Docker, this is how you run multiple containers.

It's important to note that even though you are using the same image, each container you run will operate in isolation from the others. Each image we create has several examples, such as backend and frontend images that we created during a hackathon. I also wrote about this in my Telegram channel.

Now, let's look at the image details. We can see when it was created, its size, and that it is 1.7 GB. We have a set of actions, such as deleting or running our image. If a container is currently using this image, we won't be able to delete it because there is a dependency.

Let's see what an image or layer consists of. I realize I have been using anglicisms here, but I think it's acceptable. When we click on the name, we see a new abstraction we haven't encountered yet: layers.

Based on the fact that we have 47 layers, we can deduce that all these images consist of layers. This is also a significant feature of our Docker containers. Each layer essentially represents a command executed that changes the state of this image.

Let's break it down in simpler terms. You have a bare operating system, and you want to install something. If you did this from the console, you would write `apt-get install` and your operating system's state would change after executing that command. Each line in the Dockerfile represents the execution of a command.

For example, if we have a command to install a library, set environment variables, and so on, each of these commands forms a new layer of our image. Why is this important? As I mentioned, this is a crucial abstraction, and it allows us to cache each of the layers. If we have commands that install dependencies, we only need to execute that command once for the image.

If we scroll down, we might have a command to copy Nginx settings. If we change the Nginx settings and want to recreate the image, the layers that haven't changed won't need to be executed again. You can believe me when I say that executing these commands, especially when installing system dependencies, can take a long time.

So, we understand that Docker container images consist of layers, and we can operate with them when creating our own image.

The last thing we haven't looked at is how the containers tab looks. Let's correct that and move to it. I want to draw your attention to the fact that I have filtered out some working projects. Please don't peek; I have containers running.

Here, we can see the familiar names of images, such as backend and frontend, which we discussed earlier. We can also check their status, whether a container is running or not.

Let's summarize again: we have Docker container images, which represent packaged but not yet running applications. These images consist of layers, and each layer is a specific command that changes the state of the container. A running image is called a container.

Now, let's move on to the practical part. We are going to run our first container and download our first image.

As I mentioned, we looked at how Docker Desktop looks, understood what images and containers are, and now let's try to run our very first container. The difference between images and containers has been explained, and we will see how to do this not through the visual interface but using the console, as you will likely be working with the console.

Why? Because if you log into servers and try to run Docker containers there, you won't have a visual interface. I want to emphasize that I strongly recommend you not just watch this video or at least watch it and then go home to practice everything we do together. Without practice, you will probably only be able to answer questions in interviews, but unfortunately, you won't be able to repeat anything without doing it yourself.

So, how can we run our first container? Let's open the terminal.

We have opened the terminal, and now we can type `docker run hello-world`. Let's see what happens.

Magic has happened! Let's break down what we have executed. The `docker` command appeared in the terminal after we installed Docker Desktop. The `run` command executes the container. The next argument after the `run` command is the name of the container, which is `hello-world`.

Looking at the logs, we see that `hello-world` was not found locally, so it was downloaded. The result of executing our container is a message saying "Hello from Docker!" and some meta-information.

Now, where did I get the name `hello-world` from? I didn't just make it up. Can we write anything we want after the `run` command? To find out, let's go to the browser.

You can go to Docker Hub, which is a single storage location provided by Docker where various images are stored. You can also add your own images there. If we draw an analogy, it's like GitHub for source code, while Docker Hub is for images.

If we look at the top right corner, we see another command that is not the one we executed. It says `docker pull hello-world`. The `pull` command, as you can guess, is responsible for downloading an image from some repository to our local machine so that we can then run containers from it.

As we saw when executing the `run` command, Docker checks if the image is available locally. If not, it executes the `pull` command behind the scenes and then runs the container.

To summarize at this point: Docker Hub is a single storage location where all Docker images are stored, which we can download and run. We can look for existing images using the search function.

Now, let's return to the terminal and try to execute the command again. Let's see what it outputs. You can voice your guesses out loud or write them in the comments.

Let's try typing `hello-world` again, and we see the output indicating that the status of this image has already been downloaded. It was downloaded when we executed the `run` command, so we don't need to do anything with it.

When we executed the `docker run` command, something happened. The creators of this image wrote that when the container starts, it will output the following lines. This is normal behavior.

When we discuss how to create an image for our FastAPI application, we will specify the behavior, such as "please run this command and start the web server on port 8000."

So, we have learned how to download images and run containers. But how do we see what is currently running? For that, we have the `docker ps` command, which will display all currently running containers.

Let's see what we have. We have the ID, the name of the image, the command that was executed when starting this container, when it was created, its status, and the names of the running containers.

Great! We have learned how to view containers and run them, but we would also like to know how to stop them. Let's learn how to do that.

For this, we will need the ID of the container, and as you can guess, we have a command called `docker stop`. We will pass the container ID as an argument and execute our command.

Great! The container stopped in 10 seconds. Let's check that it is no longer running. We see that the container with the image we used is no longer among the available containers, so our stop command was successful.

Now, can we see the images we have on our local machine? First, we can understand what else we might want to run. Secondly, as we saw earlier, images take up a significant amount of space, so you might want to periodically get rid of images you no longer need.

For that, there is a separate command, and you can list all the names of the images on your local machine. For example, I have an old version of the `hello-world` image that we downloaded, along with some other working images.

Let's say you have a very large image, like 10 GB. If you want to delete it, you can use the command `docker rmi` followed by the image name to free up disk space.

So, it seems we have learned how to run images, create containers from them, and stop containers. But where is the practical benefit of all this?

Let's try to apply our knowledge in practice and see how it speeds up all processes related to development. I remember when I was a student, and later when I explained to my students the benefits of Docker, I used the example of running a database.

We learned about Oracle, and installing Oracle, especially on Windows, was a challenge for many. It required a lot of steps, and it could take hours or even days for some.

Now, let's see how we can run PostgreSQL using Docker and look at the commands we haven't fully explored yet.

Let's write `docker run -e POSTGRES_PASSWORD=postgres -d postgres`.

We encounter an error. It says that it would love to start PostgreSQL, but we didn't provide the password for our superuser, so PostgreSQL cannot start.

This introduces us to a new mechanism: environment variables within Docker containers. They work similarly to environment variables on our host machine.

To set an environment variable, we write `export VAR_NAME=value`, and then the variable will be available to any process in our operating system.

In Docker containers, we can set our environment variables, and here it suggests the syntax for how we can do this. Let's write `docker run -e POSTGRES_PASSWORD=postgres -d postgres`.

Now, we see that PostgreSQL has started.

What problems might we encounter at this stage? Okay, we started the PostgreSQL container, but if our application is running locally, we need to connect to it somehow.

We will look at two main mechanisms that will help us in this process. The first is how to find the IP address of our container, and the second is how to conveniently forward ports from our container to our local machine.

Before we do that, let me introduce you to a command option. If you noticed, when we started our application, the terminal was blocked by the running Docker container process. This is not always convenient, so I will show you how to avoid this.

Press `Ctrl+C` to stop our container. Now, to avoid this problem, there is an option called `-d`, which stands for detached. This option allows us to run the container in a separate process.

Let's run it again, and we see that we have been given the container ID. Now, the terminal is fully available.

Now, we can check the logs of this container. The command `docker logs` allows us to view the logs of a container.

Let's copy the container ID and run the command `docker logs -f <container_id>`. The `-f` option allows us to output not just a batch of logs that are currently available but all logs that this container outputs in real-time.

If you have a web server running, you can see what it is writing.

The last part of the options I want to discuss is when we want to output only a certain number of lines at the end.

Now, we have learned how to run a container in a separate process, view its logs, but how do we connect to it?

The first thing we can do is use the command `docker inspect <container_id>`. This will give us all the metadata about the container, including environment variables and configurations.

We look for the parameter called "IP address" and see that our container is running on IP address `172.17.0.2`.

Now, let's try to connect to our database using this address.

I wanted to show you how to connect to a container by IP address, but I forgot that this doesn't work for macOS and Windows.

However, this information is very useful when you have a database or another container on a remote server.

Often, you might want to install something, and it would be beneficial to have that IP address on a remote server.

Let's see how we would connect if it were a remote server.

We open our database management tool, and we need to add a new Data Source.

For connecting to a remote server, we need to go to the SS tab and use SSL to open a connection, sending all traffic to an external server.

Here, we would select a configuration, specify the server address, the port we are connecting to (usually 5432), the username, and the password we set earlier.

This way, we could connect to a remote server and access the database.

Now, we have not resolved the question of what to do when we have a local setup. For that, there is a mechanism called port forwarding.

Let's return to the terminal and see how we can forward these ports.

Port forwarding allows us to take a port from the Docker container and make it accessible on the host machine.

Let's break it down with an example. We have PostgreSQL running in a container, and we want to connect to it on port 5432.

We can make this port accessible on our host machine.

It's important to note that we can forward the port to any other port on the host machine.

For example, we can say that localhost:4949 will forward all traffic to the Docker container on port 5432.

This way, we can connect to our database using localhost:4949, even though it actually leads to the desired Docker container.

To see how to execute this command, let's first stop our current container.

Now, let's run a new PostgreSQL container with port forwarding.

We execute the command `docker run -e POSTGRES_PASSWORD=postgres -p 4949:5432 -d postgres`.

Now, let's check the running containers with `docker ps`.

I want to draw your attention to the column we haven't focused on before: ports.

Here, we can see all the ports that are mapped from the host machine to the Docker container.

Now, we have PostgreSQL running, and we need to check if we can connect to it.

Let's go to PyCharm and try to connect to it live.

We return to PyCharm, click on databases, and set the host to localhost.

The port is 5432 because we are forwarding it.

The username is `postgres`, and the password is `postgres`, as we set it in the environment variables.

Let's test the connection.

We see that we successfully connected.

Now, let's create a new table in the database.

This procedure is straightforward, and we can connect and develop easily.

Now, let's check the next case.

We want to add a new connection.

Let's create a new Data Source for PostgreSQL.

We can connect to it using localhost:4949, and the password is `postgres`.

We can also connect successfully.

Now, an important principle of our containers is that each new container launch does not save the state of the previous one.

So, we can summarize that launching or restarting a container starts it as if from a clean slate.

This is an important property that plays to our advantage.

Now, we smoothly approach the next topic: how can we ensure that our Docker container retains some data?

For applications that store data, we would not want all data to be lost.

This abstraction is called volumes.

Volumes allow us to map some storage from our local machine into the Docker container, making it persistent.

For example, a database stores all our data in optimized binary files on the disk system.

To ensure that our data, tables, and their contents are preserved, we need to save a specific folder somewhere that we can map into each Docker container.

When the Docker container starts, it will see these files, and the database files, tables, schemas, and so on will already exist.

This is precisely why we use volumes.

Let's first see how to create a volume, and then I will explain the two main types of volumes: named and unnamed.

First, we want to stop our containers.

Now, we will create a volume.

Let's say we create a volume called `my_volume` and map it to the PostgreSQL data directory.

The command would look like this: `docker run -v my_volume:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -d postgres`.

Now, we want to confirm that our data is preserved.

Let's create a new table in the database.

Now, we will stop our Docker container and restart it with the same volume.

We check that everything is running, and we see that the table we created is still there.

This way, we can map parts of the file system into our Docker containers, preserving all the data we have.

Now that we have learned about volumes, the next logical question is: where is this volume stored?

If we can map it into the container, it must physically exist on our host machine.

For that, Docker has an additional command to see all our local volumes.

We can use the command `docker volume ls` to list all volumes.

To see the physical folder created for our volume, we can use the command `docker volume inspect my_volume`.

This will show us the path where the volume is stored on our local machine.

We can open a file explorer and navigate to this folder to see what is stored there.

Similarly, we can pass the ID or name of our volume to delete it when we no longer need it.

Now, you might logically ask: "Great, we learned how to create volumes, but they are stored in Docker's internal directory, which is not very convenient. Is there a mechanism for us to specify where we want this volume to be stored?"

Yes, there is!

Let's stop our previous container.

Now, we will create a named volume.

We will specify a path for the volume, which can be either relative or absolute.

If we specify a relative path, it will create the volume in the current directory.

If we specify an absolute path, it will look like this: `/Users/username/my_volume`.

If we specify the path, we will create a named volume that we can manage.

Let's run the command again, and we will see that the volume is created in the specified location.

Now, we can check the contents of the volume.

If we create a new table in the database, we can confirm that the data is preserved even after stopping and restarting the container.

This way, we can map parts of the file system into our Docker containers, ensuring that all data is retained.

Now that we have learned how to create volumes, the next logical question is: where are these volumes stored?

If we can map them into the container, they must physically exist on our host machine.