Transcription
Hello. In this video, we will cover everything you need to know about Elasticsearch. To start, we will talk about what Elasticsearch is, why it is needed, and explore the basic concepts for working with it. We will also look at how to deploy an Elasticsearch server locally, study the basic commands for working with documents and indexes, and finally, we will also look at Elasticsearch using the example of a web application with full-text search. In the description of this video, you will also find a brief summary and a link to the project repository so that you can independently repeat everything we cover here. Like and subscribe to the channel so you don't miss new video releases. Get comfortable, and let's begin.
What is Elasticsearch? To begin, let's talk about why we might need Elasticsearch in an application and what it is. Imagine we are developing a movie catalog web application, similar to IMDb or Kinopoisk. Users can view information about a specific movie or see the entire list of movies available on our site. In simplified terms, the architecture of such an application can be depicted as follows: there is a server that processes user requests, and there is a database that stores information about movies. The database can be anything, relational or non-relational; it doesn't matter now. But for definiteness, let's choose a relational database. When a user requests information about a specific movie, for example, with ID 23, they send a GET request to the path movies/23. The server processes such a request by querying the database with a query like `SELECT ID, title, and other fields FROM films WHERE ID = 23`. And when they request a list of all movies, the query might look something like this. These queries are sufficient for now. All we need is to get information about a specific movie or view a list of all movies one by one.
Viewing a list of all movies one by one in search of the desired one can be tedious. Therefore, for user convenience, it is necessary to add the ability to search for a movie by title, phrase, movie description, country of production, or something else. We can try to organize such a search using the means already available to us, the database. When the user enters a string, we can search for the movie in the films table by substring or pattern, and the query to the database might look something like this. If you have no experience with relational databases and these queries seem unclear, don't worry; it's not critical for understanding how Elasticsearch works. But if you want to understand the basics of SQL and learn to work confidently with databases, I have an SQL and PostgreSQL course for you. It is available on the platforms Pik and Udemy. This course covers everything you need to start working with relational databases, and also contains a large number of practical tasks for writing SQL code and test exercises. The course is constantly updated and supplemented with new materials. Links to the courses and a discount promo code will be in the description of this video.
And we will return to our example. In general, organizing search using database tools is possible. However, this approach has several drawbacks: queries can be slow with large volumes of data, it is difficult to implement effective search with composite conditions, and search results may not be entirely accurate and relevant. We need full-text search. If you haven't encountered this concept before, in brief, it's a type of search that allows you to search for text in documents by keywords or phrases. With full-text search, the content of the text is analyzed, and the most relevant documents are returned as a result. This is where Elasticsearch comes to the rescue.
So, what is Elasticsearch? In short, Elasticsearch is a document store with the ability to create full-text indexes for subsequent searching, most often used as a search engine. Elasticsearch adds to the capabilities of the Apache Lucene library, on which it is based, features such as sharding, replication, a convenient JSON API, and many other improvements. This makes Elasticsearch one of the most popular solutions for full-text search. With Elasticsearch, the structure of our application will change slightly. We will have a search engine. How does this change the logic of our application? First of all, we need to build an index in Elasticsearch for the existing movies so that when we query it with some request, we get a list of movies satisfying that request in response. It is enough for Elasticsearch to return only the movie IDs in response to our queries, as with these IDs, we can easily and quickly get the movies themselves from the database. Next, we must keep this index up-to-date, meaning when information about movies changes in the database, we need to update the corresponding information in Elasticsearch, delete the corresponding record when a movie is deleted from the database, and so on.
How does the logic of our application change in this regard? When a user requests a movie by its ID, the server sends a request directly to the database, as before. Elasticsearch is not needed for such an operation at all. But if the user sends a search query, i.e., tries to find a movie by its description, the logic changes. Instead of an inefficient query to the database, we send a search query to Elasticsearch. In response, Elasticsearch returns a list of movie IDs satisfying this query, ranked with the most suitable documents first, then less suitable ones. With these IDs, we go with a simple query to the database to get the movie records and then return these documents to the user. Adding a search engine like Elasticsearch expands the search capabilities of our application but also increases its complexity and maintenance costs. However, if search is one of the key components of your application, the benefits of using Elasticsearch will easily outweigh all the drawbacks, as we will see later in a practical example. In the meantime, let's move on and talk about the basic concepts in Elasticsearch.
Basic Concepts. To start working with Elasticsearch, we need to familiarize ourselves with the basic concepts: indexes, documents, and queries. Let's go through them one by one.
Index. An index is used to group data into logical structures. We can have an index for movies, an index for products, or some other index. If we draw an analogy with relational databases, it's like a table in a relational database, only more flexible, as it can store documents with different schemas.
Documents. Documents are the actual records in an index. If we compare again with relational databases, it's analogous to a row in a table, but with its own differences.
Queries. Queries in Elasticsearch are used for searching and filtering data in indexes. With queries, we can find documents that meet certain conditions, sort results by relevance, perform aggregations, and much more. If we compare with a relational database, it's like regular SQL queries, only they look different.
In addition to these basic concepts, when working with Elasticsearch, it is also useful to understand how shards and replicas are structured, what analyzers exist, how ranking works, and much more. However, this goes beyond the scope of this video. If you want to see a video on these topics, write about it in the comments, and I will understand that the topic is of interest to you. For now, these concepts will suffice.
Installation and Launch via Docker. Let's now look at Elasticsearch in practice, and the first thing we need is to launch it locally on our computer. The easiest way to launch Elasticsearch locally is to use Docker. With Docker, you can quickly set up and launch a container with Elasticsearch without the need for manual installation and configuration. If you are not familiar with Docker, you will find a link to a video in the description of this video that explains the basics of Docker in 20 minutes. Let's go to the terminal. We type `docker run` to launch the container. Then, we map the container's port to the host's port. For Elasticsearch, we need port 9200. Let's set the environment variable `discovery.type` and set its value to `single-node`. This way, the Elasticsearch cluster will consist of one node. We are launching Elasticsearch for ourselves, not for production, so this mode is suitable for us. And finally, we write the name of the Elasticsearch image and the required version. Let's take version 7.17.2. We see that the container is running. We can check that everything is working as it should by going to the browser at `localhost:9200`. In response, we see information about the running Elasticsearch instance, the current version, the version of the Lucene library, and other information. This means everything is launched correctly. Congratulations, we have just set up a local Elasticsearch and are ready to start working with it directly.
Elasticsearch API. Let's now move on to interacting with Elasticsearch. Interaction with Elasticsearch occurs over the HTTP protocol using commands like GET, PUT, POST, and others, depending on the operations performed. That is, we can interact with Elasticsearch, for example, through the console using the command-line utility `curl` or through a convenient graphical interface like Postman. I will use Postman.
Indexes. Let's create our first index. Recall that an index in Elasticsearch is a structure similar to a table in a relational database. An index is used to store, search, and analyze data. In general, an index in Elasticsearch can be created implicitly: if you add a document to a non-existent index, Elasticsearch will automatically create that index. However, it is better to create it yourself. This way, we gain more control over the data structure, can specify which analyzers will be used, which fields should be indexed, and much more. To create an index, we will send a PUT request to our server. We write the address where the Elasticsearch server is located, which is `localhost:9200` for us, and then the index name. Let it be `first-index`. In the request body, we can specify various index settings, such as the number of shards the index is divided into, the number of replicas, define custom analyzers, and so on.
Mappings. Mappings are used to define data types. Let's define only the fields and data types in the mappings, and leave the settings and other parameters as default. To do this, we write `mappings`, and within it, we specify `properties`, where we specify the fields themselves. Let's say we will store information about products in a store. Let's define the `title` field of type `text`. We will store the title in Russian, so we will use the `russian` analyzer for this field, which is available out of the box in Elasticsearch. We will also add a `price` field of type `float` and information about the availability or unavailability of products as `boolean`. We will execute the request and see the response that the index has been successfully created.
Adding and Updating Documents. After we have created our index, let's add documents to it. To add a document to an index, you need to send a PUT request to add that document. To do this, we write `localhost:9200`, then the name of the index where we are adding the document, which is `first-index` for us, then `_doc`, and then specify the document ID. For us, it's the first document, so let it be `1`. In the request body, we specify the fields and information that we want to save: `title`: "Wireless headphones", `price`: 4999, and `available`: `true`. Let's execute the request. We see that the document has been successfully added to the `first-index`. The result of this request is `created`, meaning a new document was created. If we send a request to the same path but, say, change the price to 5999, we see the status `updated` and that the document version has been updated, meaning that as a result of such a request, an existing document was updated, not a new one created.
Creating a Document. As you can see, the same command can both create a document and update it. This can be convenient. However, this behavior also has a downside: we can accidentally update or overwrite a document with data without noticing it. Therefore, when creating new documents, it is better to use the `create` method. Its difference from the previous method is that if a document with such an ID already exists, an error will be returned in response. Let's execute a request to create a document. To do this, we add `_create` to the previous request. Let's execute the request. We see an error in response that says the document with this ID already exists. If we now change the ID to `2` in our request, we already see that the request was executed without errors.
Getting a Document. To get a document by ID, you need to perform a GET request to retrieve the document by its identifier. The request body is not needed here anymore. Let's execute the request. We see the document with index `1` in response. Let's try to get the document with index `2`. We see a new result. If we try to get a document with index `3`, we see that the document was not found.
Deleting a Document. To delete a document in an index, the HTTP method `DELETE` is used. When trying to delete a non-existent document, just like in the previous request, we get `not_found` in response. If we change the ID to one that exists in our index and execute such a request, we see the result `deleted`.
Search. Search is the key function of Elasticsearch. To perform a search on documents, a special `_search` endpoint is used. If we now execute a GET request to `localhost:9200/first-index/_search`, we will get all documents in the index in response, as we have not specified our query in any way, and all documents fit these search criteria. Currently, we only have one document in the index, and it will be difficult to see the search capabilities on such a set of documents. Let's add another 100 documents to our index. To add such a number of documents, we will use the Bulk API. To do this, we send a POST request to add multiple documents. We write `localhost:9200/first-index/_bulk`. In the request body, we need to specify the values to be added and the metadata line by line: into which index and under which ID we are adding them. I already have a template for this data. Let's paste it into the request body. At the end, there must be an empty line. [Music] Let's execute the request. We see that the command executed successfully. Let's go back to the tab with the document search request and send the request again. We see that the count is now 101 out of 1. We just added 100 more. In response, however, not all 101 documents are present, but only the first page. In the request body, we can specify the offset and the size of the output. Let's add these parameters to the request body. We write `from`, set it to `20`, and limit the output size to the text in the `title` field. To do this, we write `match` and the field we are searching by, which is `title`. Let's try to find something wireless. We write "wireless" and execute the request. Now, in the response, we see two documents that satisfy our search. In each document in the response, besides the data itself, there is also a `score` field, a parameter indicating how much more or less a particular document satisfies the search conditions. Currently, both documents have the same score, as we searched by the word "wireless," and it is present in both documents. In this regard, the documents are the same. Let's change our query slightly and search for "wireless headphones." Execute the request. We see three documents in the result. Now the documents have different scores. The document with the title "wireless headphones" is best suited for us, which is generally expected. But "headphones" and "something wireless" also fall under the query. We have a mouse, but they already have a lower score. Note how the documents are ranked.
We can search for documents not only by text but also, for example, by price range and availability or unavailability of a product in the store, and also combine multiple conditions into one. For example, let's write a query to search for products in the price range from 15 to 50 that are in stock (`available: true`). To do this, we write a `bool` query, then `must`, and in the array, we list the conditions that must be met. First, we will write a range query for the `price` field, and second, for an exact match of the `available` field with the value `true`. [Music] Let's execute the request. We see four documents in the result that satisfy this query. The price of each of the found products is in the range from 15 to 50, and the `available` value is `true` for all of them.
The search examples considered are just the tip of the iceberg of all Elasticsearch capabilities. However, a full overview of the syntax and search features deserves a separate video. For now, these examples are enough for us to see how Elasticsearch works in practice.
Example in a Real Project. Now let's consider how Elasticsearch works in a web application. Elasticsearch is not tied to any specific language or framework. Your application can be written in any programming language. We will consider the capabilities of Elasticsearch using the example of a Spring Boot application in Java. As an example, we will implement an application for searching movies, something like Kinopoisk or IMDb. The functionality will be limited to search itself. Our entire application will consist of one main web page with a search bar. Users can type and send a search query, and in response, movies satisfying the search criteria will be returned. We will display the title and description of the movie, rating, cast, and cover.
Let's consider such an application first without using Elasticsearch. We will organize the search using database tools, and then we will connect Elasticsearch and see how it transforms our application. Without using Elasticsearch, the architecture of our application is quite simple. We have a controller with two endpoints. The endpoint at the root path simply returns an empty main page with a search bar, and the `search` endpoint processes search queries. It receives a list of movies satisfying the search conditions from the service and passes this data to the model for rendering on the same page with the search bar. The `searchMovies` method is very simple; it just queries the repository to find relevant movies. The repository performs the search for movies directly in the database using such an SQL query. We convert everything to lowercase and search for a direct occurrence of the search string in the movie title or its description. We don't know how to correctly rank the results of such a search, so we will sort by rating in descending order. The `Movie` data model is a regular POJO object with the necessary fields. This is the entire code of our application. There are also scripts. I took the top 250 movies from Kinopoisk as a basis, as well as the HTML template for our page, but there is nothing interesting there. To package this application into a container, the `jib` plugin connected in `pom.xml` is used, and launching this container with the necessary database and Elasticsearch services is done using `docker-compose`. I am not dwelling on all this, as it is not important now, and you will find a link to the code repository in the description of the video.
Let's package our application. We type `mvn clean compile jib dockerBuild` and launch it with the command `docker-compose up`. For now, we don't need to start the Elasticsearch service, as it is not used in this version of the application at all, so we will only start the application itself and the database. We type `app` and `db`. For the first launch, it will take some time to download the missing images. We see that the application is running. Let's go to the browser and look at it. Let's go to `localhost:8080`. We see the main page of the site with a search bar. Let's consider various search queries. Let's start with this one. We see that 47 movies were found. The first result is "The Shawshank Redemption." In the movie description, there is the number one in the value 1.00, so our naive search returned this movie. Next, we see "The Intouchables," a quite relevant result, and then many other movies where "one" simply appears in the text, and this is not what we would expect to find. Let's try searching for "cuckoo." There are no movies with the word "cuckoo" in the title or description, so we find nothing. Let's try "green." We see two movies in the search result. In each of them, the word "green" appears in the description, so these movies are suitable for us. However, our naive search cannot find the movies "The Green Mile" or "Green Book" because the word "green" is in a different form there. Let's try searching for the movie by the word "brother." Here we see that as many as 26 movies were found. However, the relevance of the results leaves much to be desired. In the description of some movies, the word "brother" is present, but it's still not what we need. And the movie "Inception" ended up in our results because the description contains the word "reverse," and "The Diamond Arm" contains the word "take." And finally, let's try to search for movies about the sixties and eighties. Suppose we are in the mood to watch something from those times. We type "1960 1980." We find nothing for such a query. As you can see, the search generally works, but its quality leaves much to be desired.
Let's stop our application and change its logic to one where search by a visible query will be performed using Elasticsearch, and then, based on the documents found in it, we will get data from the database by their IDs. This logic is implemented in the `elastic-search` branch. Let's switch to it: `git switch elastic-search` and see how our application looks now. We have an additional utility controller and service: `IndexController` and `IndexService`. They are needed for the initial indexing of all documents. When we call the `reindexAllMovies` method, we extract all movies from the database and send them to the search index. The main controller has practically not changed. Only one line has changed: instead of searching for movies using the old method, we call a new method `searchMoviesViaElastic`. Let's look at it. First of all, we go to Elasticsearch with our search query. Then, we create a map between the movie ID and its position in the Elasticsearch output for subsequent sorting of records from the database. And then, we directly get our movies by ID from the database and sort them according to the Elasticsearch output. The most interesting part here is in the `ElasticsearchRepository` in the `searchByQuery` method. Let's look at it. This method forms such a search query for Elasticsearch. In the query, it first searches for the most relevant documents by keywords in both fields. The weight of the movie title is higher than that of the description. Then, it boosts the results where exact phrases are found in the title and description so that such documents get a higher weight. This is just one of the possible queries. You can independently configure and try your own search models, testing hypotheses about which fields to assign a greater weight to, which a lesser one, and which to ignore.
Let's recompile our application and launch it again. Let's go to the console: `mvn clean compile jib dockerBuild` and launch it with the command `docker-compose up`. Now all components will be needed, so we just type `docker-compose up`. We see that the application is running. Let's go to the browser and try all the same queries as last time. Let's type "one." We see an empty output. This is because we have not indexed our documents in the search index yet, so it is empty. But we have foreseen this and created an endpoint that handles reindexing, takes all movies from the database, and sends them to Elasticsearch. Let's start this reindexing. We type `reindex`. We see that the reindexing is complete. It all happened quite quickly, but there are not many movies in our database. Let's go back to the search and repeat the query. We see three movies that fall under the search criteria. First, we see "The Intouchables," as desired, since "one" appears in the title, and we give this match a higher weight in our query. Next is "12 Angry Men." Here, in the description, there is a distribution of votes 11:1, and the movie "Rush" about Formula 1, where "one" appears in the race title. There are no random movies where "one" simply appears somewhere, as in the naive search. Next, let's type "cuckoo." Now we found the movie "One Flew Over the Cuckoo's Nest." The word "cuckoo" in its original form is not present in the movie title, but Elasticsearch performed stemming of both words, so specific word forms do not affect the output. Next, let's type "green." Now "The Green Mile" and "Green Book" are at the top. These movies were completely absent in the first version of our search. Now, this is okay. And, as in the previous version, there are "Shrek" and "Scent of a Woman." Let's try "brother." Now the output looks as it should. First, the movies "Brother" and "Brother 2" appear in the output, as the title has a higher weight than the description in our search. And then come movies where "brother" also appears in the description. Moreover, we see that these are "American History X" or "Rain Man," meaning movies where the plot is significantly tied to the theme of brothers, and not random occurrences of parts of words, as it was last time. And the query "1960-1980." Here we see movies about the sixties and eighties. There is "Scarface" and "Green Book" with the service. As you can see, in all these examples, the accuracy and relevance of the search, when using Elasticsearch, significantly exceed the level of search based on a regular database search.
This was Elasticsearch in 30 minutes. Here we covered the basics of Elasticsearch. However, many topics remain that were not covered. The full capabilities of search, ranking, as well as an overview of the ELK stack, of which Elasticsearch is a part. These topics require a separate, deeper consideration. If you liked this video and want a continuation, then give it a like and write about it in the comments. As always, thank you for watching, and see you in the next video.