Vector DB2: Hybrid search

We've learned about semantic search for similar objects, but sometimes, we need to perform exact search. Imagine you're looking for a flight with some specific flight number. Here, matching keywords is the solution. Sometimes, we want to combine searching for keywords with semantic search. How can we do this? Hybrid search allows us to get the best of both semantic and keyword search results, and works very well in practice. It works by performing both searches in parallel and then combining both returned results. You can also control how much of each you want to influence the return values by modifying a parameter. Let's code this up. So, in this notebook, you'll notice that we've got the usual boilerplate code that sets up our data set. So, we'll just run all this code. We've talked about this multiple times, so hopefully, we know what everything is doing here. I'm just going to quickly run through all of this. Import all the data into Weaviate. We make sure that we've got the ten questions and answers that we've been using. So, now we get to the topic at hand. What we want to do first of all here is perform vector search. So, we want to look for questions that are the most related to the concept of animal. So, here, we're going to go in and write up our vector search query as we've been doing to extract out those relevant concepts. So, we can go in here and we can say, okay, our response it's equal to, and then again, we add the parentheses here to code this up nicely. We'll say client I have a query I want you to get. And we have to specify where the information that we want to get, lies we want to extract out from here. The question I want to extract out the answer, and we want to extract out the category. Because this is a text search, we're going to use the with near text here. So, we'll do with near text and we'll pass in the concept object here. Concept. And here, we're interested in the concept of an animal. So animal. And then, we'll tell it to limit the search to three objects with three most relevant objects. And then, we tell it to perform the search. In this line, we're going to print out the returned results. Run that and we get three questions and answers that have to do with the concept of animals. Right. So, in this case, the concept here is an elephant. Here, we've got information about a crocodile, and then, we've got information about an antelope. Notice how here the query matches in meaning with the returned results. And that's the whole point of vector search. We want to semantically match concepts and return those concepts. The other side of the coin here is what if we only want to look for specific words in the questions and return if those words exist? So, notice here how looking for the concept of animal, it doesn't matter if the word animal is in the question or not. Sometimes, the word might be present, sometimes, the word might not be present. So, in this case, the word is present. We've got the word animal, but in the other two, we don't have the word animal. And that's okay because we've got this matching of semantic similarity. The idea, the concepts are the same, so, vector search is good enough to extract those for us. If we want to perform exact keyword search where we only want to return questions that have the word animal in them, there, we perform keyword search. And keyword search the query looks slightly different. So, what we're going to do here is code up a query for keyword search. So, we're going to say response. Query the client say query. Want you to get the same information. So, here we're going to go in and specify where the data lies, what pieces of information we want. And these are all the same. So, we want the question, we want the answer, and we want the category. Going to go ahead and this is where things change up. We're going to use a query called with bm25. Bm25 stands for Best matching 25, and this is a keyword search algorithm that goes in and returns to you questions that have the keyword that you're going to pass in here. So, we're going to pass in a query, and this is the word that it's going to match with. So, we're going to pass in the word animal. Again, we're going to set its limit to three as we did before, and we're going to tell it to go ahead and perform that operation. So, now, let's run this query, and see what comes back, and make sense of that. Run that query, and even though we asked it to return three objects, we only got one object back. And this is telling you a story. What this is saying is that you've asked directly for a keyword match for the word animal, and it couldn't find more than one. And the one that it found here, if we read the question weighing around a ton, this eland is the largest species of this animal in Africa. So, here, the word animal matches with our query and therefore this question gets returned. Notice how for the other three here, elephant, this is conceptually related to our query, and so is this. But these are not returned because the keywords do not line up. And we can't find the keyword animal in these two queries. So, that's the main idea behind keyword search. It only returns objects if the words exactly match. In practice, what we'd ideally like is to merge these two types of searches. We want to be able to perform keyword search and semantic search and then say, I want some hybrid or some fusion of these two results. And that's known as hybrid search. So, how can we perform hybrid search or get the best of both worlds. So, to do that with Weaviate, we have to slightly change the query again. So, what I'm going to do is rather than rewrite the query again, I'm going to take this query and I'm going to show you how to modify it to turn it into a hybrid search query. So, we're going to paste that in here. And then now we're not doing a keyword search here. So, I'm going to delete that portion of the query, and I'm going to go ahead and show you what the hybrid search query looks like. So, here, I'm going to do with hybrid, and now we can pass in the query again. So, the query here is going to stay the same animal. And we can also specify this alpha parameter here. And we'll talk about what that is in a second. Let's just run this, and we'll look at the results. So, now, we asked for three results. We're getting three results. Notice how the three results are exactly the same as vector search. But what's a little different now is the ordering of the three results. So, what I'm going to do is go back up to the vector search results. In this, we had elephant, and we have the crocodile result, and then, we had the antelope result. The ordering now is slightly different. The antelope is at the highest ranking, then we have elephant, then we have the concept of this crocodile question. This reordering is significant here. It might seem small, but it's significant. The reason why it got reordered was because if you look at how highly antelope was rated by vector search, if we scroll up here, it was last. But when we're doing hybrid search, we're doing both keyword search and vector search. So, vector search doesn't give antelope a lot of significance. But let's have a look at keyword search. So, keyword search returned only antelope. So, according to keyword search this is the most important concept. So, when we do hybrid search, we take both of them and we say which of these results is the most important based on both types of searches, which is why antelope, this question gets re-ranked and gets put in at the highest priority here, and the other two results get pushed down to the second and third place. And this is the idea behind fusing or taking a hybrid between keyword search and vector search. The last thing that I want to talk about here is what happens when I modify this alpha parameter. The way to understand the alpha parameter is by understanding it as the strength of vector search. So, if I set this to completely zero. So, if alpha is equal to zero, this will purely be keyword search. So, I have vector search strength of zero. So, if I run this, I get pure keyword search. This is the exact same result as when I run bm25. And when I go to the other end, when I say I want pure vector search alpha of one, all of the weighting is given to vector search. If I rerun that now, I get pure vector search. So, you can balance between keyword search and vector search by modifying alpha to control which works best in practice. Here we introduced hybrid search, which allows you to tune how much you want to match exact words versus concepts in the return results. In the next video, we'll talk about another wildly popular application of vector databases, using them to improve large language models like ChatGPT.