Solution: Vector Search with Weaviate

Now that you've taken the time to work on this challenge, allow me to demonstrate my solution. So, in this notebook, we've got our four questions at the top here. We're going to go in and load up our data set. So, for this data set, the URL is a bit different because this is the jeopardy-like 1000 data set. So, we've got 1000 question and answers in here, significantly larger than what we've been working with. But the process to load it in is quite similar to what we've been seeing. We're going to go in here and load up our data set, and we'll look at one of the questions in there. We now have the airdate for this particular question. We have the round it was used in the value of that question, and like before, we've got the category question and the answer. So, we're going to go ahead and start up with Weaviate as we've been doing. And then we're going to see if the question class exists. If it does, we're going to get rid of it because we want to create our own question class. In this solution, we're going to do things a little bit differently, because I want to give you a different flavor of how you can create different schemas. So, here, we're going to define a class. So, the class definition is going to start off, and we're going to specify here the class. So, this is similar to what we were doing before. We're going to have the question class. We're going to go in and specify the vectorizer. The Vectorizer is the module that's going to specify the model that will be used to turn our question text into vectors. So, here, we'll use text2vec-openai which is why we had to include the OpenAI API key at the beginning here as well. If you have yet to get that set up, you can look into the Read me file for this GitHub repository. The next thing that we're going to do is set up the vector index configuration. The index config. And here, we need to specify which distance we wanted to use. So, usually, if you're comparing text, we want to set it up so that it uses cosine distance. So, once this is done, we can go in and specify the properties that we want to be imported into our data set. So, in this particular data set we've got air date, round, value, category, question, and answer. We're not going to use all of these. And the question specifically tells us to keep the answer property, the question property, and the round property because that's what we're going to be using later on. The other stuff is not that important, so we're not going to bother bringing it into our vector database. So, the properties of interest to us here are we're interested in bringing in first property, which is going to be called. Here we're going to name this thing question and the data type for this is going to be the text field. The next thing that we're going to bring in is our answer. The data type for this. It's also going to be text. And the last field that we want here is the round. The data type here is also text. So, the schema here specifies what type of data we want to store into our database and what that data looks like. So, here these are all text fields. So, we just specify that. We're going to now use this class definition, and we're going to create the class in our schema. We're going to go ahead and run that, and that creates that class. Now, what you want to do is iterate over our data one by one, using this for loop and insert into this schema data point. So, here, all we need to do is go in and create individual objects. So, we'll go ahead and have an object body, and we create. And this is just going to be our dictionary. And we have to pass in each one of these properties that we've listed out. And we've told Weaver that we're going to give to it. So, here the first one is question. The next one here is answer that we've told it that we're going to give it. So, here, we're going to pass in the answer as well. And the last one we told it we were going to provide it was the round information, and that is located in this location. So, now that we have the object body, we just have to insert it into Weaviate. So, we can say batch add data object. And we're going to specify the data object here as object body. And we're going to tell it which class to insert this into. So, we'll say class name, and the class name here is question as we've specified up here. Looks pretty good. We're going to format this nicely and we'll run this. And this is going to take a little while because we're adding a thousand objects into our vector database. So, as a part of that process, it's going to vectorize all thousand objects one by one and insert them into the vector database. And that's why this cell took a longer while to run. So, the next thing that we want to make sure is that all of our data points are actually in the vector database. So, to do this, we want to make sure that the count of objects in the database is consistent. It needs to be 1000. So, here, we'll go ahead and print out how many data points we've got in there. So, we're going to go ahead and query the client. And this is going to be an aggregation of information in the question class. And this is going to be with meta account. And then, we want to perform this query like so. We're also going to go ahead and indent this by two so that we can read it nicely. So, notice how once we ask it to count, how many objects were there in the database and told us a thousand. And this is in line with our data set. So, this data set has 1000 objects as we saw here. We've successfully inserted a portion of them, the question answer and round information into the vector database. So, for question three, we want to search for objects that are close to the concept of spicy food recipes. And we want to show the four closest concepts here. So, again, we want to do a vector search here. So, we need to create our query. So, we'll go ahead and we'll say client dot query Get from the class. So, from the question class here, we want to extract out the question, and we want to extract out the answer with near text, like so, and then, we can pass in the concept object. And the concepts that we're interested in here are spicy food recipes. And because we only want four of the most relevant ones, we're going to go ahead and set a limit here with limit four. And we're going to tell it to perform this query. We're also going to go ahead and add the distance metrics. So, we want to know how far away these concepts are. So, we're going to go ahead and say with additional and we're going to pass it distance here. So, that it tells us how far away an object is from our query. Now, we're going to go ahead and run that query. And we get these concepts, so we get popular Pennsylvania pepper pot Mexican dishes, dishes garnished. We have Indian dishes, so we have none. So, these are all concepts of spicy dishes or food that's eaten with spicy dishes. And you can look at the distance to identify how similar or how semantically relevant, this object is to your query that you've passed in here. So, next up, we want to do something a little bit more tricky. What we want to do is search for spicy food recipe related questions that have been used in double jeopardy rounds. So, here what we want to do is known as filtering. And the reason we can do that is because we saved this round information. We have that information inserted into the database. So, we want to look for specific round values where it's double jeopardy, and we want to go in and search for spicy food recipes that also meet the criteria of being questions used in double jeopardy rounds. So, to set this up, what we're going to do is start off with our old query. So, we're going to take this query, and we're going to go ahead and start off here. Now, we need to modify this query. So, the first thing that we'll do is let's limit it to three options. Arbitrary you can modify it as well, and add a where filter to this with where. And the wear filter allows us to filter over metadata so that we can select out rounds that are double jeopardy rounds. And how we're going to specify this is by doing the following. We're going to give it a path to identify which property we're interested in filtering over. So, we're interested in filtering over the round property because we want double jeopardy rounds. So, that's where the information that we need to filter over is found. We're going to identify how we're going to compare. So we know what to keep and what to throw out. We want to keep the double jeopardy round here. So, we're going to make sure that we use the equal operator because we want rounds equal to double jeopardy rounds. And we're going to pass in a value text to specify exactly what rounds we want to keep. So, here, we'll pass in double jeopardy rounds. And this filter is going to make sure that anything that's returned is not only semantically relevant to our query, but also meets this strict criteria of being a double jeopardy round question. We can go ahead and run this. And now you can see that some of the concepts that were returned are found here because these are double jeopardy round questions. How do we know that they're double jeopardy round questions? We can go up here, and we can concatenate this, and we can say, okay, I also want to know the round information for this. So, if I re-run this query now, we get the exact same results. But now the round is double jeopardy. And with that, we've successfully answered the fourth question. This chapter introduced the concept of vector search and how you can use Weaviate, an open-source vector database to start performing vector search over your own data. In the next chapter, we'll see how machine learning models actually generate vectors and how we can practically specify these models for use in Weaviate. We'll even build an image and text search engine.