Solution: CRUD and performance

Now that you've taken the time to work on this challenge, allow me to demonstrate my solution. The interesting part about challenge four is that we're going to be working with a real vector database that has lots and lots of objects in here, and you get to see the scalability and the power of a vector database in action. Because this database is too big to be ran locally, we're actually going to connect to an instance of Weaviate running remotely on the cloud. And for that, we need to specify the URL. So, the client needs to know exactly where it can connect to this database endpoint. So, we're going to pass that in here. We're going to specify this URL. And we also have a publicly available API key that you can use to query this database. So, we're going to pass this in over here. The other thing that you'll need for this is a coherent API key, and the instructions to set this up yourself are in the Readme file for the GitHub repo for the course. I've already set this up, so this will be completed for me here. So, once we run this, we're going to go ahead and check whether or not the client is ready at the end here. And so, once it gives us a true that's a green light, we're ready to go and start querying the database. The very first thing that we're going to do is see how many objects are in this real vector database. So, up until now, the most objects that we've stored in the database are a thousand jeopardy-like questions. But now, let's see how many data points we have in here. So, similarly, we're going to go ahead and say, print out the answer to this query, and we're going to query our client query. We're going to aggregate the information that we have in this class, which is articles. And this is a with meta query go ahead with meta count. We're going to go ahead and get it to perform this. And then, we're also going to go on and indent this so that it looks nice. So, it runs that query. And now, we found out that there's about 9.5 million objects stored in this particular database. So, imagine having 9.5 million text objects stored in a database and then being able to search over it. So, here next, we just go over and search for things that we're interested in. So, for example, if you want to perform a semantic or vector search query over this 9 million objects and see which ones are the most relevant, we can go in and query here. We can start off with specifying where we want the information to come from. We can go into articles, and we want to extract out the text concepts from here. We want to extract out title concepts, URL concepts, and we want to look at the views property, as well as the language that this document has. I'm going to go ahead. We're going to go in and run our with near text. We're going to pass in our concepts object here. Let's say we're interested in searching up vacation spots in California. And then, we're going to go ahead and make it so that we only get back five of the most relevant objects here. Look at five, and then we'll go ahead and perform that query. So, now, when I hit enter on this code cell, this is going to go ahead and perform semantic search where it matches this concept with the closest objects that are stored in my 9.5 million object vector database. So, when it comes back, it shows me that this is the information that it found that is most relevant to what I am interested in vacation spots in California. And as you can see here, this is quite similar to what I'm interested in. So, I can go ahead and read that. Vacation spots in Los Angeles, Hollywood. You've got different beaches here. So, a lot of relevant things. The other thing that you'll notice here is that you've also got multiple languages that this text comes from. We've got English, we've got Italian, we've got French, we've got Spanish. So, this is a multilingual data set. And all of these languages contribute to this 9.5 million object count that we're seeing up there. So, the next thing that we're going to do is a little bit more interesting. We're going to go ahead and filter out specific languages. So, we're going to go in and say, I want objects only from one particular language to be returned. And that gives us more control over what comes back. So, here we're going to create two queries. We're going to perform a text query, and we're going to break that up the text query as we've been specifying, can be here. So, let's say we do easy to cook tasty meals. We're interested in this. And then, we're going to add a where filter, and the where filter's job is to make sure that only the language that we're interested in seeing results for gets returned. So, let's say we only understand English. We can go ahead and specify it a path. This is the information of language, and this is the property that it lives in. And we want to only look for the English language. So, the operator that we want to use is the equal operator. We want to see where the language is equal to English, and the value string that we're interested in comparing it to here in this particular case is English. We'll change it up later on. So, we'll set this up. So, now we've primed our query. Now, we're going to go ahead and write the query out. So, we'll go ahead and say, client dot query. I want to get information from the articles class here. And the properties that I want that are specified for me here are the text, title, the URL, views, and the language, and we can use this to verify that in fact, we did get back the language that we're interested in seeing results for. We're going to go ahead here and use the with near text. We're going to pass in our near text filter. And we're going to go ahead and use our with-wear filter, and we're going to pass in our wear filter that we crafted up here. This filter is going to make sure that we get semantically relevant objects to our query, which is easy-to-cook tasty meals. And this filter is going to make sure that we only get back objects where the language field is equal to English. We're going to go ahead and set a limit so that we only get back five results. We're going to go ahead and perform that query. So, now we can go in, run this query, and now it's filtering for all of the English language documents that mean or semantically similar to this query. So, let's see what we get back. So, here we get information about cooking, more information, beginner cooks. Read into this. So, here the other thing I want you to notice is that all of these are in the English language. Whereas before, when I didn't specify a particular language, I got back results from multiple languages. I got Spanish here, I got French here, I got English here, I got Italian. So, now I can further filter by language as well. And so, what I've done with this solution is created a nice function called semantic search, where you can pass in a query as well as the language that you want to search over. And I've just used these parameters to then curate specific objects. And I've written up another function here that nicely prints out the results. So, what you can use this function for is search for easy to cook meals, and I want only results in Japanese to be returned, and I can print those results out. So, let's see what happens when we run this. So, now here, we can get Japanese results for easy-to-cook meals. And you can go ahead and try different things here as well. Feel free to experiment with this and see what else you can get back. In this chapter, we introduced the concept of recall, latency, and memory efficiency as they pertain to vector databases. We also practice performing CRUD operations that can be used to keep our data up to date and fresh. In the next chapter, we'll see three very common applications of vector databases in industry.