Solution: Search with images and text

Now that you've taken the time to work on this challenge, I'll walk you through my solution. The solution for challenge three is exciting because we're going to build our own recommender system, and you're going to walk through the code as well. So, the first thing that we need to do is import our libraries that we're going to be using. The second thing that we need to do is start Weaviate up locally. So, what you'll notice is in the GitHub repository, you have access to a Docker compose file. We're going to use that Docker compose file to initialize our instance of Weaviate locally. So, we're going to go ahead and say Docker compose up. And that's all you need to do. You go into your terminal. You locate that file. You say docker compose up. And that should get Weaviate running in the background for you. And now, what we want to do is connect to that locally running instance of Weaviate through the Python client over here. So, to connect it we're going to write code like so. Client with Weaviate because we've already imported it we can autocomplete. And then, here, by default, we Weaviate launches up at the local host port 8080. So, we'll do localhost and then port 8080. And then to make sure that this is okay, we can go ahead and just print out whether or not this is created. Then we can pass in here statement. And is ready. So, once we've run this, we know that we've now established a connection from our Python environment here to the Weaviate container that's running in the background. The next thing that we're going to do for question one, we need to go ahead and create a schema and specify a class within that schema in Weaviate. And to do that, the first thing that we want to do is make sure that there isn't already a schema in there. So, we want to make sure that the class that we want to create is not already there. We're going to go ahead and check if there is, and if there is, we want to go ahead and delete that class. So, we're going to run this. And then the next thing we do is create the class ourselves. So, here, we want to create a class object like so, and then we can give it a name. So, you can name this whatever you'd like. I'll name it text image search. We also need to specify a config for this module. So, here, we can use different models. Weaving offers two multi modal modules. One of them is called multi2vec-clip. The other one is multi2vec-bind. So, here, because we're only performing image and text search, multi2vec-clip is enough. If you are performing audio search, video search, that type of application, then multi2vec-bind would be the more appropriate application for you. Inside multi2vec-clip, we have to specify where it will be able to locate the image field. So, we have to specify image field and then where that image field is located. So, this has to be one of the properties that's found in your database that we'll define later on. Next what we're going to do is specify a vectorizer. And the vectorizer here is also multi2vec-clip. And then come our properties. So, here, we only need two properties. So, we're going to leave space for two of them here. We'll create two of these properties. The first property that we're going to create is the name of the file. So, we have the name of the file we're going to call this text. And this is of type string. So, the data type that we're going to specify here is string because it's just the name of the file. The second thing is the actual image contents of the file. So, we're going to call this image. And what you call this has to be consistent with where you specified and told the multi2vec-clip where to find the images. So, that's why this has to be called image. And the data type here is of type blob. So, we style that, and then this should be good. We're going to take this class object and use it to create our class. So, we're all good. The schema has been created, all the code above here was able to run, and therefore, we got to the print statement. So, that means everything is good. So, the next thing we're going to do is take the data set that we've provided on the GitHub repository for this course, and we're going to add it to the vector database. And for this, feel free to actually import and include your own images into the image repository. And what that will do is instead of observing the data points that we've provided, it will take your images on your local computer, and it will upload them into Weaviate and everything is running locally, so everything is private. You're just going to be able to search over your own data, and that's a lot of fun as well. So, here, I'm going to be using my own images that we're providing in the course. But you can go ahead and use your own images to make it more interesting. So, we're going to go ahead and loop over that images folder. And we're going to grab each file one at a time. And there's a couple of things that we want to do. So, the first thing that we want to do is make sure that we can see all of the images as they're being uploaded into the Weaviate instance. So, here, I'm going to add a bit of a check that checks and prints out which images are currently being uploaded. So, this is just a print statement that's going to print out the name of the image that's being loaded up. The next thing that we need to do is encode the image into base64, which is required for any image that you want to put into Weaviate. So, here, we're going to do an encoded image version of this. And we're going to use the utility function within Weaviate to encode our image. So, all we need to do here is provide the path to the image, and it'll grab it, and it'll encode it accordingly. So, we'll provide the path to the image one at a time. It'll encode the image, and then we can pass it into Weaviate. So, before we pass it into Weaviate, we need to package it so that we have the name of the image as well as the image itself in encoded format stored in this dictionary object. So, here, we're going to specify that the image here is our encoded image, and the name for this image is like so. So, now that we have this one image packaged along with its name, we're going to go ahead and talk to the client, and we're going to say you have a data object incoming. We're going to create that internally into the database. We're going to take this data property, and we're going to tell it which class this needs to be stored in. So, the class name is here. This has to be the same as what you provided here. So, text image search is the class that we want all of these images inserted into like so. And then, we can run this for all of the images within that images repository. So, now, as you can see, as the images are being added one by one, it's telling me that this image has been added. If you take your own images and you drop them into that image repository, it'll do the same thing for your images, and you'll be able to search over your own files in your computer. Depending on how large the images are, this might take a while. If the images are smaller, lower resolution, it'll be a lot quicker. So, now that all the images are in there, what we want to do is perform text search over these images. And so, in order to perform text search, we want to write out a query or a question, or a concept that we're interested in, and then see which images have the same concept. We're going to go ahead and write our query. And we're going to style it using these brackets. So, we'll go ahead and say client, I have a query for you. And that query requires you to get from a particular class. In this case, the class is the text image search class. And what I want it to extract for me is the name of the image that it thinks is the closest to a particular concept. And then, I also want it to extract for me additional information of distance. How far away that images from my text query? Because this is a text query, we're going to be doing a with near text call here. And then this takes in a object of which concepts I'm interested in so I can go ahead and do this. And then here, we're naturally talking to the database, which is one of the highlights of a vector database. You can query them with natural language. We can just query it with cute cats if those are the images that we want. And then we limit it to three, so that we get back the three most relevant images. You can increase or decrease this number depending on how you want to interact with your data, but here, we'll limit it to three. And then we'll visualize those three to make sure that they are conceptually close to this query here. So, we'll run this, and it tells us that we've got these three images here that we can visualize to see if they are close to what we're asking for. So, to visualize this, we're going to use a IPython functionality. We're going to go ahead and say IPython display image. And then here, we just specify where this image is located. So, currently, these images are located in the image repository. And then we can take these one at a time, and we can visualize them. The other thing that you might want to do here is specify the width so that it's formatted nicely. And you can see that here is your first image that's relevant to your query that you asked for cute cats. The second image here we can also do the same thing. So, I'll take this, and I'll show all three images that we're interested in, so I'll take The second most relevant cat image, like so. And I'll take the third-most relevant one here. So, that's the second most relevant image here. And the third-most relevant one is here. So, the other thing that you can play around with is modifying the query to see if it changes what you're getting. So, here, if I'm interested in cute cats outdoors, I can run that query and see if it changes things. So, notice how now I've got in the first place with the highest matching this other image. We can go ahead and visualize what that looks like. So, I'll just pop that in here. I run that. Now, instead of getting just an image of a cat, I've got the image of a cat, as well as the fact that it's outdoors, so you can naturally communicate with the database and ask it whatever you're interested in, and it'll try to match the vectors accordingly. The next thing we're going to do is perform image search. So, for image search, what we've done is kept a subset of our data set outside of the images repository so that they're not already in the vector database. And you can use those to search. And essentially what you're doing is asking the vector database to extract out images that are the most similar to the test image that you've passed in. So, in this particular example, we're going to go ahead and pick one image from our test repository, and we're going to visualize what that looks like. So, here, this is the image that we want to pass in as a query. So, here, we've got an image of some flowers. And we want to tell the vector database give us images that are the most similar in meaning to this image. In order to do that, we're going to write up a query here. We're going to go ahead and say client dot query get. And we're going to specify the name of our class like so. We're going to specify what we want back in turn. So, we want the text, the name of the image that it thinks is the closest. And we're also going to specify the distance from this image. So, this is the exact same as what we had before. Now, before we perform the with near text search, now, we're going to perform a with near image search because we're passing in an image, and we want to know what other objects are closer to this image. So, now we want to pass in the query image, which is just this image here. So, we have to pass in the path for this image. We'll go ahead and pop that in. We want to only get back the three closest images. So, we'll do a limit, and we'll perform that query. So, now let's run this query and it's giving back three different images. We can go ahead and visualize these similar to how we visualized the text-to-image search queries. So, we can take that piece of code and we can repurpose it over here. The most similar image to that image of flowers is this image. Notice how it even captured the fact that you're looking at pink flowers here. So, you've got pink flowers here. The most similar ones are pink flowers. If we take a look at the second closest image. You can go ahead and take that. We can plot that in. Again, we've got pink flowers, but now this one is focused on flowers that haven't opened up yet, so it's slightly less similar. And then, for the last, semantically similar image that we've gotten here. We can see what that looks like as well. And so, this one is not even flowers, it's just berries. And this could be possibly because we don't have any other images that resemble flowers. So, it picked the two flower images and then whatever was left over we've got as the third most relevant image. And the interesting thing here is that if you're using your own images, you can hold them out of the images repository and keep them separate and use them to search over your other images. So, for example, if you have a pet, you can put in images of your pet into the images repository, and you can hold out a couple of those images and use those images to search and find other images of your pet. That's the really interesting application here where you're performing image to image search now over your own personal files. In this chapter, we introduced where vector representations of data come from and how you can specify and use different ML models with Weaviate to generate and store these vectors. We then use this knowledge to create a text and image search engine. In the next chapter, we'll talk about how you can measure the performance of a vector database using relevant metrics and get more practical experience using one.