From the course: Introduction to AI-Native Vector Databases

ML models and vector embeddings

Now that we know ML models help us generate vectors, let's get practical and see how to do this in a vector database. In this video, we'll look at how to vectorize images and text using Weaviate. Let's say you put all of your images into a vector database and want to search for family photos by passing in the query, family vacation at the beach. How can we accomplish this? In order to vectorize images and text with an ML model, we need one that understands how to translate or embed both of these types of data. CLIP is one such model. It's known as a multi-modal model since it understands multiple types or modalities of data. If we want it to vectorize text data alone, we could use any one of the text effect modules that we Weaviate offers as well. In this example, we'll run Weaviate locally using Docker. So, to get started with this notebook, we're going to do things a little bit differently. We're going to Weaviate run locally with Docker. And in order to do that, we're going to open up our terminal here. And I've already provided you with the Docker file that you need to get this up and running. And I've also provided instructions in the Read me file for the GitHub repo that will help you set up Docker to begin with. So, to launch, we Weaviate locally using Docker, all we need to do here is just say docker compose up. So, it tells us that the containers are up and running. The first time you do this, it might take some time because it's going to download the entire model and make it so that it's readily available and running on your computer, so just be wary of that. So, we've got Weaviate running in the background. So, now we can come back into our development environment and we can start to run the code below. Here we're going to import some of our libraries that we're going to need for this code. So, now, we're going to do is connect to that local instance of Weaviate that we just launched in Docker. In prior versions, what we've been doing is using the embedded setting of Weaviate. What we're going to do here is a little bit different where we've already launched with Weaviate in Docker, we just need to connect to it. So, we can specify that by telling it where this local instance is running. And here, by default, it just always launches on your localhost port 8080. So, here, we can specify local host and specify the port 8080. And this line of code will verify whether or not it's connected properly. So, it's going to check if we Weaviate is ready. We're going to run that. And it gives us a true. So, we're all set to go. The next thing that we're going to do is create our schema in this case. So, before we do that, we want to make sure that there isn't already a schema there. This is just a bit of a sanity check. So, we're going to retrieve all of the classes that already exist in this instance of Weaviate. And we're going to make sure that there isn't a class called clip example, which is what we want to name our class. So, we're going to go ahead, and run that to get rid of any lingering classes that might have been there. There shouldn't be any, but this is a good check. And next, we're going to create our own schema. And we're going to specify which module to use within Weaviate. And because we're building a search engine that understands both images and text, we need to use a multi2vec or a multimedia to vec module. In Weaviate, if you go here into the documentation, you can see what modules are available by going into here modules. And then you can see all of the different modules that are available. So, if you're working with text data, you can use any of these text modules. We're working with multimedia data. So, we're going to be using the multi2vec-clip. Another option that you can play around with is the multi2vec-bind. But for now, multi2vec-clip is the one that we want. So, we're going to specify this in our class instantiation. So, to do that here, we're going to create a class object that we're later going to use to create our schema, as you see in the line below. So, here, we're going to create a class. We're going to name this class clip example. We're going to specify a module config for this, and the module config is just instructions to Weaviate on how to set this up and what particular settings we want for this instance of Weaviate. So, here, we're going to specify multi2vec, and we're going to tell it that we want to clip what I was just showing you the documentation for. And within this, we're going to specify which image fields we want it to vectorize. Because the model understands images and text, we have to let it know which images it needs to vectorize. So, here, we can tell it that the image field they're stored in a property called image. So far, so good. Next, what we're going to do is define the vectorizer that we wanted to use. And so here we specify the vectorizer. And this is also a string here. And again, we're going to say multi2vec-clip so that it knows to use the clip model to vectorize our images and our text. In this particular case we only have images. So, it'll use the clip model to vectorize those. And next up, we have to define the properties that we're passing in. So, we're going to be passing in two properties here. So, we have a list. And then we'll have two dictionaries that we're going to pass in. The first one is going to be called text, and this is going to be the name of the files that we're going to pass in. So, the data type here is just a string. And the next one is a little bit more important. This is the actual image that's going to be vectorized. So, here, We're going to call this image. And this has to be the same name that we passed in over here, so that the model knows where to grab the images before it, vectorize them, and the data type for this particular for images here is of type blob. So, this sets us up. We'll just do a quick review. The name of the class is clip example. We've configured the module to use multi2vec-clip. We've pointed it towards the image property so it knows what to vectorize. And we've also passed in as additional information the name of the file. So we should be able to create this now, go in, and it tells us that the schema class has been created successfully. Now, the next thing that we want to do is go ahead and add all of our images. So, what you can do for this example is, you can have a separate folder that has all the images that you want to store, and we're going to go through one by one and add them to our vector database. So, the first thing that I like to do is print out and see what images I'm adding as I'm adding them. So, I'm going to go in here, and I'm going to just print out the image name as I loop over it. And in Weaviate, in order to add an image, the first thing that we have to do is encode it into base64. And there is utility functions that are provided to us within Weaviate to help us do this. So, within Weaviate, we can go into the util functions, and we can go into an image encoder. And we want to go in and encode the image, and we specify the image by the file path. So, we can say images. The name of the image here. So, now that we have the encoded image, we can pass it into Weaviate, and Weaviate will be able to vectorize it using the clip model. So, the way that we do that is by specifying the data properties. And this is, again, a dictionary Python. We're going to pass in the image and we're going to tell it that the encoded image here, the image and the name of the image is what we have here that we're looping over. Now that we have this data object, we're going to go ahead and create it. Data object. And we're going to create this data object. I'm going to pass in the data properties. And we're going to tell it which class this belongs to. So, that's clip example that we created earlier. And we're also going to give us a final print statement that says all images added so that we know that it's done executing. So, we're going to run this code now. And as it goes through our folder of images, it'll let us know as it's successfully added all of these images. So, one by one, it's adding all these images, and you can read the names to understand what they're about. That'll be a little bit more important later on. So, as this is happening, it's taking the image vectorizing it using the clip model, storing the image and the vector representation of that image into Weaviate that you'll be able to search over later on. And for this example, we've got some images. So, we'll be able to search over these concepts shortly. All right. So, we've got all the images added successfully. Next thing we're going to do is perform a text-to-image search. We're going to pass in a concept that we're interested in as text, and we're going to see what images are closest to that text string. So, here, we need to create a query. We're going to create a variable to store the response. And we're going to go ahead and format our query so that it looks nice. We're going to say, client, I have a query for you. Go ahead and get me from the clip example some properties, or we want it to go ahead and extract the title of the image that it thinks is the closest so that we can visualize it later on. And also, I want it to tell me how far away the object or the image is from the query itself. So, those are the two things that we get back. Because the query is a text query, we're going to be using the with near text query here. So, we're going to scroll down all the way here. With near text. And we're going to specify the concept that we're interested in by using the concepts keyword. Concept. Here, we're interested in open sea beach. We're also going to limit it so that we only get back five results. So, we're going to get back five of the most relevant images here, and we're going to tell it to perform this operation. So, that looks good to me. We should be able to take this response and print it out to see which images it thinks are the most related to the concept of open sea beach. We run that query, and we get our five images, and we can see what these images look like. So, the closest image to the concept of open sea beach, it thinks is this Point Reyes, California beach. We're going to go in here and we're going to visualize that image. To do this, you can just pass in the path to the image and the name of the image and we can visualize it. This is the closest in concept open sea beach that it finds. We can also look at the second closest here. This one we can copy in the name of the image, and we can pass it in images and we can see what that second image looks like. So, not only did it retrieve the beach images, it also got the one that had open see concept as well. So, this is pretty awesome. Now, we're going to look for another concept. So, let's go ahead and search for another interesting thing. So, we'll just take our same query that we had here. So, we'll take this same query and we're going to change the concept up a bit just to give you an intuitive understanding of what it's doing. Rather than looking for open sea beach, we'll go ahead and we'll look for greenery and trees, and we'll see what comes back. So, we run this query, and now it gives us some different images. So, it gave us this image of the forest. We can visualize that. Gave us this spiral staircase. So, this one's interesting because what does a spiral staircase have to do with greenery and trees? So, let's examine that in a bit. Let's go down again. We'll visualize what these images look like here. So, that's clearly related to the concept. The second one is a little bit more interesting. So, let's take this image and visualize it. So, we'll go here. Images pass us in. Notice how it is a spiral staircase, but you've got greenery in there. And this is the greenery that matched with our input query. It is pretty intricate in the details that it looks at when extracting out relevant results. The last thing that we're going to do here is perform an image to image search. And so what this means is that rather than passing in a concept like so, greenery and trees, we're actually going to be passing in an image and asking Weaviate to give us back the most similar images. So, we're going to go here and we can choose any image. But for this particular example, let me choose an image of the beach. And this is an image that is not stored in the database. So, for example, here if you visualize this this is in a separate folder that you can look at that's called test images. If you visualize this, this image itself is not in the database right now. We wanted to see which images actually, in the database, are close to this image. So, we'll go in, and we'll take this image and we'll specify that as the input to our search query. We'll place that here, and then we'll take our query, and now we're going to do something slightly different. So, I'm going to start off with the input query that we had here, and I'm going to modify it to show you how you can perform image to image search. So, we're going to add that there. Now, the main line that we're interested in here is this one. We no longer have a text query that we want to search over, we have an image query. So, now I need to do a with image search. So, as I scroll down here, I have a with near image, and I have to specify which image I'm interested in. So, the image that I wanted to perform semantic search with respect to is this particular image that I've just visualized up there. So, I'm going to give it the path to that particular image. And we should be good to go. If I run this query, it'll go ahead and vectorize this image, and then it'll compare the vectors of this image to all of the other images that it's got in there. So, it determined that this was the closest. So, we can go ahead and visualize this. Again, remember, this is the query image. Let's look at the returned most relevant images that it thinks are close. So, here, we'll go ahead and we'll visualize this. That's pretty close compared to that image. The most the two most similar images that we have. And then we'll even look at the second most similar image here. So, we'll take the second one, and we'll go ahead and visualize that, as well. All right. So, that's the second most similar image in there. So, let's do one last image to image query. What we're going to do here is look for images that are the most similar to this cat image. So, we're going to go ahead and take our query. This is going to be exactly the same. But now we're going to change the query vector. Here, we'll tell it to query for images that are the closest to this cat image here. And we'll see what comes back. So, here, we get back this image of a cat. And if we grab that, we can visualize what that looks like. If I scroll down, this is stored in our images folder. So, this image is what it thinks is the closest to our query image over here, and it worked pretty well. We learned how to specify, setup and use ML models that understand both images and text, the vectorized data of each type, and we Weaviate. In the next video, we'll introduce our challenge and get some hands-on experience, which will allow you to search over your own images and text with Weaviate.

Contents