From the course: Hands-On AI: Building LLM-Powered Apps

Search engine basics - Python Tutorial

From the course: Hands-On AI: Building LLM-Powered Apps

Search engine basics

“

- [Instructor] In the last video, we discussed how we can use retrieval augmented generation or RAG architecture to support a large language model with knowledge for it to provide answers based on sources. In this video, we will discuss the retrieval portion of the architecture. The system we use to store and retrieve information is called a search engine. A search engine is an information retrieval system designed to help us find information stored on a computer system. The most famous search engine out there is Google, which is a system that searches the whole internet. There are also domain-specific search engines that allow user to discover information, such as legal search, financial document search. And one of the latest type of search engine that we could use to retrieve information is called vector databases. And all these search engines are very similar to our physical libraries. There are primarily two processes in a search engine system, indexing and searching. Before we can search anything, we have to parse and index the information. Good parsing and indexing facilitates fast and relevant information retrieval. This is similar to librarians indexing the books to facilitate readers to find books they want to read with ease. Once the information has been indexed, then the user can start the search. Users will use the search engine to find the relevant pieces of information. Search by definition is incomplete and might not always return the perfect result, as our search engines might not have all of the information. There are many, many type of search algorithms, including keyword search, Boolean search, vector search, embedding search, and the combination of the above. So in summary, search engines have two processes. First, we have to parse and index the data into the search engine. Then users will be able to retrieve relevant documents from the index. In the next video, we will dive deeper into the search process.

Contents