In this application, we will build an advanced data model and use it for ingest and various search options.
Learn to ingest data from including JSON and Images, format and transform to optimize hybrid searches. This is done inside the streetcams.py application.
Learn to store data into Milvus, an efficient vector database designed for high-speed similarity searches and AI applications. In this step we are optimizing data model with scalar and multiple vector fields -- one for text and one for the camera image. We do this in the streetcams.py application.
Discover how to use scalars and multiple vectors to query data stored in Milvus and re-rank the final results in this notebook.
Build a quick output for validation and checking in this notebook.
Build a simple Python RAG application (streetcamrag.py) to use Milvus for asking about the current weather via OLLAMA. While outputing to the screen we also send the results to Slack formatted as Markdown.
By the end of this application, you’ll have a comprehensive understanding of using Milvus, data ingest object semi-structured and unstructured data, and using Open Source models to build a robust and efficient data retrieval system. For future enhancements, we can use these results to build prompts for LLM, slack bots, streaming data to Kafka and as a Street Camera search engine.
- https://511ny.org/developers/help/api/get-api-getcameras_key_format
- https://zilliz.com/blog/building-multilingual-rag-milvus-langchain-openai
- https://medium.com/@tspann/utilizing-multiple-vectors-and-advanced-search-data-model-design-for-city-data-705d68d8daf2
- https://www.youtube.com/watch?v=HaRc0rsaMo0
- https://dzone.com/articles/multiple-vectors-and-advanced-search-data-model-design
