03 RAG Multimodal Data Processing

Gemini Multimodal Processing

As a powerful multimodal model, Gemini is capable of processing and generating various forms of data, including text, images, audio, video, PDFs, and code. Its core advantages lie in:

• Native Unified Architecture: All modalities are trained within the same representational space from the pre-training stage, rather than through post-processed add-on modules. This minimizes information loss and preserves temporal and detailed information more comprehensively.
• End-to-End Inference: Utilizes a single set of Transformer parameters to directly process any combination of inputs, such as simultaneously processing CT images and medical records, or converting handwritten recipe videos directly into digital recipes.
• Context Scale: Gemini 3 Pro supports long windows of up to 1 million tokens, capable of processing a 1-hour video or a 700-page PDF in a single pass and generating structured reports.
• Generative Capabilities: Beyond understanding and analysis, it supports multimodal generation, such as text-to-image generation, image editing, and audio streaming.

Gemini API Usage Examples

Step 1: Apply for a Gemini API Key

Visit https://aistudio.google.com/app/api-keys to obtain an API key.

Step 2: Set Environment Variables

Set GEMINI_API_KEY and GOOGLE_API_KEY.

Text Output

Image Understanding

Video understanding

CASE: Disney RAG Assistant

This case study aims to build a 24/7 online AI customer service assistant for Disney, capable of automatically answering frequently asked questions, providing accurate information, and handling multimodal queries.

Challenges

• Diverse Knowledge Sources: The knowledge base includes PDFs (official regulations), Word documents (internal FAQs), web announcements, and event description documents containing images and tables.
• Processing Unstructured Data: Effectively extracting and understanding information from tables and images in PDF and Word documents is key to the success of RAG.
• Effective Organization of Knowledge: How to chunk and index vast amounts of scattered knowledge points to ensure retrieval accuracy.
• Ensuring Answer Validity: How to ensure that the final generated answers are strictly based on retrieved content, avoiding hallucinations in the LLM.

Technology Selection (Solution 2 - Using Multimodal Embedding)

• Embedding Model: Multimodal-Embedding (e.g., Alibaba Cloud Tongyi tongyi-embedding-vision-plus), which uniformly processes text, images, and video.
• Vector Database: FAISS (for high-performance vector retrieval). In production environments, consider Milvus, ChromaDB, Elasticsearch, etc.
• LLM: Qwen-flash (for generating final answers).
• Workflow Orchestration: Directly use low-level APIs (without relying on frameworks like LangChain).

Using Multimodal-Embedding

The Multimodal-Embedding model converts data from different modalities—such as text, images, and video—into floating-point vectors within a unified vector space (for example, tongyi-embedding-vision-plus generates 1152-dimensional vectors). This enables cross-modal retrieval and similarity calculations within the same semantic space.

Multimodal-Embedding Model Comparison

Model Name | Vector Dimension | Text Length Limit | Image Limit | Video Clip Limit | Unit Price (per 1,000 input tokens)

qwen2.5-vl-embedding | 2048, 1024, 768, 512 | 32,000 tokens | ≤5MB, 1 image | ≤50MB | Image/Video: 0.0018 RMB Text: 0.0007 RMB

tongyi-embedding-vision-plus | 1,152 | 1,024 tokens | ≤3MB, ≤8 images | ≤10MB | 0.0005 CNY

tongyi-embedding-vision-flash | 768 | 1,024 tokens | ≤3MB, ≤8 images | ≤10MB | 0.00015 yuan

multimodal-embedding-v1 | 1,024 | 512 tokens | ≤3MB, 1 image | ≤10MB | Image/Video: 0.0009 CNY Text: 0.0007 CNY

Text Embedding Example

Image Embedding Example

Video Embedding Example

Format Processing Related

Processing .docx Files

The parse_docx function uses the python-docx library to read .docx files, iterate through all elements (paragraphs and tables), and extract them into separate content blocks.

• Paragraph processing: Extract plain text content, remove whitespace, and mark it as "type": "text".
• Table Processing: Converts Word tables to Markdown format and marks them as "type": "table".

Processing .pdf files

The parse_pdf function uses the fitz (PyMuPDF) library to open and read PDF documents page by page, breaking them down into plain text and separate image files.

• Text Extraction: Use page.get_text("text") to extract the plain text content from each page, save it as a separate block, and include the page number.
• Image extraction: Detects and extracts all embedded images on the page, saves them to the specified image_dir directory, and records the image paths.

Faiss Index Construction (4-disney_build_index.py)

1. Parse Documents: Use functions such as parse_docx() to process Word documents, extracting text paragraphs and tables.

2. Text Chunking: Use the split_text() function to split text into chunks of a fixed length (e.g., chunk_size=500 characters, with overlap=50 characters of overlap).

3. Multimodal Embedding:

   • Use the tongyi-embedding-vision-plus model to uniformly process text, images, and videos.

   • Text is encoded directly.

   • Images are sent after Base64 encoding.

   • Extract multiple frames from videos and calculate the average vector.

4. Build FAISS Index: Use IndexFlatL2 (L2 distance, Euclidean distance) for precise search indexing.

5. Persistent storage: Save the FAISS index as a .faiss file (disney_index.faiss) and the metadata as a JSON file (disney_metadata.json).

Query Processing (5-disney_query.py)

1. Load index: Load the FAISS index and metadata JSON from files.

2. Query Embedding: Convert the user query into a vector using the Multimodal-Embedding model, placing it in the same space as the vectors in the index.

3. Similarity Retrieval: Retrieve all records, sort them by L2 distance, and convert the results to similarity scores (sim = 1/(1+distance)).

4. Media Intent Detection: Determine whether the user is seeking media content by matching keywords (e.g., "image," "poster," "video," etc.).

5. Result Filtering:

   • Text: Unconditionally select the Top-K (default k=3) most similar results.

   • Images/Videos: Only adopt the closest matching media if media intent is detected and the distance is less than the threshold (e.g., 3.0).

6. LLM Generation: Construct a prompt (combining background knowledge and the user’s query), then call qwen-flash to generate an answer.

Metadata Format Example

Key Parameter Configuration

Unified Multimodal Vector Space and Retrieval Strategy

Text, images, and videos are mapped to a unified vector space via the same embedding model, enabling cross-modal semantic retrieval.

Retrieval Strategy: Unified Indexing + Post-Filtering

1. Unified Retrieval: A single query returns results across all modalities.

2. Intent Detection: Determines whether the user requires media content based on keywords.

3. Category Filtering:

   • Text: Unconditionally select the Top-K results.

   • Images/Videos: Only included if media intent is detected and the distance is less than MEDIA_DISTANCE_THRESHOLD (e.g., 3.0).

4. LLM Generation: Text results are unconditionally fed into the prompt, while media is included as an attachment to ensure answer quality.

Case Demonstration (Sample User Questions)

• User Question 1: "I would like to know the refund process for Disney tickets."

  • Result: Multiple text chunks are matched, containing information such as the refund policy; the LLM generates a detailed explanation of the refund process based on these text chunks.

• User Question 2: "What is the recent Halloween event poster?"

  • Result: Matched a text chunk and an image chunk (2-Halloween.jpeg). Upon detecting the keyword "poster," the LLM combines the text information to describe the Halloween event and includes a link to the image.

• User Question 3: "My car was scratched. Can you see the video?"

  • Result: Matched a text chunk and a video chunk (URL of the car scrape video). Acting as a Disney customer service representative, the LLM explains that it cannot view the surveillance footage but directs the user to contact the customer service center, providing the relevant video link (if applicable).

Chunking Strategy

Knowledge chunking is a core component of RAG systems, directly impacting retrieval quality and answer accuracy.

Common Chunking Strategies

1. Fixed-Length Chunking (with Overlap)

   Code

   Copy

   _Core Approach*_: Split text into fixed-length segments, prioritizing sentence boundaries to avoid breaking sentences. Segments typically overlap by a fixed length. _Pros and Cons*_: Simple to implement, fast processing speed, and uniform length; suitable for scenarios requiring consistent processing lengths and batch processing, such as technical documentation and specifications. Not suitable for semantically sensitive Q&A. _Code Example*_: (Implemented via `1-fixed-length-slicing.py`)

2. Sentence-Boundary Slicing (No Overlap)

   Code

   Copy

   _Core Approach*_: Based on natural language processing, text is split according to semantic units such as sentences and paragraphs. This preserves semantic integrity, avoids breaking sentences in the middle, and ensures that each segment is a complete semantic unit. _Advantages and Disadvantages*_: Good semantic preservation and high retrieval accuracy; however, slice lengths may be uneven. Suitable for natural language text and question-answering systems. Not suitable for documents with many long sentences. _Code Example*_: (Implemented via `2-sentence-boundary-slicing.py`)

3. LLM Semantic Slicing

   Code

   Copy

   _Core Approach*_: Leverages the LLM’s semantic understanding capabilities to achieve precise length control while maintaining semantic integrity. The LLM can intelligently select segmentation points. _Pros and Cons*_: Strong semantic understanding and intelligent segmentation point selection; however, it relies on GPUs and is relatively costly. Suitable for projects with high-quality requirements, complex semantic structures, and sufficient budget support. Not suitable for large-scale document processing or cost-sensitive scenarios. _Code Example*_: ```python prompt = f""" Please slice the following text while preserving semantic integrity, with each slice not exceeding {max_chunk_size} characters. Requirements: 1. Maintain semantic integrity 2. Split at natural breakpoints 3. Return a list of chunks in JSON format, as follows: {{ "chunks": [ "First chunk content", "Second chunk content", ... ] }} Text content: {text} Please return a list of slices in JSON format: """ # Call LLM to generate slices ```

4. Hierarchical Slicing

   Code

   Copy

   _Core Concept*_: Split the document based on its hierarchical structure (headings, sections, paragraphs), treating each structural unit as an independent block. _Pros and Cons*_: Preserves document structure, facilitates understanding of logical relationships, and supports hierarchical queries; however, it relies on document formatting. Suitable for structured documents (manuals, specifications, API documentation). Not suitable for untitled plain text. _Code Example*_: (Implemented via `4-Hierarchical-Slicing.py`)

5. Sliding Window Slicing

   Code

   Copy

   _Core Concept*_: A fixed-size window is slid across the text to generate overlapping slices. The overlapping mechanism ensures contextual continuity, reduces information loss, and improves retrieval recall. _Pros and Cons*_: Maintains contextual continuity and reduces information loss; however, it generates a large amount of overlapping content. Suitable for scenarios requiring overlapping information, processing long documents, or maintaining context. Not suitable for scenarios involving sensitive data storage or where high deduplication is required. _Code Example*_: (Implemented via `5-Sliding Window Slicing.py`)

Summary of Slicing Strategy Comparison

Method | Core Concept | Overlap | Uniform Length | Semantic Integrity | Implementation Cost | Suitable Scenarios | Unsuitable Scenarios

Fixed-Length Slicing | Split by character count, optimized for sentence boundaries | Yes | High | Medium | Low | General-purpose, batch processing, length-constrained tasks | Semantically sensitive Q&A

Sentence-Boundary Slicing | Split by periods, then merge | None | Low | High | Low | Natural language text, Q&A systems | Documents with many long sentences

LLM Semantic Slicing | Sliced after LLM understands content | None | Medium | Highest | High | High-quality requirements, complex semantic structures | Large-scale documents, cost-sensitive

Hierarchical Slicing | Split by headings/chapters | None | Low | High | Medium | Structured documents (manuals, specifications, API documentation) | Untitled plain text

Sliding Window Slicing | Fixed-window sliding with significant overlap | Large | High | Medium | Low | Requires contextual continuity and long-document retrieval | Storage-sensitive, high de-duplication requirements

Scenario Selection Recommendations

• General Scenarios: Fixed-length slices (simple and reliable)
• Technical Documentation: Hierarchical Slicing (Preserves Structure)
• High-quality requirements: LLM semantic slicing (best performance)
• Long Document Retrieval: Sliding Window Slicing (No Information Loss)

Key Points

• Gemini’s Native Multimodal Capabilities: With the ability to uniformly understand, reason about, and generate text, images, videos, and other multimodal data, Gemini serves as the foundation for building advanced RAG applications.
• Core Role of Multimodal Embedding: Maps data from different modalities into a unified vector space, enabling cross-modal semantic retrieval and similarity calculations, thereby simplifying the implementation of multimodal RAG.
• RAG Assistant Workflow: Covers the complete process, including data processing (parsing, segmentation), vectorization (embedding, FAISS indexing), retrieval (unified retrieval, intent detection, filtering), and generation (LLM answer construction).
• The Importance and Diversity of Slicing Strategies: Different knowledge slicing strategies (fixed length, sentence boundaries, LLM semantics, hierarchical, sliding window) each have their own advantages and disadvantages. The most suitable strategy must be selected based on specific scenarios and requirements to improve retrieval quality and answer accuracy.
• Unified Indexing and Post-Filtering Mechanism: By constructing a single multimodal vector index and combining it with intent detection and result filtering, we achieve efficient and precise multimodal information retrieval while avoiding the complexity of maintaining multiple indexing systems.

embed