Add Multimodal Semantic Search quickstart using gemini-embedding-2

[NaveenKumarG-dev]

Description

This PR contributes a new notebook example demonstrating built-in multimodal semantic search and cross-modal retrieval using the gemini-embedding-2 model and the official google-genai Python SDK.

The notebook serves as a task-oriented recipe to help developers understand the direct SDK usage of multimodal embeddings.

Overview of Content

• Setup & Authentication: Initializing the new google-genai client and configuring it for gemini-embedding-2.
• Dataset Preparation: Constructing a mixed local dataset of images and text files with intentionally misleading filenames to showcase the difference between keyword and semantic matching.
• Embedding Generation: Building a helper to format document title/content inputs with appropriate task prefixes (task: search result | query: for queries; and title: | text: for documents) before calling client.models.embed_content.
• Search & Similarity Function: Defining a lightweight cosine similarity comparison to retrieve the closest files matching a natural language text query.
• Visual Demonstration: Displaying retrieved results in an inline HTML grid alongside a side-by-side keyword vs. semantic search comparison table.

Licensing & Compliance

• All external media assets (Wikimedia Commons) are CC0 / public domain.
• The notebook copyright header and Apache 2.0 license block are included.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[gemini-code-assist]

Note

Gemini is unable to generate a summary for this pull request due to the file types involved not being currently supported.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

markmcd commented on 2026-07-21T07:09:00Z
----------------------------------------------------------------

Thanks for the PR, this is a great guide!

Since this is in examples/, feel free to add a byline with a link to a social profile or similar. You can see an example on the Sketch2Paint notebook, but it can be simple too if you prefer.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

markmcd commented on 2026-07-21T07:09:01Z
----------------------------------------------------------------

Can you rename the notebook such that it omits the model name from the filename?

This way we can update the notebook in future when new models are released.

The title is OK to leave, just rename the file.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

markmcd commented on 2026-07-21T07:09:02Z
----------------------------------------------------------------

Note: gemini-embedding-2 is a preview model

I don't think this is true. If you saw it somewhere, please lmk so we can update the docs.

https://ai.google.dev/gemini-api/docs/models/gemini

This link can change to https://ai.google.dev/gemini-api/docs/embeddings

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

markmcd commented on 2026-07-21T07:09:03Z
----------------------------------------------------------------

public domain (CC0 / public domain).

Can you provide a source for this claim? It looks to me that they are licensed under the GNU FDL or CC-BY-SA (I only checked the first image)

I think CC is fine though, we just need to correctly document the license and provide links back to each image's wiki/File page (e.g. https://commons.wikimedia.org/wiki/File:YellowLabradorLooking_new.jpg)

And they need to be checked individually as licenses differ per-file.

Oh and can you confirm that the text content is original? I can't see it online so I assume it's yours but please confirm.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

markmcd commented on 2026-07-21T07:09:03Z
----------------------------------------------------------------

User-Agent

Thanks for being a good citizen

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

markmcd commented on 2026-07-21T07:09:04Z
----------------------------------------------------------------

def generate_embedding(
  content: str | bytes,
  mime_type: str | None = None,
  is_query: bool = False,
  title: str | None = None,
) -> list[float]:

Re: list[float], when dealing with a small sample size, using Python data types to manage embedding vectors is OK, but almost any real scenario will want to use numpy or something else made for vector calculations.

I don't think you have to change it for this, but if you choose not to, please make a note along the lines of "For real applications, consider using a library like numpy to handle vector arithmetic."

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

markmcd commented on 2026-07-21T07:09:05Z
----------------------------------------------------------------

def cosine_similarity(vec1: list[float], vec2: list[float]) -> float:
    """Returns the cosine similarity between two embedding vectors."""
    a = np.array(vec1)
    b = np.array(vec2)
    norm_a = np.linalg.norm(a)
    norm_b = np.linalg.norm(b)
    if norm_a == 0 or norm_b == 0:
        return 0.0
    return float(np.dot(a, b) / (norm_a * norm_b))

Couple things here:

1) the output from the GE2 embedding model is always a unit vector, so this function can be replaced with float(np.dot(vec1, vec2))

2) if you want to work with np types throughout, this can be done in batch (with CPU / GPU acceleration, etc). so that scored = [ .. ] (which has to calculate each embedding individually over the comprehension), can be results = embeddings @ queries. And you could use a pandas dataframe to store your other metadata too.

You should be able to find some similar examples in the cookbook that use this approach, or see this older notebook I wrote. And btw you don't have to do this second part - I think your example is illustrative enough with Python primitives.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

markmcd commented on 2026-07-21T07:09:06Z
----------------------------------------------------------------

Notice that the results are ranked by semantic content, not by filename.

It's both really. You can see the 3rd result for "a dog running in a field" is the cat picture with a document title "dog_picture.jpg". It's worth noting that the titles are still factored in, and they are given some weight, though it is not exclusive.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

markmcd commented on 2026-07-21T07:09:06Z
----------------------------------------------------------------

When you set the background colour, you need to set the foreground (text colour) too. In dark mode, the result is light text on the light grey background.

(this is a nice summary display btw)

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

markmcd commented on 2026-07-21T07:09:07Z
----------------------------------------------------------------

Some future work that could be good to add to or supplement this one day (not part of this PR, just if you want to riff on it further).

• Embedding multimodal documents (docs made of text and images, beyond just the textual title)
• Querying by image
• Optimising with pandas/numpy
• Doing a similar exercise via the Agents API

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

markmcd commented on 2026-07-21T07:09:08Z
----------------------------------------------------------------

I think drop the two points on "Continue your discovery..." as they are a bit simple to "continue" from here.

[NaveenKumarG-dev]

Hi @markmcd,

Thanks again for the detailed review and the helpful suggestions!

I've addressed all of the review comments and pushed the requested changes, including:

• Added a byline.
• Renamed the notebook to remove the model name from the filename.
• Corrected my mistake about gemini-embedding-2 being a preview model.
• Updated the licensing section with the correct licenses and source links for each image, and confirmed that the text content is original.
• Added a note recommending NumPy for production vector operations.
• Simplified the cosine similarity implementation for unit vectors.
• Updated the explanation to clarify that both semantic content and titles influence the ranking.
• Fixed the dark mode text color issue.
• Removed the "Continue your discovery..." points as suggested.

Thanks as well for the future work ideas—they're great suggestions and I'll definitely keep them in mind for follow-up examples.

I've re-requested your review. When you have a chance, I'd appreciate another look. Thanks!

[kkorpal]

Hi @NaveenKumarG-dev Thanks for the patience ,I found the CI failure cause.
The notebook format check is failing on: examples/Multimodal_Semantic_Search.ipynb

Please run:

python3 -m pip install -U git+https://github.com/tensorflow/docs
python3 -m tensorflow_docs.tools.nbfmt examples/Multimodal_Semantic_Search.ipynb
git add examples/Multimodal_Semantic_Search.ipynb
git commit -m "Format notebook with nbfmt"
git push

After pushing, the Notebooks / Notebook format job should pass.

[kkorpal]

@NaveenKumarG-dev , Also Please fix examples/Multimodal_Semantic_Search.ipynb:

Lint failure: gemini_cookbook::button_colab
Cause: Colab button URL is missing/malformed.
Use this exact Colab URL in the top button table:

https://colab.research.google.com/github/google-gemini/cookbook/blob/main/examples/Multimodal_Semantic_Search.ipynb

Then commit and rerun the workflow for PR #1286.