Agentic Vision in Gemini 3: A Hands-On Tutorial

You send an image to a vision API asking it to read a small price label. The response comes back confident and wrong. Same story with counting objects in a cluttered photo: the model glances at the full image once and guesses a number that's way off.

Gemini 3 Flash handles this differently. With code execution turned on, it can write and run Python mid-response to crop, zoom, and annotate images before committing to an answer. Google calls this agentic vision and reports a 5-10% quality lift on vision benchmarks.

This tutorial walks through four examples that get progressively harder: reading fine print from a grocery shelf, pulling drink names off a dimly lit bar menu, counting scattered coins, and extracting table data into a chart.

What Is Agentic Vision? The "Think, Act, Observe" Loop

The idea is straightforward. With code execution active, Gemini 3 can stop mid-response, write and run Python in a sandbox, look at what the code produced, and decide what to do next. 

For vision tasks, this usually means the model is cutting out regions of an image, enlarging them, drawing bounding boxes, or adjusting contrast to pull out details that aren't visible at the original resolution.

This plays out as a three-phase loop.

During Think, the model looks at what it has (the original image, or a version it already cropped or annotated) and plans its next move. It might decide a price label is too small to read and needs to be cropped and enlarged.

Act turns that plan into Python. The sandbox has 43 pre-installed libraries (PIL, OpenCV, matplotlib, and more), so the model can write whatever image processing code it needs.

After the code runs, Observe closes the loop. Output (new images, printed text, computed values) feeds back into context. From there, the model either answers or goes around again.

A single API call can cycle through this loop multiple times. In the examples ahead, you'll see it crop seven shelf regions in one request. In another, it extracts table data into a pandas DataFrame and generates a bar chart, all within the same call. 

If the agentic loop concept is new to you, DataCamp's AI Agent Fundamentals track covers it in depth.

Limitations and boundaries

• Code execution and custom function calling can't be used in the same request.
• Google's docs warn that code execution can regress performance on non-visual tasks. Only turn it on when the task benefits from image manipulation.
• Pixel-level segmentation (masks, contours) isn't part of agentic vision and stays on Gemini 2.5.
• The model sometimes skips code execution if a single look is enough. By design, not a bug.

Setting Up Your Environment For Agentic Vision

Grab an API key from ai.google.dev and store it as GOOGLE_API_KEY in your environment or a .env file. Then install the SDK and image libraries:

pip install google-genai python-dotenv Pillow matplotlib opencv-python

Start by creating the client:

from google import genai
from google.genai import types
from dotenv import load_dotenv
 
load_dotenv()
 
client = genai.Client(
    http_options=types.HttpOptions(timeout=600_000)
)
MODEL = "gemini-3-flash-preview"

That timeout=600_000 matters. Agentic vision calls involve multiple rounds of code generation and execution inside the sandbox, and a single request can take around four minutes. 

The default httpx timeout will kill the connection well before that, leaving you with a silent ReadTimeout and no clue what happened.

Next, set up the agentic config:

agentic_config = types.GenerateContentConfig(
    tools=[types.Tool(code_execution=types.ToolCodeExecution())],
	thinking_config=types.ThinkingConfig(
    	thinking_level="HIGH",
    	include_thoughts=True
	),
    media_resolution=types.MediaResolution.MEDIA_RESOLUTION_HIGH,
)

tools is the on-switch for code execution. It gives the model a sandboxed Python environment with 43 libraries (numpy, pandas, PIL, matplotlib, OpenCV, and more). Each code block gets 30 seconds to run, and the sandbox retries up to 5 times on errors.

thinking_level="HIGH" gives the model more room to plan before writing code. For multi-step vision work, this makes a noticeable difference. Setting include_thoughts=True exposes that reasoning in the response parts, useful for debugging.

media_resolution sets how many tokens the input image gets before the loop even starts. MEDIA_RESOLUTION_HIGH is a good default. You can push it to MEDIA_RESOLUTION_ULTRA_HIGH for very dense images or drop to MEDIA_RESOLUTION_LOW when speed matters more than detail.

This same setup works on Vertex AI with minimal changes. For a broader walkthrough of the Gemini 3 API beyond vision, see the companion tutorial.

Agentic Vision in Gemini 3: Reading Fine Detail with Crop and Zoom

The image below is a German grocery store shelf with three rows of products. Price labels line each shelf edge, and a "Liebe Kunden" (Dear Customers) notice sits near the bottom. The labels are readable if you zoom in yourself, but small enough that a model looking at the full image has to decide what it can and can't make out.

Parsing the multi-part response

Agentic responses aren't single text blocks. They arrive as a list of parts: thinking, code the model wrote, execution results, generated images, and the final answer. This helper walks through them:

def print_response(response, show_thoughts=False):
	for part in response.candidates[0].content.parts:
    	if part.thought and show_thoughts:
        	print("[THINK]", part.text[:500])
    	elif part.executable_code:
        	print("[CODE]")
        	print(part.executable_code.code)
    	elif part.code_execution_result:
        	print(f"[RESULT] ({part.code_execution_result.outcome})")
            print(part.code_execution_result.output)
    	elif part.inline_data:
        	print(f"[IMAGE] {part.inline_data.mime_type}, "
                  f"{len(part.inline_data.data):,} bytes")
    	elif part.text:
        	print("[ANSWER]", part.text)

Every example in this tutorial uses the same function, so define it once and reuse it.

Now load the shelf image and write a prompt that asks the model to read the sign and list every product with its price.

from pathlib import Path
 
image_path = Path("images/grocery_shelf.jpg")
image_part = types.Part.from_bytes(
	data=image_path.read_bytes(), mime_type="image/jpeg"
)
 
prompt = (
	"Read the 'Liebe Kunden' sign at the bottom of the shelf. "
	"What does it say? Also list all visible product names "
	"and their prices from the shelf labels."
)

Run it first without code execution:

standard_config = types.GenerateContentConfig(
    media_resolution=types.MediaResolution.MEDIA_RESOLUTION_HIGH,
)
 
standard_response = client.models.generate_content(
	model=MODEL,
	contents=[image_part, prompt],
	config=standard_config,
)
print_response(standard_response)

Here's a trimmed version of the output:

[ANSWER] Based on the image provided, here is the information requested:
 
### The 'Liebe Kunden' Sign
> "Dear Customers, unfortunately, we must inform you that due to
> increased demand, we cannot currently offer all products..."
 
### Product Names and Prices (from shelf labels)
 
**Top Row Labels:**
*   Uncle Ben's Express Reis Mexi-Scharf 250g: 1,69
*   Uncle Ben's Express Reis Griechisch 250g: 1,69
*   Uncle Ben's Express Reis Curry 250g: 1,69
*   Channa Masala 300g (partially obscured): 2,49
*   (Label obscured): 2,49
*   (Label obscured): 1,29
*   (Label obscured): 0,99
...
 
**Second Row Labels:**
*   Uncle Ben's Express Reis Chili con Carne 250g: 2,49
*   Uncle Ben's Spitzen-Langkorn Reis 10 Min 500g: 1,79
*   (Label obscured): 2,99
*   (Label obscured): 2,79
...

The sign translation is correct, and the model reads the Uncle Ben's labels it can recognize. But about a third of the entries are "(Label obscured)," and brands like Oryza, Reis-fit, and Reiskugeln don't appear at all.

Now the same prompt with the agentic config:

agentic_response = client.models.generate_content(
	model=MODEL,
	contents=[image_part, prompt],
	config=agentic_config,
)
print_response(agentic_response, show_thoughts=True)

The output structure is different. Instead of a single [ANSWER], it walks through the full Think/Act/Observe loop:

[THINK] Okay, so the primary goal here is twofold: First, I need to
decipher that "Liebe Kunden" sign down there. Then, I need to build a
comprehensive list of what's on this shelf...
 
[CODE]
import PIL.Image
 
img = PIL.Image.open('input_file_0.jpeg')
width, height = img.size
 
crops = [
	{'box_2d': [630, 447, 887, 559], 'label': 'liebe_kunden_sign'},
	{'box_2d': [80, 40, 240, 560], 'label': 'shelf1_left'},
	{'box_2d': [190, 550, 310, 980], 'label': 'shelf1_right'},
	{'box_2d': [400, 0, 480, 450], 'label': 'shelf2_left'},
	{'box_2d': [410, 450, 480, 1000], 'label': 'shelf2_right'},
	{'box_2d': [630, 130, 760, 460], 'label': 'shelf3_left'},
	{'box_2d': [550, 600, 660, 990], 'label': 'shelf3_right'},
]
# ... crops and saves each region
 
[IMAGE] image/png, 135,953 bytes
[IMAGE] image/png, 90,395 bytes
[IMAGE] image/png, 125,816 bytes
[IMAGE] image/png, 102,288 bytes
[IMAGE] image/png, 173,606 bytes
[IMAGE] image/png, 283,263 bytes
[IMAGE] image/png, 100,522 bytes

The seven crops the model generated:

After observing those crops, the model produces a complete answer:

[ANSWER]
**Top Shelf (Left to Right):**
*   Uncle Ben's Express Reis Mexi. Scharf 250g: 1,69 €
*   Channa Masala 285g: 2,49 €
*   Madras Curry 285g: 2,49 €
*   Steam Basmati Reis Kochbeutel 250g: 1,99 €
*   Reiskugeln Pur 4x2 St: 1,99 €
...
 
**Middle Shelf (Left to Right):**
*   Uncle Ben's Express Reis Chili con Carne 250g: 2,49 €
*   Oryza Langkorn & Wildreis 500g Kochbeutel: 2,99 €
*   Oryza Basmati & Wildreis 500g Kochbeutel: 2,79 €
*   Oryza Jasmin Reis 500g Kochbeutel: 2,79 €
...
 
**Bottom Shelf (Left to Right):**
*   Uncle Ben's Natur Reis 10-Min. Kochbeutel 1kg: 3,69 €
*   Reis-fit 10 Minuten Natur-Reis 500g: 1,89 €
*   Reis-fit Langkorn & Wildreis 500g: 1,89 €
...

The full untruncated output for both calls is available in this gist.

The agentic response reads 30+ products across all three shelves. The Oryza, Reis-fit, Madras Curry, Reiskugeln, and Steam Basmati items that standard vision labeled "(obscured)" all show up with names and prices.

The difference isn't about hallucination vs. accuracy; standard vision is honest about what it can't read. Agentic vision just refuses to accept that limit and zooms in until it can.

The prompt plays a role here. Asking the model to "list all visible product names and their prices" sets a completeness bar that pushes it toward cropping. 

A vaguer prompt like "describe this image" would likely produce a single-pass answer even with code execution on, because the model sees no reason to go further.