Google Gemini - Portkey Docs

Portkey provides a robust and secure gateway to facilitate the integration of various Large Language Models (LLMs) into your applications, including Google Gemini APIs. With Portkey, you can take advantage of features like fast AI gateway access, observability, prompt management, and more, all while ensuring the secure management of your LLM API keys through Model Catalog.

Quick Start

from portkey_ai import Portkey

# 1. Install: pip install portkey-ai
# 2. Add @google provider in Model Catalog
# 3. Use it:

portkey = Portkey(api_key="PORTKEY_API_KEY")

response = portkey.chat.completions.create(
    model="@google/gemini-1.5-pro",
    messages=[{"role": "user", "content": "Say this is a test"}]
)

print(response.choices[0].message.content)

import Portkey from 'portkey-ai'

// 1. Install: npm install portkey-ai
// 2. Add @google provider in Model Catalog
// 3. Use it:

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY"
})

const response = await portkey.chat.completions.create({
    model: "@google/gemini-1.5-pro",
    messages: [{ role: "user", content: "Say this is a test" }]
})

console.log(response.choices[0].message.content)

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL

# 1. Install: pip install openai portkey-ai
# 2. Add @google provider in Model Catalog
# 3. Use it:

client = OpenAI(
    api_key="PORTKEY_API_KEY",  # Portkey API key
    base_url=PORTKEY_GATEWAY_URL
)

response = client.chat.completions.create(
    model="@google/gemini-1.5-pro",
    messages=[{"role": "user", "content": "Say this is a test"}]
)

print(response.choices[0].message.content)

import OpenAI from "openai"
import { PORTKEY_GATEWAY_URL } from "portkey-ai"

// 1. Install: npm install openai portkey-ai
// 2. Add @google provider in Model Catalog
// 3. Use it:

const client = new OpenAI({
    apiKey: "PORTKEY_API_KEY",  // Portkey API key
    baseURL: PORTKEY_GATEWAY_URL
})

const response = await client.chat.completions.create({
    model: "@google/gemini-1.5-pro",
    messages: [{ role: "user", content: "Say this is a test" }]
})

console.log(response.choices[0].message.content)

# 1. Add @google provider in Model Catalog
# 2. Use it:

curl https://api.portkey.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY" \
  -d '{
    "model": "@google/gemini-1.5-pro",
    "messages": [{"role": "user", "content": "Say this is a test"}]
  }'

Add Provider in Model Catalog

Navigate to Model Catalog

Go to Model Catalog → Add Provider in your Portkey dashboard.

Select Google Gemini

Find and select Google from the provider list.

Enter API Key

Get your API key from Google AI Studio and enter it in Model Catalog.

Save and Use

Save your configuration. Your provider slug will be @google (or a custom name you specify).

Portkey supports the system_instructions parameter for Google Gemini 1.5 - allowing you to control the behavior and output of your Gemini-powered applications with ease.Simply include your Gemini system prompt as part of the {"role":"system"} message within the messages array of your request body. Portkey Gateway will automatically transform your message to ensure seamless compatibility with the Google Gemini API.

Gemini Capabilities

Function Calling

Portkey supports function calling mode on Google’s Gemini Models. Explore this cookbook for a deep dive and examples: Function Calling

Advanced Multimodal Capabilities with Gemini

Gemini models are inherently multimodal, capable of processing and understanding content from a wide array of file types. Portkey streamlines the integration of these powerful features by providing a unified, OpenAI-compatible API.

The Portkey Advantage: A Unified Format for All MediaTo simplify development, Portkey uses a consistent format for all multimodal requests. Whether you’re sending an image, audio, video, or document, you will use an object with type: 'image_url' within the user message’s content array.Portkey’s AI Gateway intelligently interprets your request—based on the URL or data URI you provide—and translates it into the precise format required by the Google Gemini API. This means you only need to learn one structure for all your media processing needs.

Image Processing

Gemini can analyze images to describe their content, answer visual questions, or identify objects.

Gemini Image Understanding Docs

Method 1: Sending an Image via Google Files URL Use the Google Files API to upload your image and get a URL. This is the recommended approach for larger files or when you need persistent storage.

To upload files and get Google Files URLs, use the Files API. The URL format will be similar to: https://generativelanguage.googleapis.com/v1beta/files/[FILE_ID]

const chatCompletion = await portkey.chat.completions.create({
    model: 'gemini-1.5-pro',
    messages: [{
        role: 'user',
        content: [
            {
                type: 'image_url',
                image_url: {
                    url: 'https://generativelanguage.googleapis.com/v1beta/files/your-file-id'
                }
            },
            {
                type: 'text',
                text: 'Describe this image in detail.'
            }
        ]
    }],
});
console.log(chatCompletion.choices[0].message.content);

completion = portkey.chat.completions.create(
    model='gemini-1.5-pro',
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://generativelanguage.googleapis.com/v1beta/files/your-file-id"
                }
            },
            {
                "type": "text",
                "text": "Describe this image in detail."
            }
        ]
    }],
)
print(completion.choices[0].message.content)

curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
    "model": "gemini-1.5-pro",
    "messages": [{
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Describe this image in detail."
            },
            {
                "type": "image_url",
                "image_url": { "url": "https://generativelanguage.googleapis.com/v1beta/files/your-file-id" }
            }
        ]
    }]
}'

Method 2: Sending a Local Image as Base64 Data Use this method for local image files. The file is encoded into a Base64 string and sent as a data URI. This is ideal for smaller files when you don’t want to use the Files API. The data URI format is: data:<MIME_TYPE>;base64,<YOUR_BASE64_DATA>

import fs from 'fs';

const imageBytes = fs.readFileSync('local-image.png');
const base64Image = imageBytes.toString('base64');
const imageUri = `data:image/png;base64,${base64Image}`;

const chatCompletion = await portkey.chat.completions.create({
    model: 'gemini-1.5-pro',
    messages: [{
        role: 'user',
        content: [
            { type: 'image_url', image_url: { url: imageUri }},
            { type: 'text', text: 'What is in this picture?' }
        ]
    }],
});
console.log(chatCompletion.choices[0].message.content);

import base64

with open("local-image.png", "rb") as image_file:
    image_bytes = image_file.read()

base64_image = base64.b64encode(image_bytes).decode('utf-8')
image_uri = f"data:image/png;base64,{base64_image}"

completion = portkey.chat.completions.create(
    model='gemini-1.5-pro',
    messages=[{
        "role": "user",
        "content": [
            { "type": "image_url", "image_url": { "url": image_uri }},
            { "type": "text", "text": "What is in this picture?" }
        ]
    }],
)
print(completion.choices[0].message.content)

# First, encode your image file to base64
# For example: base64 -i local-image.png -o image.b64
# Then use the encoded content in the request

curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
    "model": "gemini-1.5-pro",
    "messages": [{
        "role": "user",
        "content": [
            {"type": "text", "text": "What is in this picture?"},
            {"type": "image_url", "image_url": {"url": "data:image/png;base64,YOUR_BASE64_IMAGE_DATA"}}
        ]
    }]
}'

Supported Image MIME types: image/png, image/jpeg, image/webp, image/heic, image/heif

Audio Processing

Gemini can transcribe speech, summarize audio content, or answer questions about sounds.

Gemini Audio Understanding Docs

Method 1: Sending Audio via Google Files URL Upload your audio file using the Files API to get a Google Files URL.

const chatCompletion = await portkey.chat.completions.create({
    model: 'gemini-1.5-pro',
    messages: [{
        role: 'user',
        content: [
            {
                type: 'image_url',
                image_url: {
                    url: 'https://generativelanguage.googleapis.com/v1beta/files/your-audio-file-id'
                }
            },
            { type: 'text', text: 'Please transcribe the speech in this audio.' }
        ]
    }],
});
console.log(chatCompletion.choices[0].message.content);

completion = portkey.chat.completions.create(
    model='gemini-1.5-pro',
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://generativelanguage.googleapis.com/v1beta/files/your-audio-file-id"
                }
            },
            { "type": "text", "text": "Please transcribe the speech in this audio." }
        ]
    }],
)
print(completion.choices[0].message.content)

curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
    "model": "gemini-1.5-pro",
    "messages": [{
        "role": "user",
        "content": [
            {"type": "text", "text": "Please transcribe the speech in this audio."},
            {"type": "image_url", "image_url": {"url": "https://generativelanguage.googleapis.com/v1beta/files/your-audio-file-id"}}
        ]
    }]
}'

Method 2: Sending Local Audio as Base64 Data This is the standard way to process local audio files directly through the API.

import fs from 'fs';

const audioBytes = fs.readFileSync('audio-example.mp3');
const base64Audio = audioBytes.toString('base64');
const audioUri = `data:audio/mp3;base64,${base64Audio}`;

const chatCompletion = await portkey.chat.completions.create({
    model: 'gemini-1.5-pro',
    messages: [{
        role: 'user',
        content: [
            { type: 'image_url', image_url: { url: audioUri }},
            { type: 'text', text: 'Describe this audio' }
        ]
    }],
});
console.log(chatCompletion.choices[0].message.content);

import base64

with open('audio-example.mp3', 'rb') as audio_file:
    audio_bytes = audio_file.read()

base64_audio = base64.b64encode(audio_bytes).decode('utf-8')
audio_uri = f"data:audio/mp3;base64,{base64_audio}"

completion = portkey.chat.completions.create(
    model='gemini-1.5-pro',
    messages=[{
        "role": "user",
        "content": [
            { "type": "image_url", "image_url": { "url": audio_uri }},
            { "type": "text", "text": "Describe this audio" }
        ]
    }],
)
print(completion.choices[0].message.content)

# First, encode your audio file to base64
# For example: base64 -i audio-example.mp3 -o audio.b64
# Then use the encoded content in the request

curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
    "model": "gemini-1.5-pro",
    "messages": [{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this audio"},
            {"type": "image_url", "image_url": {"url": "data:audio/mp3;base64,YOUR_BASE64_AUDIO_DATA"}}
        ]
    }]
}'

Supported Audio MIME types: audio/wav, audio/mp3, audio/aiff, audio/aac, audio/ogg, audio/flac, audio/pcm, audio/m4a, audio/mpeg, audio/mpga, audio/mp4, audio/webm

Video Processing

Gemini can summarize videos, answer questions about specific events, and describe scenes.

Gemini Video Understanding Docs

Method 1: Sending a Video via YouTube URL YouTube is the only supported public URL source for videos. Simply provide the YouTube video URL.

const chatCompletion = await portkey.chat.completions.create({
    model: 'gemini-1.5-pro',
    messages: [{
        role: 'user',
        content: [
            {
                type: 'text',
                text: 'Describe this video in 3 sentences.'
            },
            {
                type: 'image_url',
                image_url: {
                    url: 'https://www.youtube.com/watch?v=9hE5-98ZeCg'
                }
            }
        ]
    }],
});
console.log(chatCompletion.choices[0].message.content);

completion = portkey.chat.completions.create(
    model='gemini-1.5-pro',
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Describe this video in 3 sentences."
            },
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://www.youtube.com/watch?v=9hE5-98ZeCg"
                }
            }
        ]
    }],
)
print(completion.choices[0].message.content)

curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
    "model": "gemini-1.5-pro",
    "messages": [{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this video in 3 sentences."},
            {"type": "image_url", "image_url": {"url": "https://www.youtube.com/watch?v=9hE5-98ZeCg"}}
        ]
    }]
}'

Method 2: Sending Local Video as Base64 Data For smaller video files, you can encode them as base64. Note that this method has size limitations.

import fs from 'fs';

const videoBytes = fs.readFileSync('video-example.mp4');
const base64Video = videoBytes.toString('base64');
const videoUri = `data:video/mp4;base64,${base64Video}`;

const chatCompletion = await portkey.chat.completions.create({
    model: 'gemini-1.5-pro',
    messages: [{
        role: 'user',
        content: [
            { type: 'image_url', image_url: { url: videoUri }},
            { type: 'text', text: 'Describe this video' }
        ]
    }],
});
console.log(chatCompletion.choices[0].message.content);

import base64

with open('video-example.mp4', 'rb') as video_file:
    video_bytes = video_file.read()

base64_video = base64.b64encode(video_bytes).decode('utf-8')
video_uri = f"data:video/mp4;base64,{base64_video}"

completion = portkey.chat.completions.create(
    model='gemini-1.5-pro',
    messages=[{
        "role": "user",
        "content": [
            { "type": "image_url", "image_url": { "url": video_uri }},
            { "type": "text", "text": "Describe this video" }
        ]
    }],
)
print(completion.choices[0].message.content)

# First, encode your video file to base64
# For example: base64 -i video-example.mp4 -o video.b64
# Then use the encoded content in the request

curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
    "model": "gemini-1.5-pro",
    "messages": [{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this video"},
            {"type": "image_url", "image_url": {"url": "data:video/mp4;base64,YOUR_BASE64_VIDEO_DATA"}}
        ]
    }]
}'

Method 3: Sending Video via Google Files URL For larger video files, upload them using the Files API to get a Google Files URL.

const chatCompletion = await portkey.chat.completions.create({
    model: 'gemini-1.5-pro',
    messages: [{
        role: 'user',
        content: [
            {
                type: 'image_url',
                image_url: {
                    url: 'https://generativelanguage.googleapis.com/v1beta/files/your-video-file-id'
                }
            },
            { type: 'text', text: 'Please describe the main events in this video.' }
        ]
    }],
});
console.log(chatCompletion.choices[0].message.content);

completion = portkey.chat.completions.create(
    model='gemini-1.5-pro',
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://generativelanguage.googleapis.com/v1beta/files/your-video-file-id"
                }
            },
            { "type": "text", "text": "Please describe the main events in this video." }
        ]
    }],
)
print(completion.choices[0].message.content)

curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
    "model": "gemini-1.5-pro",
    "messages": [{
        "role": "user",
        "content": [
            {"type": "text", "text": "Please describe the main events in this video."},
            {"type": "image_url", "image_url": {"url": "https://generativelanguage.googleapis.com/v1beta/files/your-video-file-id"}}
        ]
    }]
}'

Supported Video MIME types: video/mp4, video/mpeg, video/mov, video/avi, video/webm, video/wmv

Document Processing (PDF)

Gemini’s vision capabilities excel at understanding the content of PDF documents, including text, tables, and images.

Gemini Documents Understanding Docs

Method 1: Sending a Document via Google Files URL Upload your PDF using the Files API to get a Google Files URL.

const chatCompletion = await portkey.chat.completions.create({
    model: 'gemini-1.5-pro',
    messages: [{
        role: 'user',
        content: [
            {
                type: 'image_url',
                image_url: {
                    url: 'https://generativelanguage.googleapis.com/v1beta/files/your-pdf-file-id'
                }
            },
            { type: 'text', text: 'Summarize the key findings of this research paper.' }
        ]
    }],
});
console.log(chatCompletion.choices[0].message.content);

completion = portkey.chat.completions.create(
    model='gemini-1.5-pro',
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://generativelanguage.googleapis.com/v1beta/files/your-pdf-file-id"
                }
            },
            { "type": "text", "text": "Summarize the key findings of this research paper." }
        ]
    }],
)
print(completion.choices[0].message.content)

curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
    "model": "gemini-1.5-pro",
    "messages": [{
        "role": "user",
        "content": [
            {"type": "text", "text": "Summarize the key findings of this research paper."},
            {"type": "image_url", "image_url": {"url": "https://generativelanguage.googleapis.com/v1beta/files/your-pdf-file-id"}}
        ]
    }]
}'

Method 2: Sending a Local Document as Base64 Data This is suitable for smaller, local PDF files.

import fs from 'fs';

const pdfBytes = fs.readFileSync('whitepaper.pdf');
const base64Pdf = pdfBytes.toString('base64');
const pdfUri = `data:application/pdf;base64,${base64Pdf}`;

const chatCompletion = await portkey.chat.completions.create({
    model: 'gemini-1.5-pro',
    messages: [{
        role: 'user',
        content: [
            { type: 'image_url', image_url: { url: pdfUri }},
            { type: 'text', text: 'What is the main conclusion of this document?' }
        ]
    }],
});
console.log(chatCompletion.choices[0].message.content);

import base64

with open("whitepaper.pdf", "rb") as pdf_file:
    pdf_bytes = pdf_file.read()

base64_pdf = base64.b64encode(pdf_bytes).decode('utf-8')
pdf_uri = f"data:application/pdf;base64,{base64_pdf}"

completion = portkey.chat.completions.create(
    model='gemini-1.5-pro',
    messages=[{
        "role": "user",
        "content": [
            { "type": "image_url", "image_url": { "url": pdf_uri }},
            { "type": "text", "text": "What is the main conclusion of this document?" }
        ]
    }],
)
print(completion.choices[0].message.content)

# First, encode your PDF file to base64
# For example: base64 -i whitepaper.pdf -o pdf.b64
# Then use the encoded content in the request

curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
    "model": "gemini-1.5-pro",
    "messages": [{
        "role": "user",
        "content": [
            {"type": "text", "text": "What is the main conclusion of this document?"},
            {"type": "image_url", "image_url": {"url": "data:application/pdf;base64,YOUR_BASE64_PDF_DATA"}}
        ]
    }]
}'

While you can send other document types like .txt or .html, they will be treated as plain text. Gemini’s native document vision capabilities are optimized for the application/pdf MIME type.

Important: For all file uploads (except YouTube videos), it’s recommended to use the Google Files API to upload your files first, then use the returned file URL in your requests. This approach provides better performance and reliability for larger files.

Media Resolution

The media_resolution parameter allows you to control token allocation for media inputs (images, videos, PDFs) when using Gemini models. This helps balance between processing detail and cost/speed.

Supported values

Value	Description
`MEDIA_RESOLUTION_LOW`	Reduced tokens for faster, cheaper processing
`MEDIA_RESOLUTION_MEDIUM`	Balanced approach between detail and cost
`MEDIA_RESOLUTION_HIGH`	Maximum tokens for detailed analysis
`MEDIA_RESOLUTION_ULTRA_HIGH`	Highest resolution (per-part only, for specialized tasks)

Top-level configuration

Apply media resolution globally to all media in the request:

const chatCompletion = await portkey.chat.completions.create({
    model: 'gemini-1.5-pro',
    media_resolution: 'MEDIA_RESOLUTION_HIGH',
    messages: [{
        role: 'user',
        content: [
            {
                type: 'image_url',
                image_url: {
                    url: 'https://generativelanguage.googleapis.com/v1beta/files/your-file-id'
                }
            },
            { type: 'text', text: 'Analyze this image in detail.' }
        ]
    }]
});

completion = portkey.chat.completions.create(
    model='gemini-1.5-pro',
    media_resolution='MEDIA_RESOLUTION_HIGH',
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://generativelanguage.googleapis.com/v1beta/files/your-file-id"
                }
            },
            { "type": "text", "text": "Analyze this image in detail." }
        ]
    }]
)

curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
    "model": "gemini-1.5-pro",
    "media_resolution": "MEDIA_RESOLUTION_HIGH",
    "messages": [{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": "https://generativelanguage.googleapis.com/v1beta/files/your-file-id"}},
            {"type": "text", "text": "Analyze this image in detail."}
        ]
    }]
}'

Per-part configuration (Gemini 3 only)

For Gemini 3 models, you can specify media resolution on individual media parts. Per-part settings take precedence over global settings when both are specified.

const chatCompletion = await portkey.chat.completions.create({
    model: 'gemini-3.0-pro',
    messages: [{
        role: 'user',
        content: [
            {
                type: 'image_url',
                image_url: {
                    url: 'https://generativelanguage.googleapis.com/v1beta/files/your-file-id',
                    media_resolution: 'MEDIA_RESOLUTION_HIGH'
                }
            },
            { type: 'text', text: 'Analyze this image in detail.' }
        ]
    }]
});

completion = portkey.chat.completions.create(
    model='gemini-3.0-pro',
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://generativelanguage.googleapis.com/v1beta/files/your-file-id",
                    "media_resolution": "MEDIA_RESOLUTION_HIGH"
                }
            },
            { "type": "text", "text": "Analyze this image in detail." }
        ]
    }]
)

curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
    "model": "gemini-3.0-pro",
    "messages": [{
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://generativelanguage.googleapis.com/v1beta/files/your-file-id",
                    "media_resolution": "MEDIA_RESOLUTION_HIGH"
                }
            },
            {"type": "text", "text": "Analyze this image in detail."}
        ]
    }]
}'

Google Gemini Media Resolution Documentation

Code Execution Tool

Gemini can use a built-in code interpreter tool to solve complex computational problems, perform calculations, and generate code. To enable this, simply include the code_execution tool in your request. The model will automatically decide when to invoke it.

const response = await portkey.chat.completions.create({
    model: "gemini-1.5-pro",
    messages: [{
        "role": "user",
        "content": "Calculate the 20th Fibonacci number. Then find the nearest palindrome to it."
    }],
    tools: [{ "type": "code_execution" }]
});

console.log(response.choices[0].message.content);

response = portkey.chat.completions.create(
    model="gemini-1.5-pro",
    messages=[{
        "role": "user",
        "content": "Calculate the 20th Fibonacci number. Then find the nearest palindrome to it."
    }],
    tools=[{ "type": "code_execution" }]
)

print(response.choices[0].message.content)

curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
    "model": "gemini-1.5-pro",
    "messages": [
        {
            "role": "user",
            "content": "Calculate the 20th Fibonacci number. Then find the nearest palindrome to it."
        }
    ],
    "tools": [{ "type": "code_execution" }]
}'

Thought Signatures (Tool Calling Verification)

Set x-portkey-strict-open-ai-compliance to false to receive the thought_signature in the response. This header must be included in all requests when using thought signatures.

Google’s Gemini 3 Pro model requires passing a thought_signature parameter in tool calling conversations for verifying the payload. This signature is returned by the model in the assistant’s tool call response and must be included when continuing multi-turn conversations.

Google Gemini Thought Signatures Documentation

Single turn conversation

In a single-turn conversation, you make a request with tools defined, and the model returns tool calls with thought signatures.

curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'Content-Type: application/json' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'x-portkey-strict-open-ai-compliance: false' \
--data '{
    "model": "gemini-3-pro-preview",
    "max_tokens": 1000,
    "stream": true,
    "messages": [
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": "You are a helpful assistant"
                }
            ]
        },
        {
            "role": "user",
            "content": "What is the current time in Bombay?"
        }
    ],
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "get_current_time",
                "description": "Get the current time for a specific location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g., San Francisco, CA"
                        }
                    },
                    "required": [
                        "location"
                    ]
                }
            }
        }
    ]
}'

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",
    provider="@PROVIDER",
    strict_open_ai_compliance=False
)

response = portkey.chat.completions.create(
    model="gemini-3-pro-preview",
    max_tokens=1000,
    stream=True,
    messages=[
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": "You are a helpful assistant"
                }
            ]
        },
        {
            "role": "user",
            "content": "What is the current time in Bombay?"
        }
    ],
    tools=[
        {
            "type": "function",
            "function": {
                "name": "get_current_time",
                "description": "Get the current time for a specific location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g., San Francisco, CA"
                        }
                    },
                    "required": [
                        "location"
                    ]
                }
            }
        }
    ]
)
print(response)

import Portkey from 'portkey-ai';

const portkey = new Portkey({
  apiKey: 'PORTKEY_API_KEY',
  provider: '@PROVIDER',
  strictOpenAiCompliance: false
});

const response = await portkey.chat.completions.create({
  model: 'gemini-3-pro-preview',
  max_tokens: 1000,
  stream: true,
  messages: [
    {
      role: 'system',
      content: [
        {
          type: 'text',
          text: 'You are a helpful assistant'
        }
      ]
    },
    {
      role: 'user',
      content: 'What is the current time in Bombay?'
    }
  ],
  tools: [
    {
      type: 'function',
      function: {
        name: 'get_current_time',
        description: 'Get the current time for a specific location',
        parameters: {
          type: 'object',
          properties: {
            location: {
              type: 'string',
              description: 'The city and state, e.g., San Francisco, CA'
            }
          },
          required: [
            'location'
          ]
        }
      }
    }
  ]
});
console.log(response);

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

openai = OpenAI(
    api_key='GEMINI_API_KEY',
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider='google',
        api_key='PORTKEY_API_KEY',
        strict_open_ai_compliance=False
    )
)

response = openai.chat.completions.create(
    model='gemini-3-pro-preview',
    max_tokens=1000,
    stream=True,
    messages=[
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": "You are a helpful assistant"
                }
            ]
        },
        {
            "role": "user",
            "content": "What is the current time in Bombay?"
        }
    ],
    tools=[
        {
            "type": "function",
            "function": {
                "name": "get_current_time",
                "description": "Get the current time for a specific location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g., San Francisco, CA"
                        }
                    },
                    "required": [
                        "location"
                    ]
                }
            }
        }
    ]
)
print(response)

import OpenAI from 'openai';
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai';

const openai = new OpenAI({
  apiKey: 'GEMINI_API_KEY',
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({
    provider: 'google',
    apiKey: 'PORTKEY_API_KEY',
    strictOpenAiCompliance: false
  })
});

const response = await openai.chat.completions.create({
  model: 'gemini-3-pro-preview',
  max_tokens: 1000,
  stream: true,
  messages: [
    {
      role: 'system',
      content: [
        {
          type: 'text',
          text: 'You are a helpful assistant'
        }
      ]
    },
    {
      role: 'user',
      content: 'What is the current time in Bombay?'
    }
  ],
  tools: [
    {
      type: 'function',
      function: {
        name: 'get_current_time',
        description: 'Get the current time for a specific location',
        parameters: {
          type: 'object',
          properties: {
            location: {
              type: 'string',
              description: 'The city and state, e.g., San Francisco, CA'
            }
          },
          required: [
            'location'
          ]
        }
      }
    }
  ]
});
console.log(response);

Multi turn conversation

In multi-turn conversations, you must include the thought_signature field in the assistant’s tool call when continuing the conversation.

curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: google' \
--header 'Content-Type: application/json' \
--header 'x-portkey-api-key: YOUR_PORTKEY_API_KEY' \
--header 'Authorization: YOUR_GEMINI_API_KEY' \
--header 'x-portkey-strict-open-ai-compliance: false' \
--data '{
    "model": "gemini-3-pro-preview",
    "max_tokens": 1000,
    "stream": true,
    "messages": [
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": "You are a helpful assistant"
                }
            ]
        },
        {
            "role": "user",
            "content": "Check the time in Chennai and if it is later than 9Pm get the temperature"
        },
        {
            "role": "assistant",
            "tool_calls": [
                {
                    "id": "portkey-1dcd51a0-a20a-482d-b244-2d4aff5aebdb",
                    "type": "function",
                    "function": {
                        "name": "get_current_time",
                        "arguments": "{\"location\":\"Chennai, India\"}",
                        "thought_signature": "CtQBAePx/17ARdotHH1RN31zOtCF+YpuOFTpU//tJRF4dEvegfDKLUaZnuG38II1POmVFdzBbzt87cTDr0TsEKHyHScN9PURHrhRer7liusjRrLR5QF4n1ZYJJYF3C+3bgC9YJsJyQhY/HAgVZQ53gq7n4I63CgXhYA+tzNN3CnHqdStgY0wLK0mCu/tb1kReSrXYMbre27SB5t2eRA7Wl+OKasKCOk7sYCJ8VkT+NaD+s6+NVTX2Au3RmUGVxYdjapo0vc7nnjvfmpTJHviyGJZIGIdXWw="
                    }
                }
            ]
        },
        {
            "role": "tool",
            "content": "{ '\''time'\'': '\''10PM'\'' }",
            "tool_call_id": "toolu_014jEfKqGbfFvRaKfiauxgPv"
        }
    ],
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "get_current_time",
                "description": "Get the current time for a specific location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g., San Francisco, CA"
                        }
                    },
                    "required": [
                        "location"
                    ]
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "get_current_temperature",
                "description": "Get the current temperature for a specific location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g., San Francisco, CA"
                        },
                        "unit": {
                            "type": "string",
                            "enum": [
                                "Celsius",
                                "Fahrenheit"
                            ],
                            "description": "The temperature unit to use. Infer this from the user'\''s location."
                        }
                    },
                    "required": [
                        "location",
                        "unit"
                    ]
                }
            }
        }
    ]
}'

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",
    provider="@PROVIDER",
    strict_open_ai_compliance=False
)

response = portkey.chat.completions.create(
    model="gemini-3-pro-preview",
    max_tokens=1000,
    stream=True,
    messages=[
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": "You are a helpful assistant"
                }
            ]
        },
        {
            "role": "user",
            "content": "Check the time in Chennai and if it is later than 9Pm get the temperature"
        },
        {
            "role": "assistant",
            "tool_calls": [
                {
                    "id": "portkey-1dcd51a0-a20a-482d-b244-2d4aff5aebdb",
                    "type": "function",
                    "function": {
                        "name": "get_current_time",
                        "arguments": "{\"location\":\"Chennai, India\"}",
                        "thought_signature": "CtQBAePx/17ARdotHH1RN31zOtCF+YpuOFTpU//tJRF4dEvegfDKLUaZnuG38II1POmVFdzBbzt87cTDr0TsEKHyHScN9PURHrhRer7liusjRrLR5QF4n1ZYJJYF3C+3bgC9YJsJyQhY/HAgVZQ53gq7n4I63CgXhYA+tzNN3CnHqdStgY0wLK0mCu/tb1kReSrXYMbre27SB5t2eRA7Wl+OKasKCOk7sYCJ8VkT+NaD+s6+NVTX2Au3RmUGVxYdjapo0vc7nnjvfmpTJHviyGJZIGIdXWw="
                    }
                }
            ]
        },
        {
            "role": "tool",
            "content": "{ 'time': '10PM' }",
            "tool_call_id": "toolu_014jEfKqGbfFvRaKfiauxgPv"
        }
    ],
    tools=[
        {
            "type": "function",
            "function": {
                "name": "get_current_time",
                "description": "Get the current time for a specific location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g., San Francisco, CA"
                        }
                    },
                    "required": [
                        "location"
                    ]
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "get_current_temperature",
                "description": "Get the current temperature for a specific location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g., San Francisco, CA"
                        },
                        "unit": {
                            "type": "string",
                            "enum": [
                                "Celsius",
                                "Fahrenheit"
                            ],
                            "description": "The temperature unit to use. Infer this from the user's location."
                        }
                    },
                    "required": [
                        "location",
                        "unit"
                    ]
                }
            }
        }
    ]
)
print(response)

import Portkey from 'portkey-ai';

const portkey = new Portkey({
  apiKey: 'PORTKEY_API_KEY',
  provider: '@PROVIDER'
});

const response = await portkey.chat.completions.create({
  model: 'gemini-3-pro-preview',
  max_tokens: 1000,
  stream: true,
  messages: [
    {
      role: 'system',
      content: [
        {
          type: 'text',
          text: 'You are a helpful assistant'
        }
      ]
    },
    {
      role: 'user',
      content: 'Check the time in Chennai and if it is later than 9Pm get the temperature'
    },
    {
      role: 'assistant',
      tool_calls: [
        {
          id: 'portkey-1dcd51a0-a20a-482d-b244-2d4aff5aebdb',
          type: 'function',
          function: {
            name: 'get_current_time',
            arguments: '{"location":"Chennai, India"}',
            thought_signature: 'CtQBAePx/17ARdotHH1RN31zOtCF+YpuOFTpU//tJRF4dEvegfDKLUaZnuG38II1POmVFdzBbzt87cTDr0TsEKHyHScN9PURHrhRer7liusjRrLR5QF4n1ZYJJYF3C+3bgC9YJsJyQhY/HAgVZQ53gq7n4I63CgXhYA+tzNN3CnHqdStgY0wLK0mCu/tb1kReSrXYMbre27SB5t2eRA7Wl+OKasKCOk7sYCJ8VkT+NaD+s6+NVTX2Au3RmUGVxYdjapo0vc7nnjvfmpTJHviyGJZIGIdXWw='
          }
        }
      ]
    },
    {
      role: 'tool',
      content: "{ 'time': '10PM' }",
      tool_call_id: 'toolu_014jEfKqGbfFvRaKfiauxgPv'
    }
  ],
  tools: [
    {
      type: 'function',
      function: {
        name: 'get_current_time',
        description: 'Get the current time for a specific location',
        parameters: {
          type: 'object',
          properties: {
            location: {
              type: 'string',
              description: 'The city and state, e.g., San Francisco, CA'
            }
          },
          required: [
            'location'
          ]
        }
      }
    },
    {
      type: 'function',
      function: {
        name: 'get_current_temperature',
        description: 'Get the current temperature for a specific location',
        parameters: {
          type: 'object',
          properties: {
            location: {
              type: 'string',
              description: 'The city and state, e.g., San Francisco, CA'
            },
            unit: {
              type: 'string',
              enum: [
                'Celsius',
                'Fahrenheit'
              ],
              description: 'The temperature unit to use. Infer this from the user\'s location.'
            }
          },
          required: [
            'location',
            'unit'
          ]
        }
      }
    }
  ]
});
console.log(response);

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

openai = OpenAI(
    api_key='GEMINI_API_KEY',
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider='google',
        api_key='PORTKEY_API_KEY',
        strict_open_ai_compliance=False
    )
)

response = openai.chat.completions.create(
    model='gemini-3-pro-preview',
    max_tokens=1000,
    stream=True,
    messages=[
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": "You are a helpful assistant"
                }
            ]
        },
        {
            "role": "user",
            "content": "Check the time in Chennai and if it is later than 9Pm get the temperature"
        },
        {
            "role": "assistant",
            "tool_calls": [
                {
                    "id": "portkey-1dcd51a0-a20a-482d-b244-2d4aff5aebdb",
                    "type": "function",
                    "function": {
                        "name": "get_current_time",
                        "arguments": "{\"location\":\"Chennai, India\"}",
                        "thought_signature": "CtQBAePx/17ARdotHH1RN31zOtCF+YpuOFTpU//tJRF4dEvegfDKLUaZnuG38II1POmVFdzBbzt87cTDr0TsEKHyHScN9PURHrhRer7liusjRrLR5QF4n1ZYJJYF3C+3bgC9YJsJyQhY/HAgVZQ53gq7n4I63CgXhYA+tzNN3CnHqdStgY0wLK0mCu/tb1kReSrXYMbre27SB5t2eRA7Wl+OKasKCOk7sYCJ8VkT+NaD+s6+NVTX2Au3RmUGVxYdjapo0vc7nnjvfmpTJHviyGJZIGIdXWw="
                    }
                }
            ]
        },
        {
            "role": "tool",
            "content": "{ 'time': '10PM' }",
            "tool_call_id": "toolu_014jEfKqGbfFvRaKfiauxgPv"
        }
    ],
    tools=[
        {
            "type": "function",
            "function": {
                "name": "get_current_time",
                "description": "Get the current time for a specific location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g., San Francisco, CA"
                        }
                    },
                    "required": [
                        "location"
                    ]
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "get_current_temperature",
                "description": "Get the current temperature for a specific location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g., San Francisco, CA"
                        },
                        "unit": {
                            "type": "string",
                            "enum": [
                                "Celsius",
                                "Fahrenheit"
                            ],
                            "description": "The temperature unit to use. Infer this from the user's location."
                        }
                    },
                    "required": [
                        "location",
                        "unit"
                    ]
                }
            }
        }
    ]
)
print(response)

import OpenAI from 'openai';
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai';

const openai = new OpenAI({
  apiKey: 'GEMINI_API_KEY',
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({
    provider: 'google',
    apiKey: 'PORTKEY_API_KEY'
  })
});

const response = await openai.chat.completions.create({
  model: 'gemini-3-pro-preview',
  max_tokens: 1000,
  stream: true,
  messages: [
    {
      role: 'system',
      content: [
        {
          type: 'text',
          text: 'You are a helpful assistant'
        }
      ]
    },
    {
      role: 'user',
      content: 'Check the time in Chennai and if it is later than 9Pm get the temperature'
    },
    {
      role: 'assistant',
      tool_calls: [
        {
          id: 'portkey-1dcd51a0-a20a-482d-b244-2d4aff5aebdb',
          type: 'function',
          function: {
            name: 'get_current_time',
            arguments: '{"location":"Chennai, India"}',
            thought_signature: 'CtQBAePx/17ARdotHH1RN31zOtCF+YpuOFTpU//tJRF4dEvegfDKLUaZnuG38II1POmVFdzBbzt87cTDr0TsEKHyHScN9PURHrhRer7liusjRrLR5QF4n1ZYJJYF3C+3bgC9YJsJyQhY/HAgVZQ53gq7n4I63CgXhYA+tzNN3CnHqdStgY0wLK0mCu/tb1kReSrXYMbre27SB5t2eRA7Wl+OKasKCOk7sYCJ8VkT+NaD+s6+NVTX2Au3RmUGVxYdjapo0vc7nnjvfmpTJHviyGJZIGIdXWw='
          }
        }
      ]
    },
    {
      role: 'tool',
      content: "{ 'time': '10PM' }",
      tool_call_id: 'toolu_014jEfKqGbfFvRaKfiauxgPv'
    }
  ],
  tools: [
    {
      type: 'function',
      function: {
        name: 'get_current_time',
        description: 'Get the current time for a specific location',
        parameters: {
          type: 'object',
          properties: {
            location: {
              type: 'string',
              description: 'The city and state, e.g., San Francisco, CA'
            }
          },
          required: [
            'location'
          ]
        }
      }
    },
    {
      type: 'function',
      function: {
        name: 'get_current_temperature',
        description: 'Get the current temperature for a specific location',
        parameters: {
          type: 'object',
          properties: {
            location: {
              type: 'string',
              description: 'The city and state, e.g., San Francisco, CA'
            },
            unit: {
              type: 'string',
              enum: [
                'Celsius',
                'Fahrenheit'
              ],
              description: 'The temperature unit to use. Infer this from the user\'s location.'
            }
          },
          required: [
            'location',
            'unit'
          ]
        }
      }
    }
  ]
});
console.log(response);

The thought_signature is automatically generated by the model and returned in the tool call response. You must preserve this signature when including the assistant’s message in subsequent requests.

Computer Use (Browser Automation) (Preview)

Set strict_open_ai_compliance to false to use the Computer Use tool.

Single turn conversation

import Portkey from 'portkey-ai';

const portkey = new Portkey({
  apiKey: 'PORTKEY_API_KEY',
  provider: '@PROVIDER',
  strictOpenAiCompliance: false
});

const response = await portkey.chat.completions.create({
  model: 'gemini-2.5-computer-use-preview-10-2025',
  stream: false,
  messages: [
    { role: 'system', content: 'You are a helpful assistant' },
    { role: 'user', content: "Go to google.com and search for 'weather in New York'" }
  ],
  tools: [
    {
      type: 'function',
      function: {
        name: 'computer_use',
        parameters: { environment: 'ENVIRONMENT_BROWSER' }
      }
    }
  ]
});
console.log(response);

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",
    provider="@PROVIDER",
    strict_open_ai_compliance=False
)

response = portkey.chat.completions.create(
    model="gemini-2.5-computer-use-preview-10-2025",
    stream=False,
    messages=[
        {"role": "system", "content": "You are a helpful assistant"},
        {"role": "user", "content": "Go to google.com and search for 'weather in New York'"}
    ],
    tools=[{
        "type": "function",
        "function": {
            "name": "computer_use",
            "parameters": {"environment": "ENVIRONMENT_BROWSER"}
        }
    }]
)
print(response)

import OpenAI from 'openai';
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai';

const openai = new OpenAI({
  apiKey: 'GEMINI_API_KEY',
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({
    provider: 'google',
    apiKey: 'PORTKEY_API_KEY',
    strictOpenAiCompliance: false
  })
});

const response = await openai.chat.completions.create({
  model: 'gemini-2.5-computer-use-preview-10-2025',
  stream: false,
  messages: [
    { role: 'system', content: 'You are a helpful assistant' },
    { role: 'user', content: "Go to google.com and search for 'weather in New York'" }
  ],
  tools: [{
    type: 'function',
    function: { name: 'computer_use', parameters: { environment: 'ENVIRONMENT_BROWSER' } }
  }]
});
console.log(response);

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

openai = OpenAI(
    api_key='GEMINI_API_KEY',
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider='google',
        api_key='PORTKEY_API_KEY',
        strict_open_ai_compliance=False
    )
)

response = openai.chat.completions.create(
    model='gemini-2.5-computer-use-preview-10-2025',
    stream=False,
    messages=[
        {"role": "system", "content": "You are a helpful assistant"},
        {"role": "user", "content": "Go to google.com and search for 'weather in New York'"}
    ],
    tools=[{
        "type": "function",
        "function": {"name": "computer_use", "parameters": {"environment": "ENVIRONMENT_BROWSER"}}
    }]
)
print(response)

curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: @my-vertex-ai-provider' \
--header 'Content-Type: application/json' \
--header 'x-portkey-api-key: your-api-key' \
--header 'x-portkey-strict-open-ai-compliance: false' \
--data '{
    "model": "gemini-2.5-computer-use-preview-10-2025",
    "stream": false,
    "messages": [
        {"role": "system", "content": "You are a helpful assistant"},
        {"role": "user", "content": "Go to google.com and search for '\''weather in New York'\''"}
    ],
    "tools": [
        {"type": "function", "function": {"name": "computer_use", "parameters": {"environment": "ENVIRONMENT_BROWSER"}}}
    ]
}'

Multi turn conversation

import Portkey from 'portkey-ai';

const portkey = new Portkey({
  apiKey: 'PORTKEY_API_KEY',
  provider: '@PROVIDER',
  strictOpenAiCompliance: false
});

const response = await portkey.chat.completions.create({
  model: 'gemini-2.5-computer-use-preview-10-2025',
  stream: false,
  messages: [
    { role: 'system', content: 'You are a helpful assistant' },
    { role: 'user', content: "Go to google.com and search for 'weather in New York'" },
    {
      role: 'assistant',
      tool_calls: [
        { id: 'portkey-50925c03-b8cc-4057-948b-13a9d9de19e0', type: 'function', function: { name: 'open_web_browser', arguments: '{}' } }
      ]
    },
    { role: 'user', content: "I've opened the browser" }
  ],
  tools: [{
    type: 'function',
    function: { name: 'computerUse', parameters: { environment: 'ENVIRONMENT_BROWSER' } }
  }]
});
console.log(response);

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",
    provider="@PROVIDER",
    strict_open_ai_compliance=False
)

response = portkey.chat.completions.create(
    model="gemini-2.5-computer-use-preview-10-2025",
    stream=False,
    messages=[
        {"role": "system", "content": "You are a helpful assistant"},
        {"role": "user", "content": "Go to google.com and search for 'weather in New York'"},
        {
            "role": "assistant",
            "tool_calls": [
                {"id": "portkey-50925c03-b8cc-4057-948b-13a9d9de19e0", "type": "function", "function": {"name": "open_web_browser", "arguments": "{}"}}
            ]
        },
        {"role": "user", "content": "I've opened the browser"}
    ],
    tools=[{
        "type": "function",
        "function": {"name": "computerUse", "parameters": {"environment": "ENVIRONMENT_BROWSER"}}
    }]
)
print(response)

import OpenAI from 'openai';
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai';

const openai = new OpenAI({
  apiKey: 'GEMINI_API_KEY',
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({ provider: 'google', apiKey: 'PORTKEY_API_KEY', strictOpenAiCompliance: false })
});

const response = await openai.chat.completions.create({
  model: 'gemini-2.5-computer-use-preview-10-2025',
  stream: false,
  messages: [
    { role: 'system', content: 'You are a helpful assistant' },
    { role: 'user', content: "Go to google.com and search for 'weather in New York'" },
    { role: 'assistant', tool_calls: [{ id: 'portkey-50925c03-b8cc-4057-948b-13a9d9de19e0', type: 'function', function: { name: 'open_web_browser', arguments: '{}' } }] },
    { role: 'user', content: "I've opened the browser" }
  ],
  tools: [{ type: 'function', function: { name: 'computerUse', parameters: { environment: 'ENVIRONMENT_BROWSER' } } }]
});
console.log(response);

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

openai = OpenAI(
    api_key='GEMINI_API_KEY',
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(provider='google', api_key='PORTKEY_API_KEY', strict_open_ai_compliance=False)
)

response = openai.chat.completions.create(
    model='gemini-2.5-computer-use-preview-10-2025',
    stream=False,
    messages=[
        {"role": "system", "content": "You are a helpful assistant"},
        {"role": "user", "content": "Go to google.com and search for 'weather in New York'"},
        {"role": "assistant", "tool_calls": [{"id": "portkey-50925c03-b8cc-4057-948b-13a9d9de19e0", "type": "function", "function": {"name": "open_web_browser", "arguments": "{}"}}]},
        {"role": "user", "content": "I've opened the browser"}
    ],
    tools=[{ "type": "function", "function": { "name": "computerUse", "parameters": { "environment": "ENVIRONMENT_BROWSER" } } }]
)
print(response)

curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: @my-vertex-ai-provider' \
--header 'Content-Type: application/json' \
--header 'x-portkey-api-key: your-api-key' \
--header 'x-portkey-strict-open-ai-compliance: false' \
--data '{
    "model": "gemini-2.5-computer-use-preview-10-2025",
    "stream": false,
    "messages": [
        {"role": "system", "content": "You are a helpful assistant"},
        {"role": "user", "content": "Go to google.com and search for 'weather in New York'"},
        {"role": "assistant", "tool_calls": [{"id": "portkey-50925c03-b8cc-4057-948b-13a9d9de19e0", "type": "function", "function": {"name": "open_web_browser", "arguments": "{}"}}]},
        {"role": "user", "content": "I've opened the browser"}
    ],
    "tools": [{"type": "function", "function": {"name": "computerUse", "parameters": {"environment": "ENVIRONMENT_BROWSER"}}}]
}'

Grounding with Google Search

Vertex AI supports grounding with Google Search. This is a feature that allows you to ground your LLM responses with real-time search results. Grounding is invoked by passing the google_search tool (for newer models like gemini-2.0-flash-001), and google_search_retrieval (for older models like gemini-1.5-flash) in the tools array.

"tools": [
    {
        "type": "function",
        "function": {
            "name": "google_search" // or google_search_retrieval for older models
        }
    }]

If you mix regular tools with grounding tools, vertex might throw an error saying only one tool can be used at a time.

Extended Thinking (Reasoning Models) (Beta)

The assistants thinking response is returned in the response_chunk.choices[0].delta.content_blocks array, not the response.choices[0].message.content string.

Models like gemini-2.5-flash-preview-04-17 gemini-2.5-flash-preview-04-17 support extended thinking. This is similar to openai thinking, but you get the model’s reasoning as it processes the request as well. Note that you will have to set strict_open_ai_compliance=False in the headers to use this feature.

Single turn conversation

from portkey_ai import Portkey

# Initialize the Portkey client
portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Replace with your Portkey API key
    provider="@PROVIDER",
    strict_open_ai_compliance=False
)

# Create the request
response = portkey.chat.completions.create(
  model="gemini-2.5-flash-preview-04-17",
  max_tokens=3000,
  thinking={
      "type": "enabled",
      "budget_tokens": 2030
  },
  stream=True,
  messages=[
      {
          "role": "user",
          "content": [
              {
                  "type": "text",
                  "text": "when does the flight from new york to bengaluru land tomorrow, what time, what is its flight number, and what is its baggage belt?"
              }
          ]
      }
  ]
)
print(response)
# in case of streaming responses you'd have to parse the response_chunk.choices[0].delta.content_blocks array
# response = portkey.chat.completions.create(
#   ...same config as above but with stream: true
# )
# for chunk in response:
#     if chunk.choices[0].delta:
#         content_blocks = chunk.choices[0].delta.get("content_blocks")
#         if content_blocks is not None:
#             for content_block in content_blocks:
#                 print(content_block)

import Portkey from 'portkey-ai';

// Initialize the Portkey client
const portkey = new Portkey({
  apiKey: "PORTKEY_API_KEY", // Replace with your Portkey API key
  provider:"@PROVIDER", // your Vertex AI provider slug
  strictOpenAiCompliance: false
});

// Generate a chat completion
async function getChatCompletionFunctions() {
const response = await portkey.chat.completions.create({
      model: "gemini-2.5-flash-preview-04-17",
      max_tokens: 3000,
      thinking: {
          type: "enabled",
          budget_tokens: 2030
      },
      stream: true,
      messages: [
          {
              role: "user",
              content: [
                  {
                      type: "text",
                      text: "when does the flight from new york to bengaluru land tomorrow, what time, what is its flight number, and what is its baggage belt?"
                  }
              ]
          }
      ]
    });
    console.log(response);
  // in case of streaming responses you'd have to parse the response_chunk.choices[0].delta.content_blocks array
  // const response = await portkey.chat.completions.create({
  //   ...same config as above but with stream: true
  // });
  // for await (const chunk of response) {
  //   if (chunk.choices[0].delta?.content_blocks) {
  //     for (const contentBlock of chunk.choices[0].delta.content_blocks) {
  //       console.log(contentBlock);
  //     }
  //   }
  // }
  }
// Call the function
getChatCompletionFunctions();

import OpenAI from 'openai'; // We're using the v4 SDK
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai'

const openai = new OpenAI({
  apiKey: 'VERTEX_API_KEY', // defaults to process.env["OPENAI_API_KEY"],
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({
    provider: "vertex-ai",
    apiKey: "PORTKEY_API_KEY", // defaults to process.env["PORTKEY_API_KEY"]
    strictOpenAiCompliance: false
  })
});

// Generate a chat completion with streaming
async function getChatCompletionFunctions(){
  const response = await openai.chat.completions.create({
    model: "gemini-2.5-flash-preview-04-17",
    max_tokens: 3000,
    thinking: {
        type: "enabled",
        budget_tokens: 2030
    },
    stream: true,
    messages: [
        {
            role: "user",
            content: [
                {
                    type: "text",
                    text: "when does the flight from new york to bengaluru land tomorrow, what time, what is its flight number, and what is its baggage belt?"
                }
            ]
        }
    ],
  });

  console.log(response)
  // in case of streaming responses you'd have to parse the response_chunk.choices[0].delta.content_blocks array
  // const response = await openai.chat.completions.create({
  //   ...same config as above but with stream: true
  // });
  // for await (const chunk of response) {
  //   if (chunk.choices[0].delta?.content_blocks) {
  //     for (const contentBlock of chunk.choices[0].delta.content_blocks) {
  //       console.log(contentBlock);
  //     }
  //   }
  // }
}
await getChatCompletionFunctions();

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

openai = OpenAI(
    api_key='VERTEX_API_KEY',
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider="vertex-ai",
        api_key="PORTKEY_API_KEY",
        strict_open_ai_compliance=False
    )
)


response = openai.chat.completions.create(
    model="gemini-2.5-flash-preview-04-17",
    max_tokens=3000,
    thinking={
        "type": "enabled",
        "budget_tokens": 2030
    },
    stream=True,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "when does the flight from new york to bengaluru land tomorrow, what time, what is its flight number, and what is its baggage belt?"
                }
            ]
        }
    ]
)

print(response)

curl "https://api.portkey.ai/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY" \
  -H "x-portkey-provider: vertex-ai" \
  -H "x-api-key: $VERTEX_API_KEY" \
  -H "x-portkey-strict-open-ai-compliance: false" \
  -d '{
    "model": "gemini-2.5-flash-preview-04-17",
    "max_tokens": 3000,
    "thinking": {
      "type": "enabled",
      "budget_tokens": 2030
    },
    "stream": true,
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "when does the flight from new york to bengaluru land tomorrow, what time, what is its flight number, and what is its baggage belt?"
          }
        ]
      }
    ]
  }'

To disable thinking for gemini models like gemini-2.5-flash-preview-04-17, you are required to explicitly set budget_tokens to 0.

"thinking": {
    "type": "enabled",
    "budget_tokens": 0
}

Gemini grounding mode may not work via Portkey SDK. Contact support@portkey.ai for assistance.

Image Generation (nano banana 🍌)

Gemini models like gemini-3-pro-image-preview support native image generation capabilities. You can generate images by setting modalities to include "image" in your request.

You must set strict_open_ai_compliance=False in the headers to use image generation, as the response format includes non-standard fields like content_parts.

The generated image data is returned in the content_parts field of the response and can be used in multi-turn conversations for iterative image editing.

from portkey_ai import Portkey

# Initialize the Portkey client
portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Replace with your Portkey API key
    provider="@google",  # Your Google Provider Slug
    strict_open_ai_compliance=False
)

# Create the request
response = portkey.chat.completions.create(
    model="gemini-3-pro-image-preview",
    max_tokens=32768,
    stream=False,
    modalities=["image"],
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Create a picture of a cat eating a nano-banana in a fancy restaurant under the Gemini constellation."
                }
            ]
        }
    ]
)
print(response)

# To access the generated image data:
# The image will be in response.choices[0].message.content_parts
# Each content_part with type "image" contains base64-encoded image data

import Portkey from 'portkey-ai';

// Initialize the Portkey client
const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY", // Replace with your Portkey API key
    provider: "@google", // Your Google Provider Slug
    strictOpenAiCompliance: false
});

// Generate an image
async function generateImage() {
    const response = await portkey.chat.completions.create({
        model: "gemini-3-pro-image-preview",
        max_tokens: 32768,
        stream: false,
        modalities: ["image"],
        messages: [
            {
                role: "system",
                content: "You are a helpful assistant."
            },
            {
                role: "user",
                content: [
                    {
                        type: "text",
                        text: "Create a picture of a cat eating a nano-banana in a fancy restaurant under the Gemini constellation."
                    }
                ]
            }
        ]
    });
    console.log(response);

    // To access the generated image data:
    // The image will be in response.choices[0].message.content_parts
    // Each content_part with type "image" contains base64-encoded image data
}

generateImage();

import OpenAI from 'openai';
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai';

const openai = new OpenAI({
    apiKey: 'GEMINI_API_KEY',
    baseURL: PORTKEY_GATEWAY_URL,
    defaultHeaders: createHeaders({
        provider: "@google", // Your Google Provider Slug
        apiKey: "PORTKEY_API_KEY",
        strictOpenAiCompliance: false
    })
});

async function generateImage() {
    const response = await openai.chat.completions.create({
        model: "gemini-3-pro-image-preview",
        max_tokens: 32768,
        stream: false,
        modalities: ["image"],
        messages: [
            {
                role: "system",
                content: "You are a helpful assistant."
            },
            {
                role: "user",
                content: [
                    {
                        type: "text",
                        text: "Create a picture of a cat eating a nano-banana in a fancy restaurant under the Gemini constellation."
                    }
                ]
            }
        ]
    });

    console.log(response);
}

generateImage();

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

openai = OpenAI(
    api_key='GEMINI_API_KEY',
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider="@google", // Your Google Provider Slug
        api_key="PORTKEY_API_KEY",
        strict_open_ai_compliance=False
    )
)

response = openai.chat.completions.create(
    model="gemini-3-pro-image-preview",
    max_tokens=32768,
    stream=False,
    modalities=["image"],
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Create a picture of a cat eating a nano-banana in a fancy restaurant under the Gemini constellation."
                }
            ]
        }
    ]
)

print(response)

curl "https://api.portkey.ai/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY" \
  -H "x-portkey-provider: @YOUR_" \
  -H "Authorization: $GEMINI_API_KEY" \
  -H "x-portkey-strict-open-ai-compliance: false" \
  -d '{
    "model": "gemini-3-pro-image-preview",
    "max_tokens": 32768,
    "stream": false,
    "modalities": ["image"],
    "messages": [
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Create a picture of a cat eating a nano-banana in a fancy restaurant under the Gemini constellation."
                }
            ]
        }
    ]
}'

Image Generation with Text Response

You can also generate images along with text explanations by including both "text" and "image" in the modalities array:

response = portkey.chat.completions.create(
    model="gemini-3-pro-image-preview",
    max_tokens=32768,
    stream=False,
    modalities=["text", "image"],  # Include both text and image
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Create a picture of a sunset over mountains and describe what you created."
                }
            ]
        }
    ]
)
print(response)

const response = await portkey.chat.completions.create({
    model: "gemini-3-pro-image-preview",
    max_tokens: 32768,
    stream: false,
    modalities: ["text", "image"],  // Include both text and image
    messages: [
        {
            role: "user",
            content: [
                {
                    type: "text",
                    text: "Create a picture of a sunset over mountains and describe what you created."
                }
            ]
        }
    ]
});
console.log(response);

curl "https://api.portkey.ai/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY" \
  -H "x-portkey-provider: google" \
  -H "Authorization: $GEMINI_API_KEY" \
  -H "x-portkey-strict-open-ai-compliance: false" \
  -d '{
    "model": "gemini-3-pro-image-preview",
    "max_tokens": 32768,
    "stream": false,
    "modalities": ["text", "image"],
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Create a picture of a sunset over mountains and describe what you created."
                }
            ]
        }
    ]
}'

Image Editing (Multi-turn)

You can edit generated images by continuing the conversation. Pass the image data from the previous response back in the messages:

# First, generate an initial image
initial_response = portkey.chat.completions.create(
    model="gemini-3-pro-image-preview",
    max_tokens=32768,
    modalities=["text", "image"],
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Create a picture of a croissant on a plate."
                }
            ]
        }
    ]
)

# Extract the assistant's response with the image
assistant_message = initial_response.choices[0].message

# Continue the conversation to edit the image
edit_response = portkey.chat.completions.create(
    model="gemini-3-pro-image-preview",
    max_tokens=32768,
    modalities=["text", "image"],
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Create a picture of a croissant on a plate."
                }
            ]
        },
        assistant_message,  # Include the previous response with image
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Add some chocolate drizzle on top of the croissant."
                }
            ]
        }
    ]
)
print(edit_response)

// First, generate an initial image
const initialResponse = await portkey.chat.completions.create({
    model: "gemini-3-pro-image-preview",
    max_tokens: 32768,
    modalities: ["text", "image"],
    messages: [
        {
            role: "user",
            content: [
                {
                    type: "text",
                    text: "Create a picture of a croissant on a plate."
                }
            ]
        }
    ]
});

// Extract the assistant's response with the image
const assistantMessage = initialResponse.choices[0].message;

// Continue the conversation to edit the image
const editResponse = await portkey.chat.completions.create({
    model: "gemini-3-pro-image-preview",
    max_tokens: 32768,
    modalities: ["text", "image"],
    messages: [
        {
            role: "user",
            content: [
                {
                    type: "text",
                    text: "Create a picture of a croissant on a plate."
                }
            ]
        },
        assistantMessage,  // Include the previous response with image
        {
            role: "user",
            content: [
                {
                    type: "text",
                    text: "Add some chocolate drizzle on top of the croissant."
                }
            ]
        }
    ]
});
console.log(editResponse);

Next Steps

SDK Reference

Complete SDK documentation and API reference

Add Metadata

Add metadata to your Gemini requests

Gateway Configs

Configure advanced gateway features

Request Tracing

Trace and monitor your Gemini requests

Setup Fallbacks

Create fallback configurations between providers

​Quick Start

​Add Provider in Model Catalog

​Gemini Capabilities

​Function Calling

​Advanced Multimodal Capabilities with Gemini

​Image Processing

Gemini Image Understanding Docs

​Audio Processing

Gemini Audio Understanding Docs

​Video Processing

Gemini Video Understanding Docs

​Document Processing (PDF)

Gemini Documents Understanding Docs

​Media Resolution

​Supported values

​Top-level configuration

​Per-part configuration (Gemini 3 only)

Google Gemini Media Resolution Documentation

​Code Execution Tool

​Thought Signatures (Tool Calling Verification)

Google Gemini Thought Signatures Documentation

​Single turn conversation

​Multi turn conversation

​Computer Use (Browser Automation) (Preview)

​Single turn conversation

​Multi turn conversation

​Grounding with Google Search

​Extended Thinking (Reasoning Models) (Beta)

​Single turn conversation

​Image Generation (nano banana 🍌)

​Image Generation with Text Response

​Image Editing (Multi-turn)

​Next Steps

SDK Reference

Add Metadata

Gateway Configs

Request Tracing

Setup Fallbacks

Quick Start

Add Provider in Model Catalog

Gemini Capabilities

Function Calling

Advanced Multimodal Capabilities with Gemini

Image Processing

Audio Processing

Video Processing

Document Processing (PDF)

Media Resolution

Supported values

Top-level configuration

Per-part configuration (Gemini 3 only)

Code Execution Tool

Thought Signatures (Tool Calling Verification)

Single turn conversation

Multi turn conversation

Computer Use (Browser Automation) (Preview)

Single turn conversation

Multi turn conversation

Grounding with Google Search

Extended Thinking (Reasoning Models) (Beta)

Single turn conversation

Image Generation (nano banana 🍌)

Image Generation with Text Response

Image Editing (Multi-turn)

Next Steps