Output:
Chunk ID: 9ede8d
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 3783
Output:
Gemini 3 prompting guide

Give clear and specific instructions

Giving the model instructions on what to do is an effective and efficient way to customize model behavior. Be clear, concise, and direct in the instructions, providing the needed context and outlining the desired output. Instructions can be as simple as a list of step-by-step instructions or as complex as mapping out a user's experience and mindset.

How to write effective instructions
Effective instructions apply the following principles:

Tell the model what to do.
Be clear and specific.
Specify any constraints or formatting requirements for the output.
Example
For example, suppose you own a cheeseburger restaurant and you want to use a model to help you learn about which menu items are the most popular. You want the model to format transcripts of customer orders in JSON so that you can quickly identify menu items. The following tables demonstrate the different responses the model provides when given generic instructions and specific instructions that follow the instruction principles:

Generic instructions
Prompt:

Extract the transcript in JSON.
Transcript:
Speaker 1 (Customer): Hi, can I get a cheeseburger and large fries, please?
Speaker 2 (Restaurant employee): Coming right up! Anything else you'd like to add to your order?
Speaker 1: Hmmm, maybe a small orange juice. And could I get the fries with ketchup on the side?
Speaker 2: No problem, one cheeseburger, one large fries with ketchup on the side, and a small
orange juice. That'll be $5.87. Drive through to the next window please.
Response:

      {
        "speakers": [
          {
            "speaker": "Customer",
            "utterances": [
              "Hi, can I get a cheeseburger and large fries, please?"
            ]
          },
          {
            "speaker": "Restaurant Employee",
            "utterances": [
              "Coming right up! Anything else you'd like to add to your order?"
            ]
          },
          {
            "speaker": "Customer",
            "utterances": [
              "Hmm, maybe a small orange juice. And could I get the fries with ketchup on the
              side?"
            ]
          },
          {
            "speaker": "Restaurant Employee",
            "utterances": [
              "No problem, one cheeseburger, one large fries with ketchup on the side, and a
              small orange juice. That'll be $5.87. Drive through to the next window please."
            ]
          }
        ]
      }

(gemini-pro)
In the example, generic instructions lead to a response with more JSON fields than what you need to identify the menu items customers are ordering. Improve the instructions by making more specific output constraints, as demonstrated in the following table:
Specific instructions
Prompt:

Extract the items from this transcript in JSON and separate drinks from food.
Transcript:
Speaker 1 (Customer): Hi, can I get a cheeseburger and large fries, please?
Speaker 2 (Restaurant employee): Coming right up! Anything else you'd like to add to your order?
Speaker 1: Hmmm, maybe a small orange juice. And could I get the fries with ketchup on the side?
Speaker 2: No problem, one cheeseburger, one large fries with ketchup on the side, and a small
orange juice. That'll be $5.87. Drive through to the next window please.
Response:

      {
        "drinks": [
          "orange juice"
        ],
        "food": [
          "cheeseburger",
          "large fries"
        ]
      }

(gemini-pro)
By applying the instruction principles, this example went from a response that contained too much data to one that contained exactly the data required for this use case. Instructions that use the instruction princples can help you guide the model to deliver the most helpful response for your use case.

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms, and the Additional Terms for Generative AI Preview Products. You can process personal data for this product or feature as outlined in the Cloud Data Processing Addendum, subject to the obligations and restrictions described in the agreement under which you access Google Cloud. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

Caution: gemini-3-pro-preview will be deprecated and removed on March 26, 2026. Migrate any workflows to gemini-3.1-pro-preview.
To see an example of Gemini 3 Pro, run the "Intro to Gemini 3 Pro" notebook in one of the following environments:

Open in Colab | Open in Colab Enterprise | Open in Vertex AI Workbench | View on GitHub

Prompting is a key part of working with any Gemini model and the new features of Gemini 3 models can be prompted to help solve complex problems and achieve other tasks, such as interpreting large amounts of text, solving complex mathematical problems, or even creating images and videos.

This guide provides a variety of prompting strategies to help you get the most from Gemini 3 on Vertex AI for a variety of use cases.

Temperature tuning
For Gemini 3, we strongly recommend keeping the temperature parameter at its default value of 1.0.

Gemini 3's reasoning capabilities are optimized for the default temperature setting and don't necessarily benefit from tuning temperature. Changing the temperature (setting it to less than 1.0) may lead to unexpected behavior, looping, or degraded performance, particularly with complex mathematical or reasoning tasks.

Prompting strategies
The following sections describe a variety of prompting strategies that you can use with Gemini 3 models.

Lowering response latency
For lower latency responses, try setting the thinking level to LOW and using system instructions like think silently.

Distinguishing between deduction and external information
In some cases, providing open-ended system instructions like do not infer or do not guess may cause the model to over-index on that instruction and fail to perform basic logic or arithmetic or synthesize information found in different parts of a document.

Rather than a large blanket negative constraint, tell the model explicitly to use the provided additional information or context for deductions and avoid using outside knowledge.

Examples

What was the profit? Do not infer.
This instruction is ineffective because the do not infer instruction is too broad.

You are expected to perform calculations and logical deductions based strictly
on the provided text. Do not introduce external information.
Here, the instruction makes it clear that the model should use the provided context for calculations and reasoning.

Using split-step verification
When the model encounters a topic it doesn't have sufficient information for (such as an obscure place) or is asked to perform an action it doesn't have capability for (such as accessing a specific live URL), it may generate seemingly plausible but incorrect information in an attempt to satisfy the request.

To avoid this, split the prompt into two steps: first, verify that the information or intended capability exists, then generate the answer based off of that information or capability.

Example

Verify with high confidence if you're able to access the New York Times home page.
If you cannot verify, state 'No Info' and STOP. If verified, proceed to generate
a response.

Query: Summarize the headlines from The New York Times today.
Organizing important information and constraints
When dealing with sufficiently complex requests, the model may drop negative constraints (specific instructions on what not to do) or formatting or quantitative constraints (instructions like word counts) if they appear too early in the prompt.

To mitigate this, place your core request and most critical restrictions as the final line of your instruction. In particular, negative constraints should be placed at the end of the instruction. A well-structured prompt might look like this:

[Context and source material]
[Main task instructions]
[Negative, formatting, and quantitative constraints]
Using personas
The model is designed to treat the persona it is assigned seriously and will sometimes ignore instructions in order to maintain adherence to the described persona. When using a persona with your prompts, review the persona that's assigned to the model and avoid ambiguous situations.

Example

You are a data extractor. You are forbidden from clarifying, explaining, or
expanding terms. Output text exactly as it appears. Do not explain why.
Maintaining grounding
The model may use its own knowledge to answer your prompt, which might conflict with any provided context. While the model is designed to be helpful, if you provide a hypothetical scenario that contradicts real-world facts (prompting with context such as Crabs are fictional and have never existed.), the model may revert to its training data rather than your prompt to align your request with its existing information.

If you need to work in context that isn't grounded in real-world information, explicitly state that the provided context is the only source of truth for the current session.

Example

You are a strictly grounded assistant limited to the information provided in the
User Context. In your answers, rely **only** on the facts that are directly
mentioned in that context. You must **not** access or utilize your own knowledge
or common sense to answer. Do not assume or infer from the provided facts;
simply report them exactly as they appear. Your answer must be factual and
fully truthful to the provided text, leaving absolutely no room for speculation
or interpretation. Treat the provided context as the absolute limit of truth;
any facts or details that are not directly mentioned in the context must be
considered **completely untruthful** and **completely unsupported**. If the
exact answer is not explicitly written in the context, you must state that the
information is not available.
Synthesizing multiple sources of information
When information is presented in multiple places across a source of context, the model can sometimes stop processing additional information after the first relevant match.

When working with large datasets, like entire books, codebases, or long videos, place your specific instructions or questions at the end of the prompt, after the data context. You can also anchor the model's reasoning to the provided data by starting your question with a phrase like Based on the entire document above....

Example instruction

Based on the entire document above, provide a comprehensive answer. Synthesize
all relevant information from the text that pertains to the question's scenario.
Steering output verbosity
By default, Gemini 3 models are less verbose and designed to prioritize providing direct and efficient answers.

If your use case requires a more conversational persona, you must explicitly steer the model to be chattier in the prompt.

Example instruction

Explain this as a friendly, talkative assistant.

Tune the sampling parameters
In each request, you send not only the multimodal prompt but a set of sampling parameters to the model. The model can generate different results for different parameter values. Experiment with the different parameters to get the best values for the task. The most commonly adjusted parameters are the following:

Temperature
top-P
Temperature
Temperature is used for sampling during response generation, which occurs when top-P and top-K are applied. Temperature controls the degree of randomness in token selection. Lower temperatures are good for prompts that require a more deterministic and less open-ended or creative response, while higher temperatures can lead to more diverse or creative results. A temperature of 0 is deterministic, meaning that the highest probability response is most likely to be selected.

For most use cases, start with a temperature of 0.4. If you need more creative results, increase the temperature. If you observe clear hallucinations, reduce the temperature.

Top-P
Top-P changes how the model selects tokens for output. Tokens are selected from the most to least probable until the sum of their probabilities equals the top-P value. For example, if tokens A, B, and C have a probability of 0.6, 0.3, 0.1 and the top-P value is 0.9, then the model will select either A or B as the next token by using temperature and excludes C as a candidate.

Specify a lower value for less random responses and a higher value for more random responses. The default value of top-P is 1.0.

Structure prompts

How you structure a prompt can affect the model's ability to parse the information in the prompt. It also helps the model correctly interpret how to use the given information. To give structure to a prompt, you can use prefixes or XML tags to delimit different parts or components of a prompt.

Use prefixes to structure simple prompts
A prefix is a word or phrase followed by a colon to label the information in a prompt. In the following example, "TASK:, "CLASSES:," and "OBJECTS:" are the prefixes.

Prompt:

TASK:
Classify the OBJECTS.

CLASSES:
- Large
- Small
OBJECTS:
- Rhino
- Mouse
- Snail
- Elephant
Response:

- Rhino: Large
- Mouse: Small
- Snail: Small
- Elephant: Large
(gemini-pro)
Use XML and other delimiters to structure complex prompts
For complex prompts, use XML and other delimiters to separate components of a prompt. You can use BEGIN and END or {} section delimiters for complex and lengthy prompt components to clearly distinguish them from the actual instructions.

You are a chatbot agent answering  customer's questions in a chat.
Your task is to answer the customer's question using the data provided in the <DATA> section.
  - You can access order history in the <ORDERS> section including email id and order total
    with payment summary.
  - Refer to <ORDERLINES> for item level details within each order in <ORDERS>.

Today is 2024-01-29

<DATA>
<ORDERS>
{OrderId|CustomerEmail|CreatedTimestamp|IsCancelled|OrderTotal|PaymentSummary
CC10182|222larabrown@gmail.com|2024-01-19|true|0.0|Not available
CC10183|baklavainthebalkans@gmail.com|2024-01-19|true|0.0|Not available}
{...}
...
</ORDERS>

<ORDERLINES>
OrderId|OrderLineId|CreatedTimestamp|ItemDescription|Quantity|FulfillmentStatus|ExpectedDeliveryDate
|ActualDeliveryDate|ActualShipDate|ExpectedShipDate|TrackingInformation|ShipToAddress|CarrierCode|De
liveryMethod|UnitPrice|OrderLineSubTotal|LineShippingCharge|TotalTaxes|Payments CC10182|1||Shorts|0.
0|unshipped|2024-01-31|2024-02-01|2024-01-30|2024-01-29||||ShipToAddress|115.99|0.0|0.0|0.0|
...
</ORDERLINES>
</DATA>

<INSTRUCTIONS>
- If there is no data that can help answer the question, respond with "I do not have this
  information. Please contact customer service".
- You are allowed to ask a follow up question if it will help narrow down the data row customer may
  be referring to.
- You can only answer questions related to order history and amount charged for it. Include OrderId


Output:
Chunk ID: 1d2c3d
Wall time: 0.0508 seconds
Process exited with code 0
Original token count: 3171
Output:
Start building free of charge with generous limits, then scale up with pay-as-you-go
pricing for your production ready applications.

### Free

For developers and small projects getting started with the Gemini API.

- check_circleLimited access to certain models
- check_circleFree input \& output tokens
- check_circleGoogle AI Studio access
- check_circleContent used to improve our products[\*](https://ai.google.dev/gemini-api/terms)

[Get started for Free](https://aistudio.google.com)

### Paid

For production applications that require higher volumes and advanced features.

- check_circleHigher rate limits for production deployments
- check_circleAccess to Context caching
- check_circleBatch API (50% cost reduction)
- check_circleAccess to Google's most advanced models
- check_circleContent **not** used to improve our products[\*](https://ai.google.dev/gemini-api/terms)

[Upgrade to Paid](https://aistudio.google.com/api-keys)

### Enterprise

For large-scale deployments with custom needs for security, support, and compliance, powered by [Vertex AI](https://cloud.google.com/vertex-ai).

- check_circleAll features in Paid, plus optional access to:
- check_circleDedicated support channels
- check_circleAdvanced security \& compliance
- check_circleProvisioned throughput
- check_circleVolume-based discounts (based on usage)
- check_circleML ops, model garden and more

[Contact Sales](https://cloud.google.com/contact)

## Gemini 3 Pro Preview

*`gemini-3-pro-preview`*

[Try it in Google AI Studio](https://aistudio.google.com?model=gemini-3-pro-preview)

The best model in the world for multimodal understanding, and our most powerful
agentic and vibe-coding model yet.

### Standard

|   | Free Tier | Paid Tier, per 1M tokens in USD |
|---|---|---|
| Input price | Not available | $2.00, prompts \<= 200k tokens $4.00, prompts \> 200k tokens |
| Output price (including thinking tokens) | Not available | $12.00, prompts \<= 200k tokens $18.00, prompts \> 200k |
| Context caching price | Not available | $0.20, prompts \<= 200k tokens $0.40, prompts \> 200k $4.50 / 1,000,000 tokens per hour (storage price) |
| Grounding with Google Search^\*^ | Not available | 5,000 prompts per month (free), then $14 / 1,000 search queries |
| Grounding with Google Maps | Not available | Not available |
| Used to improve our products | [Yes](https://ai.google.dev/gemini-api/terms) | [No](https://ai.google.dev/gemini-api/terms) |

### Batch

|   | Free Tier | Paid Tier, per 1M tokens in USD |
|---|---|---|
| Input price | Not available | $1.00, prompts \<= 200k tokens $2.00, prompts \> 200k tokens |
| Output price (including thinking tokens) | Not available | $6.00, prompts \<= 200k tokens $9.00, prompts \> 200k |
| Context caching price | Not available | $0.20, prompts \<= 200k tokens $0.40, prompts \> 200k $4.50 / 1,000,000 tokens per hour (storage price) |
| Grounding with Google Search^\*^ | Not available | 1,500 RPD (free), then $14 / 1,000 search queries |
| Grounding with Google Maps | Not available | Not available |
| Used to improve our products | [Yes](https://ai.google.dev/gemini-api/terms) | [No](https://ai.google.dev/gemini-api/terms) |

^\*^ A customer-submitted request to Gemini may result in one or more
queries to Google Search. You will be charged for each individual search query
performed.

## Gemini 3 Flash Preview

*`gemini-3-flash-preview`*

[Try it in Google AI Studio](https://aistudio.google.com?model=gemini-3-flash-preview)

Our most intelligent model built for speed, combining frontier intelligence with
superior search and grounding.

### Standard

|   | Free Tier | Paid Tier, per 1M tokens in USD |
|---|---|---|
| Input price | Free of charge | $0.50 (text / image / video) $1.00 (audio) |
| Output price (including thinking tokens) | Free of charge | $3.00 |
| Context caching price | Free of charge | $0.05 (text / image / video) $0.10 (audio) $1.00 / 1,000,000 tokens per hour (storage price) |
| Grounding with Google Search^\*^ | Not available | 5,000 prompts per month (free), then $14 / 1,000 search queries |
| Grounding with Google Maps | Not available | Not available |
| Used to improve our products | [Yes](https://ai.google.dev/gemini-api/terms) | [No](https://ai.google.dev/gemini-api/terms) |

### Batch

|   | Free Tier | Paid Tier, per 1M tokens in USD |
|---|---|---|
| Input price | Not available | $0.25 (text / image / video) $0.50 (audio) |
| Output price (including thinking tokens) | Not available | $1.50 |
| Context caching price | Not available | *Same as Standard, Batch pricing not yet implemented* $0.05 (text / image / video) $0.10 (audio) $1.00 / 1,000,000 tokens per hour (storage price) |
| Grounding with Google Search^\*^ | Not available | 1,500 RPD (free), then $14 / 1,000 search queries |
| Grounding with Google Maps | Not available | Not available |
| Used to improve our products | [Yes](https://ai.google.dev/gemini-api/terms) | [No](https://ai.google.dev/gemini-api/terms) |

^\*^ A customer-submitted request to Gemini may result in one or more
queries to Google Search. You will be charged for each individual search query
performed.

## Gemini 3 Pro Image Preview 🍌

*`gemini-3-pro-image-preview`*

[Try it in Google AI Studio](https://aistudio.google.com?model=gemini-3-pro-image-preview)

Our native image generation model, optimized for speed, flexibility, and
contextual understanding. **Text input and output** is priced the same as
[Gemini 3 Pro](https://ai.google.dev/gemini-api/docs/pricing#gemini-3-pro-preview).

Preview models may change before becoming stable and have more restrictive rate
limits.

### Standard

|   | Free Tier | Paid Tier, per 1M tokens in USD |
|---|---|---|
| Input price | Not available | $2.00 (text/image), equivalent to $0.0011 per image^\*^ |
| Output price | Not available | $12.00 (text and thinking) $120.00 (images) equivalent to $0.134 per 1K/2K image^\*\*^ and $0.24 per 4K image^\*\*^ |
| Used to improve our products | [Yes](https://ai.google.dev/gemini-api/terms) | [No](https://ai.google.dev/gemini-api/terms) |

### Batch

|   | Free Tier | Paid Tier, per 1M tokens in USD |
|---|---|---|
| Input price | Not available | $1.00 (text), $0.0006 (image)^\*^ |
| Output price | Not available | $6.00 (text and thinking) $0.067 per 1K/2K image^\*\*^ $0.12 per 4K image^\*\*^ |
| Used to improve our products | [Yes](https://ai.google.dev/gemini-api/terms) | [No](https://ai.google.dev/gemini-api/terms) |

^\*^ Image input is set at 560 tokens or $0.0011 per image.

^\*\*^ Image output is priced at $120 per 1,000,000 tokens. Output
images from 1024x1024px (1K) and up to 2048x2048px (2K) consume 1120 tokens and
are equivalent to $0.134 per image. Output images up to 4096x4096px (4K) consume
2000 tokens and are equivalent to $0.24 per image.

## Gemini 2.5 Pro

*`gemini-2.5-pro`*

[Try it in Google AI Studio](https://aistudio.google.com?model=gemini-2.5-pro)

Our state-of-the-art multipurpose model, which excels at coding and complex
reasoning tasks.

### Standard

|   | Free Tier | Paid Tier, per 1M tokens in USD |
|---|---|---|
| Input price | Free of charge | $1.25, prompts \<= 200k tokens $2.50, prompts \> 200k tokens |
| Output price (including thinking tokens) | Free of charge | $10.00, prompts \<= 200k tokens $15.00, prompts \> 200k |
| Context caching price | Not available | $0.125, prompts \<= 200k tokens $0.25, prompts \> 200k $4.50 / 1,000,000 tokens per hour (storage price) |
| Grounding with Google Search | Not available | 1,500 RPD (free), then $35 / 1,000 grounded prompts |
| Grounding with Google Maps | Not available | 10,000 RPD (free), then $25 / 1,000 grounded prompts |
| Used to improve our products | [Yes](https://ai.google.dev/gemini-api/terms) | [No](https://ai.google.dev/gemini-api/terms) |

### Batch

|   | Free Tier | Paid Tier, per 1M tokens in USD |
|---|---|---|
| Input price | Not available | $0.625, prompts \<= 200k tokens $1.25, prompts \> 200k tokens |
| Output price (including thinking tokens) | Not available | $5.00, prompts \<= 200k tokens $7.50, prompts \> 200k |
| Context caching price | Not available | $0.125, prompts \<= 200k tokens $0.25, prompts \> 200k $4.50 / 1,000,000 tokens per hour (storage price) |
| Grounding with Google Search | Not available | 1,500 RPD (free), then $35 / 1,000 grounded prompts |
| Grounding with Google Maps | Not available | Not available |
| Used to improve our products | [Yes](https://ai.google.dev/gemini-api/terms) | [No](https://ai.google.dev/gemini-api/terms) |

## Gemini 2.5 Flash

*`gemini-2.5-flash`*

[Try it in Google AI Studio](https://aistudio.google.com?model=gemini-2.5-flash)

Our first hybrid reasoning model which supports a 1M token context window and
has thinking budgets.

### Standard

|   | Free Tier | Paid Tier, per 1M tokens in USD |
|---|---|---|
| Input price | Free of charge | $0.30 (text / image / video) $1.00 (audio) |
| Output price (including thinking tokens) | Free of charge | $2.50 |
| Context caching price | Not available | $0.03 (text / image / video) $0.1 (audio) $1.00 / 1,000,000 tokens per hour (storage price) |
| Grounding with Google Search | Free of charge, up to 500 RPD (limit shared with Flash-Lite RPD) | 1,500 RPD (free, limit shared with Flash-Lite RPD), then $35 / 1,000 grounded prompts |
| Grounding with Google Maps | 500 RPD | 1,500 RPD (free), then $25 / 1,000 grounded prompts |
| Used to improve our products | [Yes](https://ai.google.dev/gemini-api/terms) | [No](https://ai.google.dev/gemini-api/terms) |

### Batch

|   | Free Tier | Paid Tier, per 1M tokens in USD |
|---|---|---|
| Input price | Not available | $0.15 (text / image / video) $0.50 (audio) |
| Output price (including thinking tokens) | Not available | $1.25 |
| Context caching price | Not available | $0.03 (text / image / video) $0.1 (audio) $1.00 / 1,000,000 tokens per hour (storage price) |
| Grounding with Google Search | Not available | 1,500 RPD (free, limit shared with Flash-Lite RPD), then $35 / 1,000 grounded prompts |
| Grounding with Google Maps | Not available | Not available |
| Used to improve our products | [Yes](https://ai.google.dev/gemini-api/terms) | [No](https://ai.google.dev/gemini-api/terms) |

## Gemini 2.5 Flash Preview

*`gemini-2.5-flash-preview-09-2025`*

[Try it in Google AI Studio](https://aistudio.google.com?model=gemini-2.5-flash-preview-09-2025)

The latest model based on the 2.5 Flash model. 2.5 Flash Preview is best for
large scale processing, low-latency,
high volume tasks that require thinking, and agentic use cases.

### Standard

|   | Free Tier | Paid Tier, per 1M tokens in USD |
|---|---|---|
| Input price | Free of charge | $0.30 (text / image / video) $1.00 (audio) |
| Output price (including thinking tokens) | Free of charge | $2.50 |
| Context caching price | Not available | $0.03 (text / image / video) $0.1 (audio) $1.00 / 1,000,000 tokens per hour (storage price) |
| Grounding with Google Search | Free of charge, up to 500 RPD (limit shared with Flash-Lite RPD) | 1,500 RPD (free, limit shared with Flash-Lite RPD), then $35 / 1,000 grounded prompts |
| Used to improve our products | [Yes](https://ai.google.dev/gemini-api/terms) | [No](https://ai.google.dev/gemini-api/terms) |

### Batch

|   | Free Tier | Paid Tier, per 1M tokens in USD |
|---|---|---|
| Input price | Not available | $0.15 (text / image / video) $0.50 (audio) |
| Output price (including thinking tokens) | Not available | $1.25 |
| Context caching price | Not available | $0.03 (text / image / video) $0.1 (audio) $1.00 / 1,000,000 tokens per hour (storage price) |
| Grounding with Google Search | Not available | 1,500 RPD (free, limit shared with Flash-Lite RPD), then $35 / 1,000 grounded prompts |
| Used to improve our products | [Yes](https://ai.google.dev/gemini-api/terms) | [No](https://ai.google.dev/gemini-api/terms) |

## Gemini 2.5 Flash-Lite

*`gemini-2.5-flash-lite`*

[Try it in Google AI Studio](https://aistudio.google.com?model=gemini-2.5-flash-lite)

Our smallest and most cost effective model, built for at scale usage.

### Standard

|   | Free Tier | Paid Tier, per 1M tokens in USD |
|---|---|---|
| Input price (text, image, video) | Free of charge | $0.10 (text / image / video) $0.30 (audio) |
| Output price (including thinking tokens) | Free of charge | $0.40 |
| Context caching price | Not available | $0.01 (text / image / video) $0.03 (audio) $1.00 / 1,000,000 tokens per hour (storage price) |
| Grounding with Google Search | Free of charge, up to 500 RPD (limit shared with Flash RPD) | 1,500 RPD (free, limit shared with Flash RPD), then $35 / 1,000 grounded prompts |


Output:
Chunk ID: afc94c
Wall time: 0.0504 seconds
Process exited with code 0
Original token count: 2504
Output:
<br />

PythonJavaScriptGoREST

In a typical AI workflow, you might pass the same input tokens over and over to a model. The Gemini API offers two different caching mechanisms:

- Implicit caching (automatically enabled on most Gemini models, no cost saving guarantee)
- Explicit caching (can be manually enabled on most models, cost saving guarantee)

Explicit caching is useful in cases where you want to guarantee cost savings, but with some added developer work.

## Implicit caching

Implicit caching is enabled by default and available for most Gemini models. We automatically pass on cost savings if your request hits caches. There is nothing you need to do in order to enable this. It is effective as of May 8th, 2025. The minimum input token count for context caching is listed in the following table for each model:

|         Model          | Min token limit |
|------------------------|-----------------|
| Gemini 3 Flash Preview | 1024            |
| Gemini 3 Pro Preview   | 4096            |
| Gemini 2.5 Flash       | 1024            |
| Gemini 2.5 Pro         | 4096            |

To increase the chance of an implicit cache hit:

- Try putting large and common contents at the beginning of your prompt
- Try to send requests with similar prefix in a short amount of time

You can see the number of tokens which were cache hits in the response object's`usage_metadata`field.

## Explicit caching

Using the Gemini API explicit caching feature, you can pass some content to the model once, cache the input tokens, and then refer to the cached tokens for subsequent requests. At certain volumes, using cached tokens is lower cost than passing in the same corpus of tokens repeatedly.

When you cache a set of tokens, you can choose how long you want the cache to exist before the tokens are automatically deleted. This caching duration is called the*time to live*(TTL). If not set, the TTL defaults to 1 hour. The cost for caching depends on the input token size and how long you want the tokens to persist.

This section assumes that you've installed a Gemini SDK (or have curl installed) and that you've configured an API key, as shown in the[quickstart](https://ai.google.dev/gemini-api/docs/quickstart).

### Generate content using a cache

The following example shows how to generate content using a cached system instruction and video file.

### Videos

    import os
    import pathlib
    import requests
    import time

    from google import genai
    from google.genai import types

    client = genai.Client()

    # Download video file
    url = 'https://storage.googleapis.com/generativeai-downloads/data/SherlockJr._10min.mp4'
    path_to_video_file = pathlib.Path('SherlockJr._10min.mp4')
    if not path_to_video_file.exists():
      with path_to_video_file.open('wb') as wf:
        response = requests.get(url, stream=True)
        for chunk in response.iter_content(chunk_size=32768):
          wf.write(chunk)

    # Upload the video using the Files API
    video_file = client.files.upload(file=path_to_video_file)

    # Wait for the file to finish processing
    while video_file.state.name == 'PROCESSING':
      print('Waiting for video to be processed.')
      time.sleep(2)
      video_file = client.files.get(name=video_file.name)

    print(f'Video processing complete: {video_file.uri}')

    # You must use an explicit version suffix: "-flash-001", not just "-flash".
    model='models/gemini-2.0-flash-001'

    # Create a cache with a 5 minute TTL
    cache = client.caches.create(
        model=model,
        config=types.CreateCachedContentConfig(
          display_name='sherlock jr movie', # used to identify the cache
          system_instruction=(
              'You are an expert video analyzer, and your job is to answer '
              'the user\'s query based on the video file you have access to.'
          ),
          contents=[video_file],
          ttl="300s",
      )
    )

    # Construct a GenerativeModel which uses the created cache.
    response = client.models.generate_content(
      model = model,
      contents= (
        'Introduce different characters in the movie by describing '
        'their personality, looks, and names. Also list the timestamps '
        'they were introduced for the first time.'),
      config=types.GenerateContentConfig(cached_content=cache.name)
    )

    print(response.usage_metadata)

    # The output should look something like this:
    #
    # prompt_token_count: 696219
    # cached_content_token_count: 696190
    # candidates_token_count: 214
    # total_token_count: 696433

    print(response.text)

### PDFs

    from google import genai
    from google.genai import types
    import io
    import httpx

    client = genai.Client()

    long_context_pdf_path = "https://www.nasa.gov/wp-content/uploads/static/history/alsj/a17/A17_FlightPlan.pdf"

    # Retrieve and upload the PDF using the File API
    doc_io = io.BytesIO(httpx.get(long_context_pdf_path).content)

    document = client.files.upload(
      file=doc_io,
      config=dict(mime_type='application/pdf')
    )

    model_name = "gemini-2.0-flash-001"
    system_instruction = "You are an expert analyzing transcripts."

    # Create a cached content object
    cache = client.caches.create(
        model=model_name,
        config=types.CreateCachedContentConfig(
          system_instruction=system_instruction,
          contents=[document],
        )
    )

    # Display the cache details
    print(f'{cache=}')

    # Generate content using the cached prompt and document
    response = client.models.generate_content(
      model=model_name,
      contents="Please summarize this transcript",
      config=types.GenerateContentConfig(
        cached_content=cache.name
      ))

    # (Optional) Print usage metadata for insights into the API call
    print(f'{response.usage_metadata=}')

    # Print the generated text
    print('\n\n', response.text)

### List caches

It's not possible to retrieve or view cached content, but you can retrieve cache metadata (`name`,`model`,`display_name`,`usage_metadata`,`create_time`,`update_time`, and`expire_time`).

To list metadata for all uploaded caches, use`CachedContent.list()`:

    for cache in client.caches.list():
      print(cache)

To fetch the metadata for one cache object, if you know its name, use`get`:

    client.caches.get(name=name)

### Update a cache

You can set a new`ttl`or`expire_time`for a cache. Changing anything else about the cache isn't supported.

The following example shows how to update the`ttl`of a cache using`client.caches.update()`.

    from google import genai
    from google.genai import types

    client.caches.update(
      name = cache.name,
      config  = types.UpdateCachedContentConfig(
          ttl='300s'
      )
    )

To set the expiry time, it will accepts either a`datetime`object or an ISO-formatted datetime string (`dt.isoformat()`, like`2025-01-27T16:02:36.473528+00:00`). Your time must include a time zone (`datetime.utcnow()`doesn't attach a time zone,`datetime.now(datetime.timezone.utc)`does attach a time zone).

    from google import genai
    from google.genai import types
    import datetime

    # You must use a time zone-aware time.
    in10min = datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(minutes=10)

    client.caches.update(
      name = cache.name,
      config  = types.UpdateCachedContentConfig(
          expire_time=in10min
      )
    )

### Delete a cache

The caching service provides a delete operation for manually removing content from the cache. The following example shows how to delete a cache:

    client.caches.delete(cache.name)

### Explicit caching using the OpenAI library

If you're using an[OpenAI library](https://ai.google.dev/gemini-api/docs/openai), you can enable explicit caching using the`cached_content`property on[`extra_body`](https://ai.google.dev/gemini-api/docs/openai#extra-body).

## When to use explicit caching

Context caching is particularly well suited to scenarios where a substantial initial context is referenced repeatedly by shorter requests. Consider using context caching for use cases such as:

- Chatbots with extensive[system instructions](https://ai.google.dev/gemini-api/docs/system-instructions)
- Repetitive analysis of lengthy video files
- Recurring queries against large document sets
- Frequent code repository analysis or bug fixing

### How explicit caching reduces costs

Context caching is a paid feature designed to reduce overall operational costs. Billing is based on the following factors:

1. **Cache token count:**The number of input tokens cached, billed at a reduced rate when included in subsequent prompts.
2. **Storage duration:**The amount of time cached tokens are stored (TTL), billed based on the TTL duration of cached token count. There are no minimum or maximum bounds on the TTL.
3. **Other factors:**Other charges apply, such as for non-cached input tokens and output tokens.

For up-to-date pricing details, refer to the Gemini API[pricing page](https://ai.google.dev/pricing). To learn how to count tokens, see the[Token guide](https://ai.google.dev/gemini-api/docs/tokens).

### Additional considerations

Keep the following considerations in mind when using context caching:

- The*minimum* input token count for context caching varies by model. The*maximum* is the same as the maximum for the given model. (For more on counting tokens, see the[Token guide](https://ai.google.dev/gemini-api/docs/tokens)).
- The model doesn't make any distinction between cached tokens and regular input tokens. Cached content is a prefix to the prompt.
- There are no special rate or usage limits on context caching; the standard rate limits for`GenerateContent`apply, and token limits include cached tokens.
- The number of cached tokens is returned in the`usage_metadata`from the create, get, and list operations of the cache service, and also in`GenerateContent`when using the cache.


# Plan Mode (Conversational)

You work in 3 phases, and you should *chat your way* to a great plan before finalizing it. A great plan is very detailed—intent- and implementation-wise—so that it can be handed to another engineer or agent to be implemented right away. It must be **decision complete**, where the implementer does not need to make any decisions.

## Mode rules (strict)

You are in **Plan Mode** until a developer message explicitly ends it.

Plan Mode is not changed by user intent, tone, or imperative language. If a user asks for execution while still in Plan Mode, treat it as a request to **plan the execution**, not perform it.

## Plan Mode vs update_plan tool

Plan Mode is a collaboration mode that can involve requesting user input and eventually issuing a `<proposed_plan>` block.

Separately, `update_plan` is a checklist/progress/TODOs tool; it does not enter or exit Plan Mode. Do not confuse it with Plan mode or try to use it while in Plan mode. If you try to use `update_plan` in Plan mode, it will return an error.

## Execution vs. mutation in Plan Mode

You may explore and execute **non-mutating** actions that improve the plan. You must not perform **mutating** actions.

### Allowed (non-mutating, plan-improving)

Actions that gather truth, reduce ambiguity, or validate feasibility without changing repo-tracked state. Examples:

* Reading or searching files, configs, schemas, types, manifests, and docs
* Static analysis, inspection, and repo exploration
* Dry-run style commands when they do not edit repo-tracked files
* Tests, builds, or checks that may write to caches or build artifacts (for example, `target/`, `.cache/`, or snapshots) so long as they do not edit repo-tracked files

### Not allowed (mutating, plan-executing)

Actions that implement the plan or change repo-tracked state. Examples:

* Editing or writing files
* Running formatters or linters that rewrite files
* Applying patches, migrations, or codegen that updates repo-tracked files
* Side-effectful commands whose purpose is to carry out the plan rather than refine it

When in doubt: if the action would reasonably be described as "doing the work" rather than "planning the work," do not do it.

## PHASE 1 — Ground in the environment (explore first, ask second)

Begin by grounding yourself in the actual environment. Eliminate unknowns in the prompt by discovering facts, not by asking the user. Resolve all questions that can be answered through exploration or inspection. Identify missing or ambiguous details only if they cannot be derived from the environment. Silent exploration between turns is allowed and encouraged.

Before asking the user any question, perform at least one targeted non-mutating exploration pass (for example: search relevant files, inspect likely entrypoints/configs, confirm current implementation shape), unless no local environment/repo is available.

Exception: you may ask clarifying questions about the user's prompt before exploring, ONLY if there are obvious ambiguities or contradictions in the prompt itself. However, if ambiguity might be resolved by exploring, always prefer exploring first.

Do not ask questions that can be answered from the repo or system (for example, "where is this struct?" or "which UI component should we use?" when exploration can make it clear). Only ask once you have exhausted reasonable non-mutating exploration.

## PHASE 2 — Intent chat (what they actually want)

* Keep asking until you can clearly state: goal + success criteria, audience, in/out of scope, constraints, current state, and the key preferences/tradeoffs.
* Bias toward questions over guessing: if any high-impact ambiguity remains, do NOT plan yet—ask.

## PHASE 3 — Implementation chat (what/how we’ll build)

* Once intent is stable, keep asking until the spec is decision complete: approach, interfaces (APIs/schemas/I/O), data flow, edge cases/failure modes, testing + acceptance criteria, rollout/monitoring, and any migrations/compat constraints.

## Asking questions

Critical rules:

* Strongly prefer using the `request_user_input` tool to ask any questions.
* Offer only meaningful multiple‑choice options; don’t include filler choices that are obviously wrong or irrelevant.
* In rare cases where an unavoidable, important question can’t be expressed with reasonable multiple‑choice options (due to extreme ambiguity), you may ask it directly without the tool.

You SHOULD ask many questions, but each question must:

* materially change the spec/plan, OR
* confirm/lock an assumption, OR
* choose between meaningful tradeoffs.
* not be answerable by non-mutating commands.

Use the `request_user_input` tool only for decisions that materially change the plan, for confirming important assumptions, or for information that cannot be discovered via non-mutating exploration.

## Two kinds of unknowns (treat differently)

1. **Discoverable facts** (repo/system truth): explore first.

   * Before asking, run targeted searches and check likely sources of truth (configs/manifests/entrypoints/schemas/types/constants).
   * Ask only if: multiple plausible candidates; nothing found but you need a missing identifier/context; or ambiguity is actually product intent.
   * If asking, present concrete candidates (paths/service names) + recommend one.
   * Never ask questions you can answer from your environment (e.g., “where is this struct”).

2. **Preferences/tradeoffs** (not discoverable): ask early.

   * These are intent or implementation preferences that cannot be derived from exploration.
   * Provide 2–4 mutually exclusive options + a recommended default.
   * If unanswered, proceed with the recommended option and record it as an assumption in the final plan.

## Finalization rule

Only output the final plan when it is decision complete and leaves no decisions to the implementer.

When you present the official plan, wrap it in a `<proposed_plan>` block so the client can render it specially:

1) The opening tag must be on its own line.
2) Start the plan content on the next line (no text on the same line as the tag).
3) The closing tag must be on its own line.
4) Use Markdown inside the block.
5) Keep the tags exactly as `<proposed_plan>` and `</proposed_plan>` (do not translate or rename them), even if the plan content is in another language.

Example:

<proposed_plan>
plan content
</proposed_plan>

plan content should be human and agent digestible. The final plan must be plan-only and include:

* A clear title
* A brief summary section
* Important changes or additions to public APIs/interfaces/types
* Test cases and scenarios
* Explicit assumptions and defaults chosen where needed

Do not ask "should I proceed?" in the final output. The user can easily switch out of Plan mode and request implementation if you have included a `<proposed_plan>` block in your response. Alternatively, they can decide to stay in Plan mode and continue refining the plan.

Only produce at most one `<proposed_plan>` block per turn, and only when you are presenting a complete spec.

If the user stays in Plan mode and asks for revisions after a prior `<proposed_plan>`, any new `<proposed_plan>` must be a complete replacement.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Planning local docs review**


I’ve confirmed local docs cover Batch API, schema enforcement, caching, and pricing details that matter for your 4-day throughput target. I’ll quickly inspect the OpenRouter docs in this repo next so fallback routing is concrete rather than generic.


sed -n '1,260p' docs/openrouter/structured_outputs.md


sed -n '1,260p' docs/openrouter/latency\ and\ performance.md


sed -n '1,260p' docs/openrouter/prompt_caching.md


Output:
Chunk ID: d47be0
Wall time: 0.0519 seconds
Process exited with code 0
Original token count: 2053
Output:
***

title: Structured Outputs
subtitle: Return structured data from your models
headline: Structured Outputs | Enforce JSON Schema in OpenRouter API Responses
canonical-url: '[https://openrouter.ai/docs/guides/features/structured-outputs](https://openrouter.ai/docs/guides/features/structured-outputs)'
'og:site\_name': OpenRouter Documentation
'og:title': Structured Outputs - Type-Safe JSON Responses from AI Models
'og:description': >-
Enforce JSON Schema validation on AI model responses. Get consistent,
type-safe outputs and avoid parsing errors with OpenRouter's structured output
feature.
'og:image':
type: url
value: >-
[https://openrouter.ai/dynamic-og?title=Structured%20Outputs\&description=Type-Safe%20JSON%20Responses%20from%20AI%20Models](https://openrouter.ai/dynamic-og?title=Structured%20Outputs\&description=Type-Safe%20JSON%20Responses%20from%20AI%20Models)
'og:image:width': 1200
'og:image:height': 630
'twitter:card': summary\_large\_image
'twitter:site': '@OpenRouter'
noindex: false
nofollow: false
---------------

OpenRouter supports structured outputs for compatible models, ensuring responses follow a specific JSON Schema format. This feature is particularly useful when you need consistent, well-formatted responses that can be reliably parsed by your application.

## Overview

Structured outputs allow you to:

* Enforce specific JSON Schema validation on model responses
* Get consistent, type-safe outputs
* Avoid parsing errors and hallucinated fields
* Simplify response handling in your application

## Using Structured Outputs

To use structured outputs, include a `response_format` parameter in your request, with `type` set to `json_schema` and the `json_schema` object containing your schema:

```typescript
{
  "messages": [
    { "role": "user", "content": "What's the weather like in London?" }
  ],
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "weather",
      "strict": true,
      "schema": {
        "type": "object",
        "properties": {
          "location": {
            "type": "string",
            "description": "City or location name"
          },
          "temperature": {
            "type": "number",
            "description": "Temperature in Celsius"
          },
          "conditions": {
            "type": "string",
            "description": "Weather conditions description"
          }
        },
        "required": ["location", "temperature", "conditions"],
        "additionalProperties": false
      }
    }
  }
}
```

The model will respond with a JSON object that strictly follows your schema:

```json
{
  "location": "London",
  "temperature": 18,
  "conditions": "Partly cloudy with light drizzle"
}
```

## Model Support

Structured outputs are supported by select models.

You can find a list of models that support structured outputs on the [models page](https://openrouter.ai/models?order=newest\&supported_parameters=structured_outputs).

* OpenAI models (GPT-4o and later versions) [Docs](https://platform.openai.com/docs/guides/structured-outputs)
* Google Gemini models [Docs](https://ai.google.dev/gemini-api/docs/structured-output)
* Anthropic models (Sonnet 4.5 and Opus 4.1) [Docs](https://docs.claude.com/en/docs/build-with-claude/structured-outputs)
* Most open-source models
* All Fireworks provided models [Docs](https://docs.fireworks.ai/structured-responses/structured-response-formatting#structured-response-modes)

To ensure your chosen model supports structured outputs:

1. Check the model's supported parameters on the [models page](https://openrouter.ai/models)
2. Set `require_parameters: true` in your provider preferences (see [Provider Routing](/docs/features/provider-routing))
3. Include `response_format` and set `type: json_schema` in the required parameters

## Best Practices

1. **Include descriptions**: Add clear descriptions to your schema properties to guide the model

2. **Use strict mode**: Always set `strict: true` to ensure the model follows your schema exactly

## Example Implementation

Here's a complete example using the Fetch API:

<Template
  data={{
  API_KEY_REF,
  MODEL: 'openai/gpt-4'
}}
>
  <CodeGroup>
    ```typescript title="TypeScript SDK"
    import { OpenRouter } from '@openrouter/sdk';

    const openRouter = new OpenRouter({
      apiKey: '{{API_KEY_REF}}',
    });

    const response = await openRouter.chat.send({
      model: '{{MODEL}}',
      messages: [
        { role: 'user', content: 'What is the weather like in London?' },
      ],
      responseFormat: {
        type: 'json_schema',
        jsonSchema: {
          name: 'weather',
          strict: true,
          schema: {
            type: 'object',
            properties: {
              location: {
                type: 'string',
                description: 'City or location name',
              },
              temperature: {
                type: 'number',
                description: 'Temperature in Celsius',
              },
              conditions: {
                type: 'string',
                description: 'Weather conditions description',
              },
            },
            required: ['location', 'temperature', 'conditions'],
            additionalProperties: false,
          },
        },
      },
      stream: false,
    });

    const weatherInfo = response.choices[0].message.content;
    ```

    ```python title="Python"
    import requests
    import json

    response = requests.post(
      "https://openrouter.ai/api/v1/chat/completions",
      headers={
        "Authorization": f"Bearer {{API_KEY_REF}}",
        "Content-Type": "application/json",
      },

      json={
        "model": "{{MODEL}}",
        "messages": [
          {"role": "user", "content": "What is the weather like in London?"},
        ],
        "response_format": {
          "type": "json_schema",
          "json_schema": {
            "name": "weather",
            "strict": True,
            "schema": {
              "type": "object",
              "properties": {
                "location": {
                  "type": "string",
                  "description": "City or location name",
                },
                "temperature": {
                  "type": "number",
                  "description": "Temperature in Celsius",
                },
                "conditions": {
                  "type": "string",
                  "description": "Weather conditions description",
                },
              },
              "required": ["location", "temperature", "conditions"],
              "additionalProperties": False,
            },
          },
        },
      },
    )

    data = response.json()
    weather_info = data["choices"][0]["message"]["content"]
    ```

    ```typescript title="TypeScript (fetch)"
    const response = await fetch('https://openrouter.ai/api/v1/chat/completions', {
      method: 'POST',
      headers: {
        Authorization: 'Bearer {{API_KEY_REF}}',
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        model: '{{MODEL}}',
        messages: [
          { role: 'user', content: 'What is the weather like in London?' },
        ],
        response_format: {
          type: 'json_schema',
          json_schema: {
            name: 'weather',
            strict: true,
            schema: {
              type: 'object',
              properties: {
                location: {
                  type: 'string',
                  description: 'City or location name',
                },
                temperature: {
                  type: 'number',
                  description: 'Temperature in Celsius',
                },
                conditions: {
                  type: 'string',
                  description: 'Weather conditions description',
                },
              },
              required: ['location', 'temperature', 'conditions'],
              additionalProperties: false,
            },
          },
        },
      }),
    });

    const data = await response.json();
    const weatherInfo = data.choices[0].message.content;
    ```
  </CodeGroup>
</Template>

## Streaming with Structured Outputs


Output:
Chunk ID: c4c15f
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 814
Output:
***

title: Latency and Performance
subtitle: Understanding OpenRouter's performance characteristics
headline: Latency and Performance | Minimizing Gateway Latency
canonical-url: '[https://openrouter.ai/docs/guides/best-practices/latency-and-performance](https://openrouter.ai/docs/guides/best-practices/latency-and-performance)'
'og:site\_name': OpenRouter Documentation
'og:title': Latency and Performance | Minimizing Gateway Latency
'og:description': >-
Learn about OpenRouter's performance characteristics, latency optimizations,
and best practices for achieving optimal response times.
'og:image':
type: url
value: >-
[https://openrouter.ai/dynamic-og?title=Latency%20and%20Performance\&description=Understanding%20OpenRouter's%20performance%20characteristics](https://openrouter.ai/dynamic-og?title=Latency%20and%20Performance\&description=Understanding%20OpenRouter's%20performance%20characteristics)
'og:image:width': 1200
'og:image:height': 630
'twitter:card': summary\_large\_image
'twitter:site': '@OpenRouter'
noindex: false
nofollow: false
---------------

OpenRouter is designed with performance as a top priority. OpenRouter is heavily optimized to add as little latency as possible to your requests.

## Minimal Overhead

OpenRouter is designed to add minimal latency to your requests. This is achieved through:

* Edge computing using Cloudflare Workers to stay as close as possible to your application
* Efficient caching of user and API key data at the edge
* Optimized routing logic that minimizes processing time

## Performance Considerations

### Cache Warming

When OpenRouter's edge caches are cold (typically during the first 1-2 minutes of operation in a new region), you may experience slightly higher latency as the caches warm up. This normalizes once the caches are populated.

### Credit Balance Checks

To maintain accurate billing and prevent overages, OpenRouter performs additional database checks when:

* A user's credit balance is low (single digit dollars)
* An API key is approaching its configured credit limit

OpenRouter expires caches more aggressively under these conditions to ensure proper billing, which increases latency until additional credits are added.

### Model Fallback

When using [model routing](/docs/routing/auto-model-selection) or [provider routing](/docs/features/provider-routing), if the primary model or provider fails, OpenRouter will automatically try the next option. A failed initial completion unsurprisingly adds latency to the specific request. OpenRouter tracks provider failures, and will attempt to intelligently route around unavailable providers so that this latency is not incurred on every request.

## Best Practices

To achieve optimal performance with OpenRouter:

1. **Maintain Healthy Credit Balance**
   * Set up auto-topup with a higher threshold and amount
   * This helps avoid forced credit checks and reduces the risk of hitting zero balance
   * Recommended minimum balance: \$10-20 to ensure smooth operation

2. **Use Provider Preferences**
   * If you have specific latency requirements (whether time to first token, or time to last), there are [provider routing](/docs/features/provider-routing) features to help you achieve your performance and cost goals.


Output:
Chunk ID: 978f4f
Wall time: 0.0505 seconds
Process exited with code 0
Original token count: 2618
Output:
***

title: Prompt Caching
subtitle: Cache prompt messages
headline: Prompt Caching | Reduce AI Model Costs with OpenRouter
canonical-url: '[https://openrouter.ai/docs/guides/best-practices/prompt-caching](https://openrouter.ai/docs/guides/best-practices/prompt-caching)'
'og:site\_name': OpenRouter Documentation
'og:title': Prompt Caching - Optimize AI Model Costs with Smart Caching
'og:description': >-
Reduce your AI model costs with OpenRouter's prompt caching feature. Learn how
to cache and reuse responses across OpenAI, Anthropic Claude, and DeepSeek
models.
'og:image':
type: url
value: >-
[https://openrouter.ai/dynamic-og?title=Prompt%20Caching\&description=Optimize%20AI%20model%20costs%20with%20OpenRouter](https://openrouter.ai/dynamic-og?title=Prompt%20Caching\&description=Optimize%20AI%20model%20costs%20with%20OpenRouter)
'og:image:width': 1200
'og:image:height': 630
'twitter:card': summary\_large\_image
'twitter:site': '@OpenRouter'
noindex: false
nofollow: false
---------------

To save on inference costs, you can enable prompt caching on supported providers and models.

Most providers automatically enable prompt caching, but note that some (see Anthropic below) require you to enable it on a per-message basis.

When using caching (whether automatically in supported models, or via the `cache_control` property), OpenRouter uses provider sticky routing to maximize cache hits — see [Provider Sticky Routing](#provider-sticky-routing) below for details.

## Provider Sticky Routing

To maximize cache hit rates, OpenRouter uses **provider sticky routing** to route your subsequent requests to the same provider endpoint after a cached request. This works automatically with both implicit caching (e.g. OpenAI, DeepSeek, Gemini 2.5) and explicit caching (e.g. Anthropic `cache_control` breakpoints).

**How it works:**

* After a request that uses prompt caching, OpenRouter remembers which provider served your request.
* Subsequent requests for the same model are routed to the same provider, keeping your cache warm.
* Sticky routing only activates when the provider's cache read pricing is cheaper than regular prompt pricing, ensuring you always benefit from cost savings.
* If the sticky provider becomes unavailable, OpenRouter automatically falls back to the next-best provider.
* Sticky routing is not used when you specify a manual [provider order](/docs/api-reference/provider-preferences) via `provider.order` — in that case, your explicit ordering takes priority.

**Sticky routing granularity:**

Sticky routing is tracked at the account level, per model, and per conversation. OpenRouter identifies conversations by hashing the first system (or developer) message and the first non-system message in each request, so requests that share the same opening messages are routed to the same provider. This means different conversations naturally stick to different providers, improving load-balancing and throughput while keeping caches warm within each conversation.

## Inspecting cache usage

To see how much caching saved on each generation, you can:

1. Click the detail button on the [Activity](/activity) page
2. Use the `/api/v1/generation` API, [documented here](/docs/api/api-reference/generations/get-generation)
3. Check the `prompt_tokens_details` object in the [usage response](/docs/guides/guides/usage-accounting) included with every API response

The `cache_discount` field in the response body will tell you how much the response saved on cache usage. Some providers, like Anthropic, will have a negative discount on cache writes, but a positive discount (which reduces total cost) on cache reads.

### Usage object fields

The usage object in API responses includes detailed cache metrics in the `prompt_tokens_details` field:

```json
{
  "usage": {
    "prompt_tokens": 10339,
    "completion_tokens": 60,
    "total_tokens": 10399,
    "prompt_tokens_details": {
      "cached_tokens": 10318,
      "cache_write_tokens": 0
    }
  }
}
```

The key fields are:

* `cached_tokens`: Number of tokens read from the cache (cache hit). When this is greater than zero, you're benefiting from cached content.
* `cache_write_tokens`: Number of tokens written to the cache. This appears on the first request when establishing a new cache entry.

## OpenAI

Caching price changes:

* **Cache writes**: no cost
* **Cache reads**: (depending on the model) charged at 0.25x or 0.50x the price of the original input pricing

[Click here to view OpenAI's cache pricing per model.](https://platform.openai.com/docs/pricing)

Prompt caching with OpenAI is automated and does not require any additional configuration. There is a minimum prompt size of 1024 tokens.

[Click here to read more about OpenAI prompt caching and its limitation.](https://platform.openai.com/docs/guides/prompt-caching)

## Grok

Caching price changes:

* **Cache writes**: no cost
* **Cache reads**: charged at {GROK_CACHE_READ_MULTIPLIER}x the price of the original input pricing

[Click here to view Grok's cache pricing per model.](https://docs.x.ai/docs/models#models-and-pricing)

Prompt caching with Grok is automated and does not require any additional configuration.

## Moonshot AI

Caching price changes:

* **Cache writes**: no cost
* **Cache reads**: charged at {MOONSHOT_CACHE_READ_MULTIPLIER}x the price of the original input pricing

Prompt caching with Moonshot AI is automated and does not require any additional configuration.

## Groq

Caching price changes:

* **Cache writes**: no cost
* **Cache reads**: charged at {GROQ_CACHE_READ_MULTIPLIER}x the price of the original input pricing

Prompt caching with Groq is automated and does not require any additional configuration. Currently available on Kimi K2 models.

[Click here to view Groq's documentation.](https://console.groq.com/docs/prompt-caching)

## Anthropic Claude

Caching price changes:

* **Cache writes (5-minute TTL)**: charged at {ANTHROPIC_CACHE_WRITE_MULTIPLIER}x the price of the original input pricing
* **Cache writes (1-hour TTL)**: charged at 2x the price of the original input pricing
* **Cache reads**: charged at {ANTHROPIC_CACHE_READ_MULTIPLIER}x the price of the original input pricing

Prompt caching with Anthropic requires the use of `cache_control` breakpoints. There is a limit of four breakpoints. By default, the cache expires after 5 minutes, but you can extend this to 1 hour by specifying `"ttl": "1h"` in the `cache_control` object. It is recommended to reserve the cache breakpoints for large bodies of text, such as character cards, CSV data, RAG data, book chapters, etc.

[Click here to read more about Anthropic prompt caching and its limitation.](https://platform.claude.com/docs/en/build-with-claude/prompt-caching)

The `cache_control` breakpoint can only be inserted into the text part of a multipart message.

### Supported models

The following Claude models support prompt caching:

* Claude Opus 4.5
* Claude Opus 4.1
* Claude Opus 4
* Claude Sonnet 4.5
* Claude Sonnet 4
* Claude Haiku 4.5
* Claude Haiku 3.5

### Minimum token requirements

Each model has a minimum cacheable prompt length:

* **4096 tokens**: Claude Opus 4.5, Claude Haiku 4.5
* **1024 tokens**: Claude Opus 4.1, Claude Opus 4, Claude Sonnet 4.5, Claude Sonnet 4

Prompts shorter than these minimums will not be cached.

### Cache TTL Options

OpenRouter supports two cache TTL values for Anthropic:

* **5 minutes** (default): `"cache_control": { "type": "ephemeral" }`
* **1 hour**: `"cache_control": { "type": "ephemeral", "ttl": "1h" }`

The 1-hour TTL is useful for longer sessions where you want to maintain cached content across multiple requests without incurring repeated cache write costs. The 1-hour TTL costs more for cache writes (2x base input price vs 1.25x for 5-minute TTL) but can save money over extended sessions by avoiding repeated cache writes. The 1-hour TTL is supported across all Claude 4.5 model providers (Anthropic, Amazon Bedrock, and Google Vertex AI).

### Examples

System message caching example (default 5-minute TTL):

```json
{
  "messages": [
    {
      "role": "system",
      "content": [
        {
          "type": "text",
          "text": "You are a historian studying the fall of the Roman Empire. You know the following book very well:"
        },
        {
          "type": "text",
          "text": "HUGE TEXT BODY",
          "cache_control": {
            "type": "ephemeral"
          }
        }
      ]
    },
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "What triggered the collapse?"
        }
      ]
    }
  ]
}
```

User message caching example with 1-hour TTL:

```json
{
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Given the book below:"
        },
        {
          "type": "text",
          "text": "HUGE TEXT BODY",
          "cache_control": {
            "type": "ephemeral",
            "ttl": "1h"
          }
        },
        {
          "type": "text",
          "text": "Name all the characters in the above book"
        }
      ]
    }
  ]
}
```

## DeepSeek

Caching price changes:

* **Cache writes**: charged at the same price as the original input pricing
* **Cache reads**: charged at {DEEPSEEK_CACHE_READ_MULTIPLIER}x the price of the original input pricing

Prompt caching with DeepSeek is automated and does not require any additional configuration.

## Google Gemini

### Implicit Caching

Gemini 2.5 Pro and 2.5 Flash models now support **implicit caching**, providing automatic caching functionality similar to OpenAI’s automatic caching. Implicit caching works seamlessly — no manual setup or additional `cache_control` breakpoints required.

Pricing Changes:

* No cache write or storage costs.
* Cached tokens are charged at {GOOGLE_CACHE_READ_MULTIPLIER}x the original input token cost.

Note that the TTL is on average 3-5 minutes, but will vary. There is a minimum of {GOOGLE_CACHE_MIN_TOKENS_2_5_FLASH} tokens for Gemini 2.5 Flash, and {GOOGLE_CACHE_MIN_TOKENS_2_5_PRO} tokens for Gemini 2.5 Pro for requests to be eligible for caching.

[Official announcement from Google](https://developers.googleblog.com/en/gemini-2-5-models-now-support-implicit-caching/)

<Tip>
  To maximize implicit cache hits, keep the initial portion of your message
  arrays consistent between requests. Push variations (such as user questions or


Summary: auto


**Planning web searches for official docs**


I’m going to verify live docs now (Google + OpenRouter) for anything time-sensitive like model IDs, batch behavior, and quota semantics, so the plan doesn’t rely on stale assumptions.


Question: Google Gemini API Gemini 3 Flash rate limits RPM TPM 2026

Google Gemini Batch API turnaround and pricing 50% 2026

Vertex AI Gemini 3.1 Flash quotas batch tokens in flight

OpenRouter structured outputs json_schema strict true


**Scanning for model details**


**Planning extraction of model details**


**Identifying batch API behavior**


**Planning temperature research**


Question: Gemini 3 prompting guide temperature keep default 1.0

Gemini API temperature parameter deterministic transcription


**Planning source referencing approach**


**Searching temperature and migration lines**


**researching transcription best practices**


Question: Gemini API audio understanding transcription prompt best practices

Gemini API transcribe audio file with Gemini 3 Flash

Vertex AI Gemini 3 Flash audio transcription

Google Gemini 3 Flash audio input duration limits


I verified live audio/transcription docs too. I’m now extracting exact operational constraints (audio tokenization, file-size boundaries, and batch/queue behavior) so your throughput and cost plan can be numerically grounded.


**Analyzing audio doc patterns**


Gemini represents each second of audio as 32 tokens


**Exploring model detail section**


**Planning OpenRouter fallback research**


Question: OpenRouter provider routing require_parameters true fallback order

OpenRouter models google/gemini-3-flash pricing

OpenRouter retry-after rate limit headers


**Searching provider preference docs**


Question: OpenRouter provider routing documentation order allow_fallbacks

openrouter provider preferences require_parameters

openrouter chat completions provider field


**Searching for model ID**


Question: OpenRouter google/gemini-3-flash model id


**Filtering search results**


I’ve pulled OpenRouter routing/limits docs and I’ll now lock down the exact fallback knobs (`provider.order`, `allow_fallbacks`, `require_parameters`) plus model slug details so the fallback design is operationally precise.


**Searching for rate limit and model identifiers**


**Gathering Indic forced alignment resources**


Question: Montreal Forced Aligner pretrained models Hindi Bengali Tamil

WhisperX forced alignment multilingual language list

AI4Bharat IndicASR models huggingface wav2vec2


**Querying MFA supported languages**


Question: Montreal Forced Aligner pretrained acoustic models language list

Montreal Forced Aligner Hindi dictionary


**Assessing MFA models sources**


Question: site:mfa-models.readthedocs.io hindi_mfa acoustic model


**Checking Hindi acoustic models availability**


Question: site:mfa-models.readthedocs.io Hindi acoustic model

site:mfa-models.readthedocs.io Marathi acoustic model

site:mfa-models.readthedocs.io Telugu acoustic model


**Testing direct URL access**


**Checking available Indic languages**


**Checking G2P language list**


**Prioritizing local resources over web calls**


I’ve gathered enough external constraints to avoid stale assumptions on limits/pricing. Next I’ll inspect local env/config shape (without exposing secrets) so the plan matches your existing runtime setup for AI Studio/Vertex/OpenRouter/Supabase/R2.


Output:
Chunk ID: f73d87
Wall time: 0.0516 seconds
Process exited with code 0
Original token count: 55
Output:
ACCOUNT_ID
DATABASE
DATABASE_URL
GEMINI_KEY
HF_TOKEN
HOST
OPENROUTER_API_KEY
PORT
PROJECT_ID
R2_ACCESS_KEY_ID
R2_BUCKET
R2_ENDPOINT_URL
R2_SECRET_ACCESS_KEY
S3_API
SUPABASE_ADMIN
SUPABASE_ANON
SUPABASE_TABLES
URL
USER


Summary: auto


**Reviewing existing prompt documentation**


Output:
Chunk ID: 3630ec
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 855
Output:
Instruction: You are a strict, verbatim transcription engine for Indian languages
Input: Audio file attached
Primary audio language: [ SPECIFY HERE]
Task:
Listen to the audio carefully
Identify the primary spoken language as specified above
Transcribe the speech exactly as spoken and produce four separate outputs as defined below
Sometimes you won't be clear about complete transcription but you know parts of it. From that you can fix the grammar. That's how you can create the best transcription.
You know the first word and the last word. You can create the best transcription. You know the middle word and the last word. You can create the best transcription.
You need to think in terms of what you know and how you can go to the end with 100% accuracy with predicting what comes here, what would be it, what is the right fit and attaching it all together and doing the combinations

STRICT GLOBAL RULES (apply to all outputs):
Verbatim only Include all repetitions filler words stammers false starts hesitations and colloquial expressions exactly as spoken
No normalization Do not correct grammar do not fix pronunciation do not standardize spellings and do not clean up dialect accent or mixed language usage
No inference Do not add meaning structure or emphasis that is not clearly audible
Script fidelity Follow the rules of each output strictly Do not translate paraphrase or summarize
OUTPUT 1: Proper native transcription
Write everything only in the native script of the primary language
Do not use any punctuation marks at all
Even if English words are spoken write them phonetically using the native script of the primary language
Preserve repetitions fillers and stammers exactly
OUTPUT 2: Native transcription with punctuation
Write everything only in the native script of the primary language
Use minimal punctuation strictly based on speech flow
Use the sentence ending symbol appropriate to the script of the primary language
Use commas only for clearly audible pauses
Use question marks only when question intonation is clearly audible
Use exclamation marks only when strong emphasis is clearly audible
Do not use ellipses quotation marks brackets colons semicolons or multiple punctuation marks together
Do not correct grammar or restructure sentences
OUTPUT 3: Code switch version
Preserve the original language switching exactly as spoken
Write Indian language parts in their native script
Write English words and phrases in English Latin script if spoken that way
Do not translate normalize or rewrite across languages
Use minimal punctuation based only on audible cues following the primary language punctuation convention
OUTPUT 4: Roman English transcription
Write the entire transcription only in Roman English
Convert all Indian language speech into Romanized form as it sounds
Preserve pronunciation exactly as spoken even if spelling looks incorrect
Preserve all repetitions fillers stammers and false starts
Do not standardize spellings or make them grammatically correct
Use minimal punctuation only if clearly audible otherwise avoid punctuation
OUTPUT FORMAT RULES:
Clearly label the four outputs as
Output 1: Proper native transcription
Output 2: Native transcription with punctuation
Output 3: Code switch version
Output 4: Roman English transcription
Under each label provide only the transcription text
Do not add explanations notes or any extra English text beyond the labels


**Preparing plan questions**


I’ve now grounded both local and live docs; the main remaining unknowns are preference tradeoffs that materially change implementation. I’ll lock those with 3 short choices, then I can produce a decision-complete plan that covers every concern as explicit TODOs.


Question: For the authoritative `transcription` field, which format should we standardize on?

Choices:
Best acoustic fidelity: each spoken language stays in its own script; no transliteration loss.
Force all words into the video’s tagged language script for consistency, but accuracy can drop on mixed speech.
Everything in Latin script for downstream uniformity, but loses native-script fidelity and ambiguity increases.

Question: How aggressive should boundary/segment filtering be before Gemini transcription?

Choices:
Aggressively trim/drop questionable segments to maximize transcript trustworthiness, accepting lower retained hours.
Keep more data with moderate cleanup and rely more on post-validation to catch errors.
Retain maximum audio and do minimal trimming, accepting higher transcript noise.

Question: What OpenRouter spend policy should the scheduler enforce in the 4-day run?

Choices:
Strict cap + emergency only (Recommended)
Use OpenRouter only when Gemini lanes are saturated and ETA risk breaches target, with hard budget cap.
Allow routine overflow to OpenRouter to protect timeline, with softer budget controls.
Use OpenRouter freely to maximize throughput and back-calculate spend later.


Output: {"answers":{"quality_gate_mode":{"answers":["Precision-first (Recommended)"]},"transcript_form":{"answers":["Code-mixed native scripts (Recommended)"]},"openrouter_budget":{"answers":["Strict cap + emergency only (Recommended)"]}}}


**Finalizing plan commentary**


No listed skill is applicable for this turn (`skill-creator`/`skill-installer` are unrelated to transcript pipeline planning), so I’m proceeding with a direct architecture/spec plan.


**Composing comprehensive final plan**


Question: NVIDIA NeMo Forced Aligner CTC docs

AI4Bharat IndicConformer ASR HuggingFace 11 Indian languages


**Exploring transliteration options**


Question: Indic NLP library transliteration python docs

AI4Bharat indic-trans transliteration model


**Outlining comprehensive phased plan**


**Finalizing detailed execution and scheduling**


# 4-Day Decision-Complete Plan: Gemini Transcript Pipeline for 80M Segments

## Summary
1. Objective: produce high-fidelity multilingual transcripts from R2-hosted podcast segments for ASR/TTS training, with deterministic JSON outputs, quality scoring, and deadline-safe throughput.
2. Locked preferences from this turn: authoritative transcript will be code-mixed native scripts; preprocessing will be precision-first; OpenRouter is emergency-only with strict cap.
3. Deadline anchor: start on **February 28, 2026** and finish within **100 hours** (target completion by **March 4, 2026**).
4. Core strategy: precision-first segment polishing -> strict Gemini structured transcription -> automated quality scoring -> retry only low-confidence tail.
5. Throughput strategy: dual Gemini lanes (Vertex Batch + AI Studio realtime), OpenRouter activated only if ETA risk breaches deadline.

## Concern-to-TODO Coverage (Every Concern Addressed)
1. `TODO-01` R2 source format (`videoID.tar` with `metadata.json` + `segments/*.flac`): define a deterministic ingestion manifest keyed by `video_id`, `segment_id`, source offsets, and language hint from Supabase.
2. `TODO-02` Imperfect diarization and acknowledgment leakage: add speaker-consistency and overlap heuristics; contaminated segments are flagged and either dropped or low-priority retried.
3. `TODO-03` Abrupt starts/ends from prior VAD: run boundary optimizer to trim to nearest low-energy/VAD valleys and enforce silence padding policy.
4. `TODO-04` Support exactly 12 languages: enforce a closed language enum for hints/detection and language-specific script checks for output validation.
5. `TODO-05` Supabase language metadata usage: treat as soft hint only, never as hard transcription constraint.
6. `TODO-06` Prompt review and Gemini behavior: keep strict anti-hallucination/verbatim rules, tighten derivation rules between `transcription` and `tagged`, and reduce ambiguity on language mismatch.
7. `TODO-07` Temperature/model settings: run at `temperature=0`, `candidate_count=1`, `thinking_level=minimal`, strict JSON schema; this prioritizes determinism for transcription tasks.
8. `TODO-08` Language mismatch handling: keep expected-language hint in prompt, require `detected_language`, and accept mismatch when audio dominance differs from metadata.
9. `TODO-09` Script policy decision: authoritative `transcription` remains code-mixed with each spoken language in its own script (no forced transliteration).
10. `TODO-10` Punctuation requirement: include punctuation in authoritative output but only constrained to audible prosody (`. , ? !`) to avoid grammar hallucinations.
11. `TODO-11` Audio event tagging experiment: keep fixed controlled tag set with confidence gating and deterministic relation to `transcription`.
12. `TODO-12` Event list standardization: use exactly 10 tags: `[laugh] [cough] [sigh] [breath] [throat_clear] [sniff] [music] [applause] [noise] [speech_overlap]`.
13. `TODO-13` Speaker metadata fields: keep `emotion`, `speaking_style`, `pace`; keep `accent` as required string with empty allowed when not confident.
14. `TODO-14` Segment duration policy: drop `<2s`; target `2–10s`; permit extension to `15s` only when waiting for cleaner split point.
15. `TODO-15` Validation/scoring architecture: build multi-signal quality score (schema, script, language match, text/audio ratio, alignment/consensus).
16. `TODO-16` Indic aligner question: do not depend on MFA as primary validator for all 12 languages; use CTC-based multilingual alignment/consensus path instead.
17. `TODO-17` Romanization validation question: make romanized comparison optional via transliteration layer for metrics only; do not make romanization the primary transcript.
18. `TODO-18` Rate-limit and throughput planning: scheduler must enforce RPM/TPM token buckets and monitor ETA continuously.
19. `TODO-19` Provider order and fallback: primary Gemini lanes first; OpenRouter only for deadline protection and only within strict budget cap.
20. `TODO-20` “Don’t get stuck on validation”: ship Validation V1 fast gates first; phase Validation V2 (deeper alignments) without blocking pipeline launch.

## Public Interfaces and Type Changes
1. Prompt template becomes parameterized with `expected_language`, `language_family`, and fixed tag inventory.
2. Response schema update to `TranscriptionSchemaV2`:
```json
{
  "type": "object",
  "properties": {
    "transcription": { "type": "string" },
    "tagged": { "type": "string" },
    "speaker": {
      "type": "object",
      "properties": {
        "emotion": { "type": "string", "enum": ["neutral","happy","sad","angry","excited","surprised"] },
        "speaking_style": { "type": "string", "enum": ["conversational","narrative","excited","calm","emphatic","sarcastic","formal"] },
        "pace": { "type": "string", "enum": ["slow","normal","fast"] },
        "accent": { "type": "string" }
      },
      "required": ["emotion","speaking_style","pace","accent"],
      "additionalProperties": false
    },
    "detected_language": {
      "type": "string",
      "enum": ["hi","mr","te","ta","kn","ml","gu","pa","bn","as","or","en","mixed","unknown"]
    },
    "language_confidence": {
      "type": "string",
      "enum": ["high","medium","low"]
    }
  },
  "required": ["transcription","tagged","speaker","detected_language","language_confidence"],
  "additionalProperties": false
}
```
3. Storage contract adds:
   `segment_manifest`, `provider_attempts`, `raw_response`, `normalized_text`, `quality_score`, `quality_flags`, `retry_reason`, `final_status`.
4. Determinism invariant: stripping event tags from `tagged` must exactly equal `transcription` after whitespace normalization.
5. Compatibility rule: version outputs with `schema_version` so old rows remain queryable.

## Pipeline Design (Implementation Plan)
1. Ingestion stage:
   Build manifest from R2 tar members and Supabase metadata, producing one row per segment with expected language and source path.
2. Boundary polishing stage:
   Decode audio to mono 16k, run VAD/energy boundary checks, trim to clean boundaries, add fixed silence pad (`80–120ms`) where feasible, reject segments that remain hard-cut mid-speech.
3. Duration normalization stage:
   Keep `2–10s` preferred windows; for long segments choose nearest non-speech split in `7–10s`; extend search to `15s`; if still no acceptable split, reject for precision-first mode.
4. Transcription request stage:
   Send polished segment + strict prompt + schema to Gemini with deterministic config; include expected language hint but enforce “hear-first” rule.
5. Provider routing stage:
   Route by quota-aware scheduler across Vertex Batch and AI Studio; keep OpenRouter disabled unless ETA risk and quota saturation conditions are met.
6. Validation V1 stage:
   Run fast validators (schema pass, script consistency, punctuation limits, tag consistency, text/audio duration sanity, language hint mismatch severity).
7. Retry and escalation stage:
   Retry only failed/low-score items with bounded attempts and modified strategy (no random retries); quarantine unresolved samples.
8. Validation V2 stage (non-blocking):
   Add multilingual CTC alignment/consensus scoring for flagged subset and periodic quality audits.
9. Persist and publish stage:
   Write final accepted records and quality metrics; keep rejected/quarantined queue for later remediation cycle.

## Throughput, Scheduling, and Deadline Control
1. Required pace for 80M in 100h: `800,000 segments/hour` (`13,333 segments/min`, `222 segments/sec`).
2. AI Studio ceiling provided by you: `20k RPM` and `20M TPM`; scheduler runs at 80% safety target by default, then ramps when stable.
3. Token feasibility gate:
   Measure real per-request token usage on calibration set; if sustained token cost exceeds plan budget, reduce prompt tokens and/or enable larger Vertex Batch share.
4. Lane policy:
   Vertex Batch handles bulk backlog first; AI Studio handles continuous realtime stream and retries.
5. OpenRouter emergency policy:
   Activate only when projected ETA misses deadline and Gemini lanes are quota-bound for sustained window; enforce hard spend cap (default: max 8% of projected run cost).
6. Control loop:
   Recompute ETA every 5 minutes from observed throughput, retry rate, and remaining queue; provider weights auto-adjust to hit deadline.

## Prompt and Behavior Spec
1. Keep your strict anti-translation and anti-hallucination core.
2. Explicitly state that expected language is hint-only and `detected_language` must reflect actual dominant speech.
3. Keep code-mixed native scripts in `transcription`; never transliterate by default.
4. Keep punctuation limited to `. , ? !` from audible prosody only.
5. Keep unknown handling strict: `[UNK]`, `[INAUDIBLE]`, `[NO_SPEECH]`.
6. Keep `tagged` as a derived copy of `transcription` plus only allowed tags, with no word edits.
7. Add confidence restraint for tags and accent: omit/downgrade when uncertain rather than guessing.

## Validation and Scoring Plan
1. Quality score range: `0.00–1.00`.
2. V1 weighted components:
   schema validity, `tagged/transcription` consistency, script-language consistency, text-length vs audio-duration ratio, punctuation sanity, mismatch severity.
3. V2 components:
   ASR consensus and CTC forced-alignment confidence for flagged samples.
4. Suggested thresholds:
   accept `>=0.85`, retry `0.65–0.85`, quarantine `<0.65`.
5. Romanization for validation only:
   optional transliteration to common script for CER/WER comparison across scripts; canonical stored transcript remains native script code-mixed.

## Test Cases and Acceptance Criteria
1. Multilingual code-mix test across all 12 supported languages with English insertions.
2. Boundary edge tests:
   hard-cut start, hard-cut end, both hard-cut, clean-silence start/end.
3. Duration policy tests:
   `<2s` drop, `2–10s` pass, `>10s` split, no split until `15s`.
4. No-speech and noise-only tests returning `[NO_SPEECH]`.
5. Event-tag tests ensuring only allowed tags and deterministic derivation from transcription.
6. Speaker metadata tests for required enums and empty-accent handling.
7. Language mismatch tests where Supabase hint differs from spoken dominant language.
8. JSON schema strictness tests with reject-on-extra-fields.
9. Retry policy tests with bounded attempts and proper quarantine behavior.
10. Load test proving sustained throughput needed for deadline.
11. Acceptance targets:
    JSON validity `>=99.9%`, deterministic `tagged` derivation `>=99.5%`, deadline ETA always within remaining wall-clock after first 12h burn-in.

## Rollout Plan (100 Hours)
1. Hours 0–8:
   finalize prompt/schema v2, build calibration set, run token+quality baseline.
2. Hours 8–24:
   deploy ingestion + boundary polishing + transcription core; run pilot at scale and tune thresholds.
3. Hours 24–84:
   full production run with adaptive scheduler, continuous monitoring, and controlled retries.
4. Hours 84–96:
   reprocess low-score tail and clear retry backlog.
5. Hours 96–100:
   finalize dataset snapshots, metrics report, and quarantine list for later cleanup.

## Assumptions and Defaults
1. Code-mixed native script is authoritative (`transcription`).
2. Precision-first retention is preferred over maximizing kept audio.
3. OpenRouter is emergency-only with strict cap.
4. Prompt-level determinism (`temperature=0`) is accepted for this use case.
5. Accent can be empty string if uncertain.
6. Validation V1 is required for launch; V2 is iterative and must not block primary run.
7. Any segment that still appears mid-sentence/hard-cut after polishing is rejected rather than guessed.

## Source References
1. Gemini model + features: https://ai.google.dev/gemini-api/docs/models/gemini
2. Gemini Batch API: https://ai.google.dev/gemini-api/docs/batch-api
3. Gemini structured outputs: https://ai.google.dev/gemini-api/docs/structured-output
4. Gemini audio tokenization and limits: https://ai.google.dev/gemini-api/docs/audio
5. Vertex Gemini quotas and batch behavior: https://cloud.google.com/vertex-ai/generative-ai/docs/quotas
6. OpenRouter structured outputs: https://openrouter.ai/docs/guides/features/structured-outputs
7. OpenRouter provider routing/preferences: https://openrouter.ai/docs/features/provider-routing
8. OpenRouter rate limits: https://openrouter.ai/docs/api-reference/limits
9. NeMo Forced Aligner (CTC-based): https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/tools/nemo_forced_aligner.html
10. AI4Bharat Indic Conformer multilingual ASR: https://huggingface.co/ai4bharat/indic-conformer-600m-multilingual
11. MFA acoustic models coverage: https://mfa-models.readthedocs.io/en/latest/acoustic/index.html
12. MFA dictionaries coverage: https://mfa-models.readthedocs.io/en/latest/dictionary/index.html
13. IndicXlit transliteration: https://github.com/AI4Bharat/IndicXlit
14. Current prompt under review: [prompt.txt](/home/ubuntu/transcripts/prompt.txt)