This example illustrates how to interact with the Gemini API using the OpenAI JS library.
This notebook will walk you through:
More details about this OpenAI compatibility on the documentation.
Install the Google GenAI SDK from npm.
$ npm install @google/genaiYou can create your API key using Google AI Studio with a single click.
Remember to treat your API key like a password. Donโt accidentally save it in a notebook or source file you later commit to GitHub. In this notebook we will be storing the API key in a .env file. You can also set it as an environment variable or use a secret manager.
Hereโs how to set it up in a .env file:
$ touch .env
$ echo "GEMINI_API_KEY=<YOUR_API_KEY>" >> .envAnother option is to set the API key as an environment variable. You can do this in your terminal with the following command:
$ export GEMINI_API_KEY="<YOUR_API_KEY>"To load the API key from the .env file, we will use the dotenv package. This package loads environment variables from a .env file into process.env.
$ npm install dotenvThen, we can load the API key in our code:
const dotenv = require("dotenv") as typeof import("dotenv");
dotenv.config({
path: "../.env",
});
const GEMINI_API_KEY = process.env.GEMINI_API_KEY ?? "";
if (!GEMINI_API_KEY) {
throw new Error("GEMINI_API_KEY is not set in the environment variables");
}
console.log("GEMINI_API_KEY is set in the environment variables");GEMINI_API_KEY is set in the environment variablesIn our particular case the .env is is one directory up from the notebook, hence we need to use ../ to go up one directory. If the .env file is in the same directory as the notebook, you can omit it altogether.
โ
โโโ .env
โโโ quickstarts
โโโ Get_started_OpenAI_Compatibility.ipynbWith the new SDK, now you only need to initialize a client with you API key (or OAuth if using Vertex AI). The model is now set in each call.
const openai = require("openai") as typeof import("openai");
const ai = new openai.OpenAI({
apiKey: GEMINI_API_KEY,
baseURL: "https://generativelanguage.googleapis.com/v1beta/openai/",
});You can start by listing the available models using the OpenAI library.
for await (const model of await ai.models.list()) {
console.log(`Model ID: ${model.id}`);
}Model ID: models/embedding-gecko-001
Model ID: models/gemini-1.0-pro-vision-latest
Model ID: models/gemini-pro-vision
Model ID: models/gemini-1.5-pro-latest
Model ID: models/gemini-1.5-pro-002
Model ID: models/gemini-1.5-pro
Model ID: models/gemini-1.5-flash-latest
Model ID: models/gemini-1.5-flash
Model ID: models/gemini-1.5-flash-002
Model ID: models/gemini-1.5-flash-8b
Model ID: models/gemini-1.5-flash-8b-001
Model ID: models/gemini-1.5-flash-8b-latest
Model ID: models/gemini-2.5-pro-preview-03-25
Model ID: models/gemini-2.5-flash-preview-04-17
Model ID: models/gemini-2.5-flash-preview-05-20
Model ID: models/gemini-2.5-flash
Model ID: models/gemini-2.5-flash-preview-04-17-thinking
Model ID: models/gemini-2.5-flash-lite-preview-06-17
Model ID: models/gemini-2.5-pro-preview-05-06
Model ID: models/gemini-2.5-pro-preview-06-05
Model ID: models/gemini-2.5-pro
Model ID: models/gemini-2.0-flash-exp
Model ID: models/gemini-2.0-flash
Model ID: models/gemini-2.0-flash-001
Model ID: models/gemini-2.0-flash-exp-image-generation
Model ID: models/gemini-2.0-flash-lite-001
Model ID: models/gemini-2.0-flash-lite
Model ID: models/gemini-2.0-flash-preview-image-generation
Model ID: models/gemini-2.0-flash-lite-preview-02-05
Model ID: models/gemini-2.0-flash-lite-preview
Model ID: models/gemini-2.0-pro-exp
Model ID: models/gemini-2.0-pro-exp-02-05
Model ID: models/gemini-exp-1206
Model ID: models/gemini-2.0-flash-thinking-exp-01-21
Model ID: models/gemini-2.0-flash-thinking-exp
Model ID: models/gemini-2.0-flash-thinking-exp-1219
Model ID: models/gemini-2.5-flash-preview-tts
Model ID: models/gemini-2.5-pro-preview-tts
Model ID: models/learnlm-2.0-flash-experimental
Model ID: models/gemma-3-1b-it
Model ID: models/gemma-3-4b-it
Model ID: models/gemma-3-12b-it
Model ID: models/gemma-3-27b-it
Model ID: models/gemma-3n-e4b-it
Model ID: models/gemma-3n-e2b-it
Model ID: models/embedding-001
Model ID: models/text-embedding-004
Model ID: models/gemini-embedding-exp-03-07
Model ID: models/gemini-embedding-exp
Model ID: models/aqa
Model ID: models/imagen-3.0-generate-002
Model ID: models/imagen-4.0-generate-preview-06-06
Model ID: models/imagen-4.0-ultra-generate-preview-06-06
Model ID: models/veo-2.0-generate-001
Model ID: models/gemini-2.5-flash-preview-native-audio-dialog
Model ID: models/gemini-2.5-flash-exp-native-audio-thinking-dialog
Model ID: models/gemini-2.0-flash-live-001
Model ID: models/gemini-live-2.5-flash-previewIn this example, you will use the gemini-2.5-flash-preview-05-20 model. For more details about the available models, check the Gemini models page from the Gemini API documentation.
const tslab = require("tslab") as typeof import("tslab");
const MODEL_ID = "gemini-2.5-flash-preview-05-20";For your first request, use the OpenAI SDK to perform text generation with a text prompt.
const textResponse = await ai.chat.completions.create({
model: MODEL_ID,
messages: [
{ role: "system", content: "You are a helpful assistant." },
{ role: "user", content: "What is generative AI?" },
],
});
tslab.display.markdown(textResponse.choices[0].message.content ?? "");Generative AI is a type of artificial intelligence that can create new, original content, rather than just analyzing or classifying existing data. Unlike traditional AI that might recognize patterns or make predictions based on input (like classifying an image as a โcatโ), generative AI can produce a brand-new image of a cat that has never existed before, write an original story, compose music, or even generate functional code.
Think of it as an AI that has become a โcreatorโ rather than just an โanalyzerโ or โdecider.โ
Generative AI is behind many of the impressive breakthroughs weโve seen recently:
While powerful, generative AI also presents challenges:
In summary, generative AI is a revolutionary leap in artificial intelligence, moving from analysis to creation, and fundamentally changing how we interact with and produce digital content.
You can work with the Gemini API to generate code for you.
const CODE_PROMPT = `
Write a C program that takes two IP addresses, representing the start and end of a range
(e.g., 192.168.1.1 and 192.168.1.254), as input arguments. The program should convert this
IP address range into the minimal set of CIDR notations that completely cover the given
range. The output should be a comma-separated list of CIDR blocks.
`;
const codeResponse = await ai.chat.completions.create({
model: MODEL_ID,
messages: [{ role: "user", content: CODE_PROMPT }],
});
tslab.display.markdown(codeResponse.choices[0].message.content ?? "");This C program takes two IP addresses (start and end of a range) as command-line arguments. It then calculates and prints the minimal set of CIDR notations that completely cover the given IP address range.
The algorithm works by iteratively finding the largest possible CIDR block that starts at the current IP address and fits entirely within the remaining part of the specified range.
Key Concepts:
ntohl() and back to network byte order for inet_ntoa() using htonl()./N CIDR block covers 2^(32-N) IP addresses./24 block (size 256) must start at an address where the last octet is 0. This means start_ip % block_size == 0, or bitwise (start_ip & (block_size - 1)) == 0.start_ip and does not exceed the end_ip of the original range. This ensures a minimal set.#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <arpa/inet.h> // For inet_addr, inet_ntoa, ntohl, htonl
#include <stdint.h> // For uint32_t, uint64_t
#include <stdbool.h> // For bool type
#include <limits.h> // For UINT32_MAX
// Function to convert an IP string to a host byte order uint32_t
static uint32_t ip_string_to_uint32(const char *ip_str) {
struct in_addr sa;
if (inet_pton(AF_INET, ip_str, &(sa)) != 1) {
fprintf(stderr, "Error: Invalid IP address format: %s\n", ip_str);
exit(EXIT_FAILURE);
}
return ntohl(sa.s_addr);
}
// Function to convert a host byte order uint32_t to an IP string
// Note: inet_ntoa uses a static buffer, so the return value is valid
// only until the next call. For simple printing, it's fine.
static const char* uint32_to_ip_string(uint32_t ip_val) {
struct in_addr sa;
sa.s_addr = htonl(ip_val);
return inet_ntoa(sa);
}
int main(int argc, char *argv[]) {
if (argc != 3) {
fprintf(stderr, "Usage: %s <start_ip> <end_ip>\n", argv[0]);
fprintf(stderr, "Example: %s 192.168.1.1 192.168.1.254\n", argv[0]);
return EXIT_FAILURE;
}
uint32_t start_ip_val = ip_string_to_uint32(argv[1]);
uint32_t end_ip_val = ip_string_to_uint32(argv[2]);
if (start_ip_val > end_ip_val) {
fprintf(stderr, "Error: Start IP (%s) cannot be greater than End IP (%s).\n", argv[1], argv[2]);
return EXIT_FAILURE;
}
uint32_t current_ip = start_ip_val;
bool first_cidr = true;
while (current_ip <= end_ip_val) {
int best_prefix_len = -1;
// Iterate from the smallest block (longest prefix /32) to largest block (shortest prefix /0)
// We are looking for the largest possible block that fits.
// So we iterate prefix_len from 32 down to 0. The first one we find is the largest.
for (int prefix_len = 32; prefix_len >= 0; prefix_len--) {
// Calculate the size of the block for this prefix length.
// Use uint64_t for block_size to avoid overflow when prefix_len is 0 (2^32)
uint64_t block_size = 1ULL << (32 - prefix_len);
// 1. Check if the current_ip is aligned for this prefix_len.
// For a block to start at current_ip with a /prefix_len, current_ip must be a multiple of block_size.
// This is equivalent to (current_ip & (block_size - 1)) == 0.
// This bitwise check works correctly even for prefix_len 0 (block_size 2^32),
// as (2^32 - 1) is 0xFFFFFFFF, so (current_ip & 0xFFFFFFFF) == 0 becomes current_ip == 0.
if ((current_ip & (block_size - 1)) != 0) {
continue; // Not aligned for this block size
}
// 2. Check if this potential block fits within the remaining target range.
// The last IP in this potential block is current_ip + block_size - 1.
// We need: (current_ip + block_size - 1) <= end_ip_val.
// To prevent overflow when calculating (current_ip + block_size - 1):
// Check if (block_size - 1) is too large for current_ip + (block_size - 1) to fit in uint32_t.
// This check (current_ip > UINT32_MAX - (block_size - 1)) is crucial.
if (block_size - 1 > UINT32_MAX - current_ip) { // check if current_ip + block_size - 1 would overflow
// If it would overflow, it definitely extends beyond end_ip_val (unless end_ip_val is also near UINT32_MAX)
// In such a case, this block can't be fully contained if max_ip is UINT32_MAX.
// It means the block extends beyond the maximum representable IP address.
// So, it cannot be contained within end_ip_val unless end_ip_val is UINT32_MAX
// AND the block starts at 0. A simpler check might be just
// if (current_ip + block_size -1 > end_ip_val)
// but that can overflow.
// Safer check: Does `current_ip + (block_size - 1)` extend beyond `end_ip_val`?
// This is equivalent to `block_size - 1 > end_ip_val - current_ip`.
// Use uint64_t to prevent overflow for `end_ip_val - current_ip`
if (block_size > (uint64_t)end_ip_val - current_ip + 1) { // block_size is too large for the remaining range
continue;
}
} else {
// Now we can safely calculate block_end as a uint32_t
uint32_t block_end = current_ip + (uint32_t)(block_size - 1);
if (block_end > end_ip_val) {
continue; // Block extends beyond the end of our target range
}
}
// If we reached here, this prefix_len represents a valid block that
// starts at current_ip and fits within the remaining range.
// Since we iterate prefix_len downwards, this is the largest such block.
best_prefix_len = prefix_len;
break; // Found the largest block, so break the inner loop
}
if (best_prefix_len == -1) {
// This should ideally not happen if current_ip <= end_ip_val,
// as a /32 block should always be possible.
fprintf(stderr, "Internal Error: Could not find a suitable CIDR block for IP %s\n", uint32_to_ip_string(current_ip));
return EXIT_FAILURE;
}
// Print the found CIDR block
if (!first_cidr) {
printf(",");
}
printf("%s/%d", uint32_to_ip_string(current_ip), best_prefix_len);
first_cidr = false;
// Advance current_ip to the IP address immediately after the processed block
uint64_t processed_block_size = 1ULL << (32 - best_prefix_len);
if (current_ip > UINT32_MAX - processed_block_size) {
// This means advancing current_ip would cause an overflow,
// implying we've covered addresses up to UINT32_MAX.
break;
}
current_ip += (uint32_t)processed_block_size;
}
printf("\n");
return EXIT_SUCCESS;
}How to Compile and Run:
ip_range_to_cidr.c (or any other .c extension).bash gcc ip_range_to_cidr.c -o ip_range_to_cidr (You might need to include -Wall -Wextra -std=c11 for better warnings and C11 standard compliance, though not strictly necessary for this code to compile.)bash ./ip_range_to_cidr 192.168.1.1 192.168.1.254Example Outputs:
Input: 192.168.1.1 192.168.1.254
Output: 192.168.1.1/32,192.168.1.2/31,192.168.1.4/30,192.168.1.8/29,192.168.1.16/28,192.168.1.32/27,192.168.1.64/26,192.168.1.128/25,192.168.1.252/30,192.168.1.254/31 (This output demonstrates the greedy algorithm finding larger blocks where possible, then breaking down remaining parts into smaller blocks. It covers 192.168.1.1 to 192.168.1.255, excluding 192.168.1.255 which is not included in the range)
Input: 192.168.1.0 192.168.1.255
Output: 192.168.1.0/24
Input: 10.0.0.0 10.255.255.255
Output: 10.0.0.0/8
Input: 192.168.0.1 192.168.0.1
Output: 192.168.0.1/32
Input: 0.0.0.0 255.255.255.255
Output: 0.0.0.0/0
Important Notes:
inet_ntoa() is considered deprecated and not thread-safe because it uses a static internal buffer. For more robust applications, inet_ntop() should be used, which requires providing a buffer. For this simple command-line utility, itโs generally acceptable.uint64_t for block_size and intermediate calculations involving large IP address ranges (especially for /0 which represents 2^32 addresses) is crucial to prevent integer overflow.Gemini models are able to process different data modatilities, such as unstructured files, images, audio and videos, allowing you to experiment with multimodal scenarios where you can ask the model to describe, explain, get insights or extract information out of those multimedia information included into your prompts. In this section you will work across different senarios with multimedia information.
The OpenAI SDK compatibility only supports inline images and audio files. For videos support, use the Gemini APIโs JS SDK.
You will first download the image you want to work with.
const fs = require("fs") as typeof import("fs");
const path = require("path") as typeof import("path");
const downloadFile = async (url: string, filePath: string) => {
const response = await fetch(url);
if (!response.ok) {
throw new Error(`Failed to download file: ${response.statusText}`);
}
fs.mkdirSync(path.dirname(filePath), { recursive: true });
const buffer = await response.blob();
const bufferData = Buffer.from(await buffer.arrayBuffer());
fs.writeFileSync(filePath, bufferData);
};const IMAGE_URL = "https://storage.googleapis.com/generativeai-downloads/images/Japanese_Bento.png";
const imageFilePath = path.join("../assets/openai_compatibility", "Japanese_Bento.png");
await downloadFile(IMAGE_URL, imageFilePath);
tslab.display.png(fs.readFileSync(imageFilePath));
Now you can encode the image and work with the OpenAI library to interact with the Gemini models.
const base64Image = fs.readFileSync(imageFilePath).toString("base64");
const imageResponse = await ai.chat.completions.create({
model: MODEL_ID,
messages: [
{
role: "user",
content: [
{
type: "text",
text: "Describe the items on this image. If there is any non-English text, translate it as well",
},
{
type: "image_url",
image_url: {
url: `data:image/png;base64,${base64Image}`,
},
},
],
},
],
});
tslab.display.markdown(imageResponse.choices[0].message.content ?? "");Here are the descriptions of the items on the image, with translations of the Japanese text:
Row 1 (Top Row):
Row 2 (Middle Row):
Row 3 (Bottom Row):
You can do the same process while sending multiple images into the same prompt.
const IMAGE_URLS = [
"https://storage.googleapis.com/github-repo/img/gemini/retail-recommendations/furnitures/cesar-couto-OB2F6CsMva8-unsplash.jpg",
"https://storage.googleapis.com/github-repo/img/gemini/retail-recommendations/furnitures/daniil-silantev-1P6AnKDw6S8-unsplash.jpg",
"https://storage.googleapis.com/github-repo/img/gemini/retail-recommendations/furnitures/ruslan-bardash-4kTbAMRAHtQ-unsplash.jpg",
"https://storage.googleapis.com/github-repo/img/gemini/retail-recommendations/furnitures/scopic-ltd-NLlWwR4d3qU-unsplash.jpg",
];
const imageFilePaths = IMAGE_URLS.map((url, index) => {
const fileName = `furniture_${index + 1}.jpg`;
return path.join("../assets/openai_compatibility", fileName);
});
await Promise.all(IMAGE_URLS.map((url, index) => downloadFile(url, imageFilePaths[index])));
imageFilePaths.forEach((filePath) => {
tslab.display.jpeg(fs.readFileSync(filePath));
});
Now you can encode the images and send them with your prompt.
import OpenAI from "openai";
const imageBase64s = imageFilePaths.map((filePath) => fs.readFileSync(filePath).toString("base64"));
const multiImageResponse = await ai.chat.completions.create({
model: MODEL_ID,
messages: [
{
role: "user",
content: [
{
type: "text",
text: "Describe for what type of living room each of those items are the best match",
},
...imageBase64s.map(
(base64) =>
({
type: "image_url",
image_url: {
url: `data:image/jpeg;base64,${base64}`,
},
}) as OpenAI.ChatCompletionContentPartImage
),
],
},
],
});
tslab.display.markdown(multiImageResponse.choices[0].message.content ?? "");Hereโs a description of the type of living room each item would best match, based on its design and aesthetic:
1. Industrial Adjustable Stool (Image 1)
2. Tufted Upholstered Armchair (Image 2)
3. Simple Wooden Bar/Counter Stool (Image 3)
4. Modern Lounge Chair with Sculptural Base (Image 4)
You can also send audio files on your prompt. Audio data provides a more rich input than text alone, and can be use for tasks like transcription, or as direct prompting like a voice assistant.
First you need to download the audio you want to use.
const AUDIO_URL = "https://storage.googleapis.com/generativeai-downloads/data/Apollo-11_Day-01-Highlights-10s.mp3";
const audioFilePath = path.join("../assets/openai_compatibility", "Apollo-11_Day-01-Highlights-10s.mp3");
await downloadFile(AUDIO_URL, audioFilePath);
tslab.display.html(
`<audio controls><source src="${audioFilePath}" type="audio/mpeg">Your browser does not support the audio element.</audio>`
);const audioBase64 = fs.readFileSync(audioFilePath).toString("base64");
const audioResponse = await ai.chat.completions.create({
model: MODEL_ID,
messages: [
{
role: "user",
content: [
{
type: "text",
text: "Transcribe this audio file. After transcribing, tell me from what this can be related to.",
},
{
type: "input_audio",
input_audio: {
data: audioBase64,
format: "mp3",
},
},
],
},
],
});
tslab.display.markdown(audioResponse.choices[0].message.content ?? "");Here is the transcription of the audio file:
โminus 10, nine, eight. We have a go for main engine start. We have main engine start.โ
What this can be related to:
This audio can be directly related to a rocket launch or a space mission countdown.
The terminology โminus 10, nine, eight,โ followed by โgo for main engine start,โ and โmain engine startโ are classic phrases used in the procedures leading up to and initiating the launch of a rocket or spacecraft.
Gemini API allows you to format the way your response you be generated via structured outputs. You can define the structure you want to be used as a defined schema and, using the OpenAI library, you send this structure as the response_format parameter.
In this example you will:
response_format parameterFirst you need to download the reference paper. You will use the Attention is all your need Google paper that introduced the Transformers architecture.
const PAPER_URL = "https://arxiv.org/pdf/1706.03762.pdf";
const paperFilePath = path.join("../assets/openai_compatibility", "Attention_is_All_You_Need.pdf");
await downloadFile(PAPER_URL, paperFilePath);Look at one of the pages:
const getPage = async (pdfPath: string, pageNumber: number): Promise<Uint8Array> => {
const { pdf } = await import("pdf-to-img");
const document = await pdf(pdfPath);
let pageIndex = 0;
for await (const page of document) {
if (pageIndex === pageNumber) {
return new Uint8Array(page);
}
pageIndex++;
}
throw new Error(`Page ${pageNumber} not found in the PDF document.`);
};
tslab.display.png(await getPage(paperFilePath, 0));
Now let us extract the text from the PDF file. You can use the pdf-parse package to extract the text from the PDF file.
const pdf_parse = require("pdf-parse") as typeof import("pdf-parse");
const pdfData = fs.readFileSync(paperFilePath);
const pdfText = await pdf_parse(pdfData);
console.log(pdfText.text.slice(0, 500)); // Display the first 500 characters of the PDF text
Provided proper attribution is provided, Google hereby grants permission to
reproduce the tables and figures in this paper solely for use in journalistic or
scholarly works.
Attention Is All You Need
Ashish Vaswani
โ
Google Brain
avaswani@google.com
Noam Shazeer
โ
Google Brain
noam@google.com
Niki Parmar
โ
Google Research
nikip@google.com
Jakob Uszkoreit
โ
Google Research
usz@google.com
Llion Jones
โ
Google Research
llion@google.com
Aidan N. Gomez
โ โ
University of Toronto
aidan@cs.toronto.eduNow you will create your reference structure. You can define the structure you want your response in using the response_format parameter. In this example, you will define a structure that includes the title, authors, abstract, introduction, and conclusion of the paper.
const z = require("zod") as typeof import("zod");
const researchPaperExtractionSchema = z.object({
title: z.string(),
authors: z.array(z.string()),
abstract: z.string(),
keywords: z.array(z.string()),
});import { zodResponseFormat } from "openai/helpers/zod";
const SYSTEM_PROMPT = `
As a specialist in knowledge organization and data refinement, your task is to transform
raw research paper content into a clearly defined structured format. I will provide you
with the original, free-form text. Your goal is to parse this text, extract the pertinent
information, and reconstruct it according to the structure outlined below. Feel free to
to use any relevant information from the text to fill in the fields.
`;
const structuredResponse = await ai.chat.completions.create({
model: MODEL_ID,
messages: [
{
role: "system",
content: SYSTEM_PROMPT,
},
{
role: "user",
content: pdfText.text,
},
],
response_format: zodResponseFormat(researchPaperExtractionSchema, "research_paper_extraction"),
});
console.log(JSON.parse(structuredResponse.choices[0].message.content ?? "{}"));{
abstract: 'The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.',
authors: [
'Ashish Vaswani',
'Noam Shazeer',
'Niki Parmar',
'Jakob Uszkoreit',
'Llion Jones',
'Aidan N. Gomez',
'ลukasz Kaiser',
'Illia Polosukhin'
],
keywords: [
'Transformer',
'Attention Mechanism',
'Self-attention',
'Machine Translation',
'Sequence Transduction',
'Neural Networks'
],
title: 'Attention Is All You Need'
}Given the Gemini API ability to handle structured outputs, you can work in more complex scenarios too - like using the structured output functionality to help you generating user interfaces.
enum UITYpe {
DIV = "div",
BUTTON = "button",
HEADER = "header",
SECTION = "section",
FIELD = "field",
FORM = "form",
}
const attributesSchema = z.object({
name: z.string(),
value: z.string(),
});
const uiSchema = z.object({
type: z.nativeEnum(UITYpe),
label: z.string(),
children: z.array(z.string()),
attributes: z.array(attributesSchema),
});
const uiResponseSchema = z.object({
ui: uiSchema,
});const uiResponse = await ai.chat.completions.create({
model: MODEL_ID,
messages: [
{
role: "system",
content: "You are a UI generation assistant. Convert the user input into a UI.",
},
{
role: "user",
content: "Make a User Profile Form including all required attributes",
},
],
response_format: zodResponseFormat(uiResponseSchema, "ui_response"),
});
console.log(JSON.parse(uiResponse.choices[0].message.content ?? "{}"));{
ui: {
attributes: [],
children: [
'First Name',
'Last Name',
'Email Address',
'Phone Number',
'Street Address',
'City',
'State',
'Zip Code',
'Save Changes'
],
label: 'User Profile',
type: 'form'
}
}The Gemini APIโs function calling feature allows you to extend the modelโs capabilities by providing descriptions of external functions or APIs.
For further understanding of how function calling works with Gemini models, check the Gemini API documentation.
import OpenAI from "openai";
import { zodFunction } from "openai/helpers/zod";
const getWeatherArgs = z.object({
location: z.string().describe("The location for which to get the weather"),
});
const tools: OpenAI.ChatCompletionTool[] = [
zodFunction({
name: "get_weather",
description: "Get the current weather for a given location",
parameters: getWeatherArgs,
}),
];const toolResponse = await ai.chat.completions.create({
model: MODEL_ID,
messages: [
{
role: "user",
content: "What's the weather like in Boston?",
},
],
tools: tools,
});
const toolCall = toolResponse.choices[0].message.tool_calls?.[0];
if (toolCall) {
console.log(JSON.stringify(toolCall, null, 2));
}{
"function": {
"arguments": "{\"location\":\"Boston\"}",
"name": "get_weather"
},
"id": "",
"type": "function"
}Gemini 2.5 models are trained to think through complex problems, leading to significantly improved reasoning. The Gemini API comes with a โthinking budgetโ parameter which gives fine grain control over how much the model will think.
Unlike the Gemini API, the OpenAI API offers three levels of thinking control: โlowโ, โmediumโ, and โhighโ, which are mapped to 1K, 8K, and 24K thinking token budgets.
If you want to disable thinking, you can set the reasoning effort to โnoneโ.
const THINKING_MODEL = "gemini-2.5-flash";
const thinkingResponse = await ai.chat.completions.create({
model: THINKING_MODEL,
reasoning_effort: "low",
messages: [
{ role: "system", content: "You are a helpful assistant." },
{ role: "user", content: " What is 45-78+5x13? Double check and explain why your answer is correct." },
],
});
tslab.display.markdown(thinkingResponse.choices[0].message.content ?? "");To solve the expression 45 - 78 + 5 x 13, we need to follow the order of operations, often remembered by the acronym PEMDAS/BODMAS:
Letโs break it down:
Step 1: Perform Multiplication There is a multiplication operation: 5 x 13 5 x 13 = 65
Now, substitute this value back into the expression: 45 - 78 + 65
Step 2: Perform Addition and Subtraction from Left to Right First, 45 - 78: 45 - 78 = -33
Now, substitute this value back into the expression: -33 + 65
Finally, perform the addition: -33 + 65 = 32
Answer: The answer is 32.
Double Check and Explanation:
45 - 78 + 5 x 135 x 13.
5 x 13 = 65.45 - 78 + 65.45 - 78. This gives -33.-33 + 65.-33 + 65 = 32.45 - 78 + 5 x 135 * 13 = 6545 - 78 + 6545 - 78 = -33-33 + 65 = 32The steps are consistent and the re-calculation confirms the result. The answer is correct because we strictly adhered to the mathematical order of operations, performing multiplication before addition and subtraction, and then performing addition and subtraction from left to right.
Text embeddings offer a compressed, vector-based representation of text, typically in a lower-dimensional space. The core principle is that semantically similar texts will have embeddings that are spatially proximate within the embedding vector space. This representation enables solutions to several prevalent NLP challenges, including:
For more details about working with the Gemini API and embeddings, check the API documentation.
In this example you will use the gemini-embedding-exp-03-07 model from the Gemini API to generate your embeddings.
const EMBEDDINGS_MODEL = "gemini-embedding-exp-03-07";
const embeddingResponse = await ai.embeddings.create({
model: EMBEDDINGS_MODEL,
input: "The quick brown fox jumps over the lazy dog.",
});console.log("Length of embedding vector: ", embeddingResponse.data[0].embedding.length);
const vector = embeddingResponse.data[0].embedding;
console.log("First 10 elements of the embedding vector: ", vector.slice(0, 10));Length of embedding vector: 3072
First 10 elements of the embedding vector: [
-0.013846100308001041,
0.016324089840054512,
-0.0018839917611330748,
-0.04919512942433357,
0.009635014459490776,
0.011142290197312832,
-0.0064404308795928955,
-0.0022762261796742678,
0.014301184564828873,
0.021092839539051056
]A simple application of text embeddings is to calculate the similarity between sentences (ie. product reviews, documents contents, etc). First you will create a group of sentences.
const danfojs = require("danfojs-node") as typeof import("danfojs-node");
const rows = [
"i really enjoyed the movie last night",
"so many amazing cinematic scenes yesterday",
"had a great time writing my Python scripts a few days ago",
"huge sense of relief when my .py script finally ran without error",
"O Romeo, Romeo, wherefore art thou Romeo?",
];
const df = new danfojs.DataFrame(
rows.map((text) => [text]),
{ columns: ["text"] }
);
df.print();โโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโ
โ โ text โ
โโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโข
โ 0 โ i really enjoyeโฆ โ
โโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโข
โ 1 โ so many amazingโฆ โ
โโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโข
โ 2 โ had a great timโฆ โ
โโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโข
โ 3 โ huge sense of rโฆ โ
โโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโข
โ 4 โ O Romeo, Romeo,โฆ โ
โโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโ
Now you can create a function to generate embeddings, apply that function to your dataframe text column and save it into a new column called embeddings.
const generateEmbeddings = async (texts: string[]): Promise<number[][]> => {
const embeddings = await ai.embeddings.create({
model: EMBEDDINGS_MODEL,
input: texts,
});
return embeddings.data.map((e) => e.embedding);
};
// eslint-disable-next-line @typescript-eslint/no-unsafe-member-access
const embeddings = await generateEmbeddings(df.text.values as string[]);const embeddingSeries = new danfojs.Series(embeddings);
df.addColumn("embedding", embeddingSeries, { inplace: true });
df.print();โโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโ
โ โ text โ embedding โ
โโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโข
โ 0 โ i really enjoyeโฆ โ -0.015758849680โฆ โ
โโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโข
โ 1 โ so many amazingโฆ โ -0.012008255347โฆ โ
โโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโข
โ 2 โ had a great timโฆ โ 0.0065507469698โฆ โ
โโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโข
โ 3 โ huge sense of rโฆ โ -0.008062172681โฆ โ
โโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโข
โ 4 โ O Romeo, Romeo,โฆ โ -0.023231310769โฆ โ
โโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโ
Now that you have the embeddings representations for all sentences, you can calculate their similarities.
function cosineSimilarity(a: number[], b: number[]): number {
const dotProduct = a.reduce((sum, value, index) => sum + value * b[index], 0);
const magnitudeA = Math.sqrt(a.reduce((sum, value) => sum + value * value, 0));
const magnitudeB = Math.sqrt(b.reduce((sum, value) => sum + value * value, 0));
return dotProduct / (magnitudeA * magnitudeB);
}
const cosineSimilarityArray = (embeddings: number[][]): number[][] => {
const result: number[][] = [];
for (let i = 0; i < embeddings.length; i++) {
result[i] = [];
for (let j = 0; j < embeddings.length; j++) {
result[i][j] = cosineSimilarity(embeddings[i], embeddings[j]);
}
}
return result;
};
const cosSimArray = cosineSimilarityArray(embeddings);
// eslint-disable-next-line @typescript-eslint/no-unsafe-member-access
const texts = df.text.values as string[];
const cosSimDf = new danfojs.DataFrame(cosSimArray, {
columns: texts,
index: texts,
});
cosSimDf.print();โโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโ
โ โ i really enjoyeโฆ โ so many amazingโฆ โ had a great timโฆ โ huge sense of rโฆ โ O Romeo, Romeo,โฆ โ
โโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโข
โ i really e โ 1 โ 0.6536475251204โฆ โ 0.6451563604059โฆ โ 0.6026055751927โฆ โ 0.5086438505027โฆ โ
โ njoyed the โ โ โ โ โ โ
โ movie last โ โ โ โ โ โ
โ night โ โ โ โ โ โ
โโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโข
โ so many am โ 0.6536475251204โฆ โ 0.9999999999999โฆ โ 0.6495669715220โฆ โ 0.6111587144454โฆ โ 0.5282164440831โฆ โ
โ azing cine โ โ โ โ โ โ
โ matic scen โ โ โ โ โ โ
โ es yesterd โ โ โ โ โ โ
โ ay โ โ โ โ โ โ
โโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโข
โ had a grea โ 0.6451563604059โฆ โ 0.6495669715220โฆ โ 1.0000000000000โฆ โ 0.8108772516390โฆ โ 0.4761749114691โฆ โ
โ t time wri โ โ โ โ โ โ
โ ting my Py โ โ โ โ โ โ
โ thon scrip โ โ โ โ โ โ
โ ts a few d โ โ โ โ โ โ
โ ays ago โ โ โ โ โ โ
โโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโข
โ huge sense โ 0.6026055751927โฆ โ 0.6111587144454โฆ โ 0.8108772516390โฆ โ 1.0000000000000โฆ โ 0.4621836049587โฆ โ
โ of relief โ โ โ โ โ โ
โ when my .p โ โ โ โ โ โ
โ y script f โ โ โ โ โ โ
โ inally ran โ โ โ โ โ โ
โ without er โ โ โ โ โ โ
โ ror โ โ โ โ โ โ
โโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโข
โ O Romeo, R โ 0.5086438505027โฆ โ 0.5282164440831โฆ โ 0.4761749114691โฆ โ 0.4621836049587โฆ โ 1.0000000000000โฆ โ
โ omeo, wher โ โ โ โ โ โ
โ efore art โ โ โ โ โ โ
โ thou Romeo โ โ โ โ โ โ
โ ? โ โ โ โ โ โ
โโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโ
You can also plot it for a better visualization.
import { createCanvas } from "canvas";
const matrix: number[][] = cosineSimilarityArray(embeddings);
// eslint-disable-next-line @typescript-eslint/no-unsafe-member-access
const labels: string[] = df.text.values as string[];
const size = 600;
const padding = 100;
const cellSize = (size - padding) / matrix.length;
const canvas = createCanvas(size, size);
const ctx = canvas.getContext("2d");
function similarityToColor(value: number) {
const intensity = Math.floor(255 * value);
return `rgb(${255 - intensity}, ${255 - intensity}, 255)`; // light to dark blue
}
for (let y = 0; y < matrix.length; y++) {
for (let x = 0; x < matrix.length; x++) {
const sim = matrix[y][x];
ctx.fillStyle = similarityToColor(sim);
ctx.fillRect(padding + x * cellSize, padding + y * cellSize, cellSize, cellSize);
ctx.fillStyle = "black";
ctx.font = "10px Arial";
ctx.fillText(sim.toFixed(2), padding + x * cellSize + 5, padding + y * cellSize + 15);
}
}
ctx.font = "10px Arial";
ctx.fillStyle = "black";
ctx.textAlign = "right";
labels.forEach((label, i) => {
ctx.save();
ctx.translate(padding - 5, padding + i * cellSize + cellSize / 2);
ctx.fillText(label.slice(0, 20), 0, 5);
ctx.restore();
});
labels.forEach((label, i) => {
ctx.save();
ctx.translate(padding + i * cellSize + cellSize / 2, padding - 5);
ctx.rotate(-Math.PI / 2);
ctx.textAlign = "right";
ctx.fillText(label.slice(0, 20), 0, 5);
ctx.restore();
});
tslab.display.html(`<img src="${canvas.toDataURL()}" alt="Cosine Similarity Matrix">`);
If you want to use more of the Gemini capabilities and especially its unique capabilities not available through the OpenAI compatibility, you should check out the Google GenAI SDK.
The Cookbook is full of examples on how to use it but it is recommended to start with the Getting started notebook to get a feel of all the models and SDK capabilities.