Gemini API: Analyze a Video - Historic Event Recognition

This notebook shows how you can use Gemini models’ multimodal capabilities to recognize which historic event is happening in the video.

Setup

Install the Google GenAI SDK

Install the Google GenAI SDK from npm.

$ npm install @google/genai

Setup your API key

You can create your API key using Google AI Studio with a single click.

Remember to treat your API key like a password. Don’t accidentally save it in a notebook or source file you later commit to GitHub. In this notebook we will be storing the API key in a .env file. You can also set it as an environment variable or use a secret manager.

Here’s how to set it up in a .env file:

$ touch .env
$ echo "GEMINI_API_KEY=<YOUR_API_KEY>" >> .env

Tip

Another option is to set the API key as an environment variable. You can do this in your terminal with the following command:

$ export GEMINI_API_KEY="<YOUR_API_KEY>"

Load the API key

To load the API key from the .env file, we will use the dotenv package. This package loads environment variables from a .env file into process.env.

$ npm install dotenv

Then, we can load the API key in our code:

const dotenv = require("dotenv") as typeof import("dotenv");

dotenv.config({
  path: "../.env",
});

const GEMINI_API_KEY = process.env.GEMINI_API_KEY ?? "";
if (!GEMINI_API_KEY) {
  throw new Error("GEMINI_API_KEY is not set in the environment variables");
}
console.log("GEMINI_API_KEY is set in the environment variables");

GEMINI_API_KEY is set in the environment variables

Note

In our particular case the .env is is one directory up from the notebook, hence we need to use ../ to go up one directory. If the .env file is in the same directory as the notebook, you can omit it altogether.

│
├── .env
└── examples
    └── Analyze_a_Video_Historic_Event_Recognition.ipynb

Initialize SDK Client

With the new SDK, now you only need to initialize a client with you API key (or OAuth if using Vertex AI). The model is now set in each call.

const google = require("@google/genai") as typeof import("@google/genai");

const ai = new google.GoogleGenAI({ apiKey: GEMINI_API_KEY });

Select a model

Now select the model you want to use in this guide, either by selecting one in the list or writing it down. Keep in mind that some models, like the 2.5 ones are thinking models and thus take slightly more time to respond (cf. thinking notebook for more details and in particular learn how to switch the thiking off).

const tslab = require("tslab") as typeof import("tslab");

const MODEL_ID = "gemini-2.5-flash-preview-05-20";

Example

This example uses video of President Ronald Reagan’s Speech at the Berlin Wall taken on June 12 1987.

const fs = require("fs") as typeof import("fs");
const path = require("path") as typeof import("path");

const downloadFile = async (url: string, filePath: string) => {
  const response = await fetch(url);
  if (!response.ok) {
    throw new Error(`Failed to download file: ${response.statusText}`);
  }
  fs.mkdirSync(path.dirname(filePath), { recursive: true });
  const buffer = await response.blob();
  const bufferData = Buffer.from(await buffer.arrayBuffer());
  fs.writeFileSync(filePath, bufferData);
};

const VIDEO_URL =
  "https://s3.amazonaws.com/NARAprodstorage/opastorage/live/16/147/6014716/content/presidential-libraries/reagan/5730544/6-12-1987-439.mp4";
const VIDEO_FILE_PATH = path.join("../assets/examples", "Reagan_Berlin_Wall.mp4");
await downloadFile(VIDEO_URL, VIDEO_FILE_PATH);

Upload the file using the File API so its easier to pass it to the model later on.

import { File, FileState } from "@google/genai";

async function deferredFileUpload(filePath: string, config: { displayName: string }): Promise<File> {
  const file = await ai.files.upload({
    file: filePath,
    config,
  });
  let getFile = await ai.files.get({ name: file.name ?? "" });
  while (getFile.state === FileState.PROCESSING) {
    getFile = await ai.files.get({ name: file.name ?? "" });
    console.log(`current file status (${getFile.displayName}): ${getFile.state ?? "unknown"}`);
    console.log("File is still processing, retrying in 5 seconds");

    await new Promise((resolve) => {
      setTimeout(resolve, 5000);
    });
  }
  if (file.state === FileState.FAILED) {
    throw new Error("File processing failed.");
  }
  return file;
}

const videoFile = await deferredFileUpload(VIDEO_FILE_PATH, {
  displayName: "President Reagan's Speech at the Berlin Wall",
});

current file status (President Reagan's Speech at the Berlin Wall): PROCESSING
File is still processing, retrying in 5 seconds
current file status (President Reagan's Speech at the Berlin Wall): PROCESSING
File is still processing, retrying in 5 seconds
current file status (President Reagan's Speech at the Berlin Wall): PROCESSING
File is still processing, retrying in 5 seconds
current file status (President Reagan's Speech at the Berlin Wall): PROCESSING
File is still processing, retrying in 5 seconds
current file status (President Reagan's Speech at the Berlin Wall): PROCESSING
File is still processing, retrying in 5 seconds
current file status (President Reagan's Speech at the Berlin Wall): PROCESSING
File is still processing, retrying in 5 seconds
current file status (President Reagan's Speech at the Berlin Wall): PROCESSING
File is still processing, retrying in 5 seconds
current file status (President Reagan's Speech at the Berlin Wall): PROCESSING
File is still processing, retrying in 5 seconds
current file status (President Reagan's Speech at the Berlin Wall): PROCESSING
File is still processing, retrying in 5 seconds
current file status (President Reagan's Speech at the Berlin Wall): PROCESSING
File is still processing, retrying in 5 seconds
current file status (President Reagan's Speech at the Berlin Wall): ACTIVE
File is still processing, retrying in 5 seconds

The uploaded video is ready for processing. This prompt instructs the model to provide basic information about the historical events portrayed in the video.

const SYSTEM_PROMPT = `
  You are historian who specializes in events caught on film.
  When you receive a video answer following questions:
  When did it happen?
  Who is the most important person in video?
  How the event is called?
`;

Some historic events touch on controversial topics that may get flagged by Gemini API, which blocks the response for the query.

Because of this, it might be a good idea to turn off safety settings.

const response = await ai.models.generateContent({
  model: MODEL_ID,
  contents: [
    "Analyze the video please",
    google.createPartFromUri(videoFile.uri ?? "", videoFile.mimeType ?? "video/mp4"),
  ],
  config: {
    systemInstruction: SYSTEM_PROMPT,
    safetySettings: [
      {
        category: google.HarmCategory.HARM_CATEGORY_HARASSMENT,
        threshold: google.HarmBlockThreshold.BLOCK_NONE,
      },
      {
        category: google.HarmCategory.HARM_CATEGORY_HATE_SPEECH,
        threshold: google.HarmBlockThreshold.BLOCK_NONE,
      },
      {
        category: google.HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT,
        threshold: google.HarmBlockThreshold.BLOCK_NONE,
      },
      {
        category: google.HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT,
        threshold: google.HarmBlockThreshold.BLOCK_NONE,
      },
    ],
  },
});
tslab.display.markdown(response.text ?? "No response text available");

Based on the video, here is the analysis:

• When did it happen? This event took place on June 12, 1987. President Reagan references President Kennedy’s visit 24 years prior (June 1963) and the 40th anniversary of the Marshall Plan (June 1947), both pointing to 1987.

• Who is the most important person in the video? The most important person in the video is Ronald Reagan, who was the President of the United States at the time. He is delivering the address and is the central focus of the recording.

• How the event is called? This iconic event is widely known as the “Tear Down This Wall” speech or Reagan’s Brandenburg Gate speech. In it, President Reagan directly challenged Soviet leader Mikhail Gorbachev to dismantle the Berlin Wall.

As you can see, the model correctly provided information about the dates, Ronald Reagan, who was the main subject of the video, and the name of this event.

You can delete the video to prevent unnecessary data storage.

await ai.files.delete({
  name: videoFile.name ?? "",
});

DeleteFileResponse {}

Summary

Now you know how you can prompt Gemini models with videos and use them to recognize historic events.

This notebook shows only one of many use cases. Check the Video understanding notebook for more examples of using the Gemini API with videos.