Analyze audio files using the Gemini API

You can ask a Gemini model to analyze audio files that you provide either inline (base64-encoded) or via URL. When you use Firebase AI Logic, you can make this request directly from your app.

With this capability, you can do things like:

  • Describe, summarize, or answer questions about audio content
  • Transcribe audio content
  • Analyze specific segments of audio using timestamps

This guide is about generating text from audio input.

Jump to code samples Jump to code for streamed responses


See other guides for additional options for working with audio
Generate structured output Multi-turn chat Text-to-speech (TTS) Bidirectional streaming

Before you begin

Click your Gemini API provider to view provider-specific content and code on this page.

If you haven't already, complete the getting started guide, which describes how to set up your Firebase project, connect your app to Firebase, add the SDK, initialize the backend service for your chosen Gemini API provider, and create a GenerativeModel instance.

For testing and iterating on your prompts, we recommend using Google AI Studio.

Models that support this capability

This guide is about generating text from audio input, and it's applicable to the following Gemini models:

  • gemini-3.1-pro-preview
  • gemini-3.7-flash (and the older gemini-3.6-flash and gemini-3.5-flash)
  • gemini-3.5-flash-lite (and the older gemini-3.1-flash-lite)

General-use Gemini 2.5 models support this capability, but they're all deprecated.

Generate text from audio files (base64-encoded)

Before trying this sample, complete the Before you begin section of this guide to set up your project and app.
In that section, you'll also click a button for your chosen Gemini API provider so that you see provider-specific content on this page.

You can ask a Gemini model to generate text by prompting with text and audio—providing the input file's mimeType and the file itself. Find requirements and recommendations for input files later on this page.

Swift

You can call generateContent() to generate text from multimodal input of text and a single audio file.


import FirebaseAILogic

// Initialize the Gemini Developer API backend service.
let ai = FirebaseAI.firebaseAI(backend: .googleAI())

// Create a `GenerativeModel` instance with a model that supports your use case.
let model = ai.generativeModel(modelName: "gemini-3.7-flash")


// Provide the audio as `Data`
guard let audioData = try? Data(contentsOf: audioURL) else {
    print("Error loading audio data.")
    return // Or handle the error appropriately
}

// Specify the appropriate audio MIME type
let audio = InlineDataPart(data: audioData, mimeType: "audio/mpeg")


// Provide a text prompt to include with the audio
let prompt = "Transcribe what's said in this audio recording."

// To generate text output, call `generateContent` with the audio and text prompt
let response = try await model.generateContent(audio, prompt)

// Print the generated text, handling the case where it might be nil
print(response.text ?? "No text in response.")

Kotlin

You can call generateContent() to generate text from multimodal input of text and a single audio file.

For Kotlin, the methods in this SDK are suspend functions and need to be called from a Coroutine scope. 

// Initialize the Gemini Developer API backend service.
// Create a `GenerativeModel` instance with a model that supports your use case.
val model = Firebase.ai(backend = GenerativeBackend.googleAI())
                        .generativeModel("gemini-3.7-flash")


val contentResolver = applicationContext.contentResolver

val inputStream = contentResolver.openInputStream(audioUri)

if (inputStream != null) {  // Check if the audio loaded successfully
    inputStream.use { stream ->
        val bytes = stream.readBytes()

        // Provide a prompt that includes the audio specified above and text
        val prompt = content {
            inlineData(bytes, "audio/mpeg")  // Specify the appropriate audio MIME type
            text(