Real-time music generation using Lyria RealTime

The Gemini API, using Lyria RealTime, provides access to a state-of-the-art, real-time, streaming music generation model. It allows developers to build applications where users can interactively create, continuously steer, and perform instrumental music.

Lyria RealTime music generation uses a persistent, bidirectional, low-latency streaming connection using WebSocket.

To experience what can be built using Lyria RealTime, try it on AI Studio using the Prompt DJ or the MIDI DJ apps.

Generate and control music

Lyria RealTime works similarly to the Live API in that it uses Websockets to maintain real-time communication with the model.

The following code demonstrates how to generate music:

Python

This example initializes the Lyria RealTime session using client.aio.live.music.connect(), then sends an initial prompt with session.set_weighted_prompts() along with an initial configuration using session.set_music_generation_config, starts the music generation using session.play() and sets up receive_audio() to process the audio chunks it receives.

  import asyncio
  from google import genai
  from google.genai import types

  client = genai.Client(http_options={'api_version': 'v1beta'})

  async def main():
      async def receive_audio(session):
        """Example background task to process incoming audio."""
        while True:
          async for message in session.receive():
            audio_data = message.server_content.audio_chunks[0].data
            # Process audio...
            await asyncio.sleep(10**-12)

      async with (
        client.aio.live.music.connect(model='models/lyria-realtime-exp') as session,
        asyncio.TaskGroup() as tg,
      ):
        # Set up task to receive server messages.
        tg.create_task(receive_audio(session))

        # Send initial prompts and config
        await session.set_weighted_prompts(
          prompts=[
            types.WeightedPrompt(text='minimal techno', weight=1.0),
          ]
        )
        await session.set_music_generation_config(
          config=types.LiveMusicGenerationConfig(bpm=90, temperature=1.0)
        )

        # Start streaming music
        await session.play()
  if __name__ == "__main__":
      asyncio.run(main())

JavaScript

This example initializes the Lyria RealTime session using client.live.music.connect(), then sends an initial prompt with session.setWeightedPrompts() along with an initial configuration using session.setMusicGenerationConfig, starts the music generation using session.play() and sets up an onMessage callback to process the audio chunks it receives.

import { GoogleGenAI } from "@google/genai";
import Speaker from "speaker";
import { Buffer } from "buffer";

const client = new GoogleGenAI({
  apiKey: GEMINI_API_KEY,
    apiVersion: "v1beta" ,
});

async function main() {
  const speaker = new Speaker({
    channels: 2,       // stereo
    bitDepth: 16,      // 16-bit PCM
    sampleRate: 44100, // 44.1 kHz
  });

  const session = await client.live.music.connect({
    model: "models/lyria-realtime-exp",
    callbacks: {
      onmessage: (message) => {
        if (message.serverContent?.audioChunks) {
          for (const chunk of message.serverContent.audioChunks) {
            const audioBuffer = Buffer.from(chunk