OpenAI API: Building Front-End Voice Apps with the Realtime API and WebRTC
48mGeneral2025-01-27
Authors
Morten Rand-Hendriksen
Senior Staff Instructor, Speaker, Web Designer, and Software Developer
Course details
With OpenAI’s Realtime API, you can build a custom real-time voice-to-voice AI chat into any front-end web app or other app through WebRTC. In this course, you’ll explore how the Realtime API works, see how to set up a secure WebRTC connection with ephemeral tokens, and get hands-on examples of how to add custom real-time AI chat with voice, text, and function calling into any web app using JavaScript.
Learning objectives
Understand what the OpenAI Realtime API is and how it works.
Discover how to use a web server to a secure ephemeral token for secure front-end apps.
Explore how to create a WebRTC connection for voice-to-voice chat.
Build AI voice apps using Vanilla JavaScript.
Create voice visualizations using the Web Audio API.
Add text transcripts to AI voice chat for accessibility.
Enable function calling for voice chats with the OpenAI Realtime API.
Learning objectives
Understand what the OpenAI Realtime API is and how it works.
Discover how to use a web server to a secure ephemeral token for secure front-end apps.
Explore how to create a WebRTC connection for voice-to-voice chat.
Build AI voice apps using Vanilla JavaScript.
Create voice visualizations using the Web Audio API.
Add text transcripts to AI voice chat for accessibility.
Enable function calling for voice chats with the OpenAI Realtime API.
Skills covered
OpenAI APIAPIsOpenAIGenerative AIFront-End Web DevelopmentArtificial Intelligence (AI)Web DevelopmentSoftware DevelopmentOne-Off
Concepts
0. Introduction
- 01 - Custom AI voice apps the easier way with WebRTC
- 02 - Caution - The price of voice and spend limits
1. Voice Chat and the Realtime API
- 03 - Testing the Realtime API in the Playground
- 04 - Configuring the Realtime API's responses
- 05 - Documentation and examples
2. Developing AI Voice Apps
- 06 - Hands on with five JavaScript AI voice apps
- 07 - OpenAI authentication with ephemeral tokens
- 08 - Understanding the WebRTC flow
- 09 - A bare-bones JavaScript Realtime API implementation
- 10 - Configuring assistant messages and settings
- 11 - Adding visualizations with the Web Audio API
- 12 - Adding text chat to a Realtime app
- 13 - Adding text transcripts for accessibility
- 14 - Function calling with the Realtime API
Conclusion
- 15 - The power and responsibility of voice chat