shinyreact

How can you add AI voice agents to a Shiny app? | Lucas Da Silva | DS Lab

Use shinyrealtime and GPT Realtime for a live voice agent, or MediaRecorder plus Deepgram for voice notes! Give the agent tools to do the work, and keep the app organized in Shiny modules. In this Data Science Lab session, Lucas Da Silva, a data analyst at Suffolk Construction, demonstrates how his team uses AI and automation to bridge the gap between programming and construction field operations. Lucas presents the evolution of ‘Vera,’ a tool designed to solve the manual friction of construction scheduling. With Shiny for Python, Deepgram transcription, and an LLM, voice notes from the job site can be mapped to schedule tasks and reviewed later, no paper needed. The Data Science Lab is a live weekly call. Register at pos.it/dslab! Discord invites go out each week on lives calls. We’d love to have you! It’s is an open, messy space for learning and asking questions. Think of it like pair coding with a friend or two. Learn something new, and share what you know to help others grow. Originally recorded September 29, 2026. 0:00 Welcome to the Data Science Lab 1:33 Meet Lucas Da Silva, Suffolk Construction 2:53 How the session works 4:26 Construction scheduling 101: planners and superintendents 5:49 The problem: a digital schedule turned into paper and back 7:42 The proposed solution: a voice assistant on the job site 9:39 Getting buy-in for AI tools 11:11 Demo: Vera, the voice agent (built with shinyrealtime) 17:15 Q&A: speed, undo, and how the agent uses tools 20:10 Q&A: BIM, change history, and exporting to P6 22:07 Field-testing lessons: why friction isn’t always a bad thing 25:28 Q&A: security and privacy 27:22 Demo: Vera Notes and Vera Review 29:29 Q&A: accents and construction lingo 34:07 Mapping notes to schedule tasks with an LLM 37:11 Learning Shiny on the job 40:37 Code walkthrough: Shiny modules 42:19 Recording audio with JavaScript and MediaRecorder 43:32 Transcription with Deepgram 46:03 Don’t one-shot it: building an app step by step 47:42 How audio flows through the app 49:04 Storing and structuring notes 50:10 Q&A: Shiny vs. Streamlit, and why Shiny for Python 52:00 Demo code on GitHub (QR code) 54:17 Wrap-up and thank you Resources: Download Positron (the IDE Lucas uses): https://positron.posit.co/download.html Lucas’ GitHub: https://github.com/Suffolk-Construction-Corporation-Inc/presentations_and_materials/tree/main/data_science_lab_2026 Lucas’ presentation: https://github.com/Suffolk-Construction-Corporation-Inc/presentations_and_materials/tree/main/data_science_lab_2026 Shiny modules: https://mastering-shiny.org/scaling-modules.html data-dict package: https://data-dict.tidyverse.org/ Voice-Enabled Applications Shinyrealtime: https://github.com/posit-dev/shinyrealtime Deepgram: https://deepgram.com Deepgram repo: https://github.com/deepgram/deepgram-python-sdk Talk To Your Data Applications commons: https://opensource.posit.co/blog/2026-09-15_commons-0-1-0/ commons repo: https://posit-dev.github.io/commons/ Other Cool Things Lucas Saw At Conf That May (or May Not) Be Relevant Shinyreact: https://github.com/posit-dev/shinyreact Data-dict: https://data-dict.tidyverse.org/