Transcript#

This transcript was generated automatically and may contain errors.

Welcome back to the Data Science Lab, everybody. We've been on break for a little while because of PositConf. Isabella and I both ran the virtual side of PositConf. So if you attended PositConf virtually in any way, whether it was virtual day or live streaming the talks on Tuesday and Wednesday, Isabella and I were running around like crazy behind the scenes making sure that everything was going well. And we're so glad that you joined us because it was a really, really good time.

At the lab, we talk about data science topics of all shapes and sizes. Sometimes that is a show and tell. We have people showing and sharing things that they built, personal projects. Sometimes it's a learning topic, generally always a technical topic. And we pride ourselves on being a place where we get to ask questions. Chat and questions happen in the Discord. Isabella and I will be your hosts today. Isabella Velasquez, would you like to say hello?

Hey, everyone. It's so good to see you again. Thanks for joining.

Thank you so much for being here. My name is Libby Heron. I do community stuff here at Posit. We are joined by our featured lab manager for today, Lucas Da Silva. Lucas is a data analyst at Suffolk Construction. We love Suffolk Construction. We also have been joined by Blake at Suffolk on the Data Science Hangout before and he is so much fun and he's given conf talks. So I'm really excited that Lucas is here today. Lucas, would you like to introduce yourself? Tell us a little bit about your role at Suffolk and then also what you will be walking us through today.

Yes. First of all, very excited to be here. So thanks for having me. As Libby was saying, I'm a data analyst at Suffolk Construction on the data product team. Kind of the focus of what I do is kind of bridge the gap between what we do on the programming side in terms of data and AI and then construction. So how do we make the lives of people who work on a job site a lot easier using data and then automation using AI? So that's kind of what I'm going to talk about today as well. So super excited to get into it.

Yeah, let's do it. Well, on the lab, if you've never been here before, we're not a huge slide presentation group. We focus more on screen sharing. So Lucas might have some slides to help us interpret and understand things. And he's also going to be doing some screen sharing. So Lucas, I will say, take it away with the screen share.

So I do have some slides. I'm sorry in advance. Oh, I think you have sound that might play. So make sure when you hit your share thing that you click the little button that lets you share system sound. Yes, we'll do. Okay, cool, cool, cool. So let me share my slides first. Hopefully, I'll try to breeze through these. But one thing I'll say as I go through this, can you see my slides? I think they're loading for me. Okay, yes, I see that. Awesome. As I go through this, what I want to say is, I love this whole idea of this is kind of community driven. So ask as many questions as you want. Even if you feel like I'm in the middle of something, I will stop and answer your question. I love that idea. I love answering questions. So and this is going to be kind of more of a code review than a kind of a live coding session. So the more questions you ask, the better I think this this conversation will be. So stop me at any time. And I think Libby and Isabella will be handling from from you to me.

Introducing the problem: construction scheduling

So like I said, go to the next slide. So I kind of just went through this. But yeah, I'm Lucas Da Silva, data analyst on the data product team here at Suffolk Construction. And then that's a picture of me. Maybe you can't maybe you can spot me in that picture on a job site. I'm in there somewhere. And yeah, there you go. And then I'm going to kind of go over one of the examples of what I do, and how I was able to automate a pain point for our team members on the job site.

So this is the most slidey slide I have. So apologies in advance. I'm going to kind of breeze through this. But feel free to ask questions if you have. But I'm going to first give a very brief overview of construction scheduling, what it means, and, you know, why this was a pain point. So really quickly, construction schedule, you have this this project on a job site, you can kind of break these down kind of line by line into individual tasks that have a start date and end date, a complete duration. And, you know, you go from one task at the start of the project to the next to the next, the next, the next. And then by the end of it, you have a finished building, right?

For big projects, you're going to have a lot of tasks. So we have someone who's their whole job is to take care of the schedule, maintain the schedule, build the initial schedule, and then coordinate with the owner of the project to make sure that they understand if the project is on time, if it's finishing quick, or if there are delays. So that's the planner. And they're in charge of like tasks that can be it can be 1300 tasks, 1500 tasks, even more than that. So they're the owner of the construction schedule. But planners are not on site, they don't walk the job. So they get updates from superintendents. So superintendents are essentially they manage the day to day of a construction job. And they provide updates back to the planner to be able to give updates the schedule that they can then go to the owner.

So quick overview. So I want to talk about the current workflow of what that maintenance looks like and what those updates look like. This is like my first foray into using I don't even want to call it AI art, like AI imagery into a presentation. So I thought these are gonna be really bad. I thought like maybe if I can atone for my sins of using AI imagery by making them not good. So you're in a safe space. We're gonna laugh with you.

So this is like the problem in the current workflow, and then kind of the worst case scenario. So you have a planner, and they have this construction schedule inside of P6, which is a software to handle construction schedules. And they would export that kind of digitized schedule into a PDF. What they're going to do then is send that over, likely via email to the superintendent, who in some cases will print out that PDF. And then once they're walking the job, they're taking notes on like a legal pad or a notebook, or maybe sometimes on the PDF itself, then they manually update this PDF, that's turned from something digitized to something manual.

And then they're sending that PDF back to the planner who then re-digitizes these updates, right. So this is a process where you have something digitized, you take it out of the digital realm into something physical, and then you re-digitize into that same system. So I think any any one of us that have this like automation, bone in our body sees like several problems with this workflow. So we were like, Okay, this is a this is a major pain point. This is taking probably too much time. And there's a lot of unnecessary steps here, you're, you're taking something digital, and then making it not digital just to bring it back.

I think any any one of us that have this like automation, bone in our body sees like several problems with this workflow.

Introducing Vera: the proposed solution

So this was our original proposed solution, and going to be the first demo of the day. So this idea came to us where, you know, what if the ideally in an ideal world, you'd have the planner join the superintendent on these on these walkthroughs, so then they could get updates straight from the superintendent and be able to digitize them kind of in one step, there would be no bringing it to a PDF, and then going back to the planner to reach it re-digitize. So obviously, we can't do that in the real world, there are constraints in terms of how many planners you have, obviously, they're not going to be able to walk the site there, they don't have time to do that. So it's like, Okay, how do we cut out the middleman a little bit, and be like, Okay, we can provide using AI, some voice assistant that can be kind of the planner without needing the planner to be there. And then that can help the superintendent provide updates. And go back to the planner, all always digitized, there's no step of being like I have this this schedule already digitized, I'm taking it into into the PDF realm, why don't we digitize it throughout the entire process and have an assistant be able to take the place of a planner on the site. So I'm going to start by showing this demo.

Feel free to ask questions as I start to pull this up if you have any.

We were having a conversation in the chat, because it's a common problem to have like a solution built that requires change management, right? Like it requires people to change the way that they work in order to do something in that more automated way. And so Shivani was asking, like, how do you even tackle that there is a solution that needs to be implemented? What if people don't use it? How do you get feedback on that and stuff like that?

Um, so as we go, if you happen to have thoughts on that, we would love to. Yeah, I think buy in is the number one thing that you need, or especially an AI product, because instantly you think of AI. And it's scary, right? Like it's scary to me, when I'm thinking of how it can automate my job in places that I don't want it to automate, where it can like take over. So and I'm going to talk about this a little more later on. But I think this concept of like meeting people where they are, like, we don't automate things without talking to people getting their pain points and understanding like, people who are in this today, know that what they're doing is not great. Like this workflow is not something that people are like up in arms about that they want to keep and maintain. So we're like, first, what are your pain points? Where can we work to fix it in terms of your overall workflow? And then how do I keep you in every step of the way? And I think when people feel like they have this responsibility, as well, over improving their own workflow, like they're a part of the process, it makes it a lot easier for them to buy in. And I'd say buy in is like, you should treat that as as one of the pillars of any product that you're going to do, especially to a non technical crowd. And it's buying for the users and leadership too, right? Like both of those parties are really important, because leadership can put pressure downward. But if the people using the tool don't like it, they still probably won't use it.

100%. Okay. So this demo is the first iteration. Yes. Of the tool that you built to solve the problem. Exactly. So this is the original version of Vera. The idea is that this would live on the phone. At Suffolk, pretty much every superintendent has a work phone that they bring with them at all times. So this idea of like, meet them where they are, like, let's use the tools that they have at their disposal, and they're used to bringing with them and provide products that way. So this is a Shiny app, I built this in R. And it's using a package called shiny real time, that I think was first developed by Joe Chang, for not this past PositConf, but the PositConf before. And I used it for this product.

So I'll just launch here. So the idea is a superintendent would have this on their phone, and would be bringing this with them for the site walkthrough instead of their legal notepad, or the PDF that was generated by the planner. And then they just say the project ID to load up their schedule. My Oh, I just remembered, I don't have sound. Share the sound. Yeah, you stop sharing and share it again. This is going to be very important for this part. So we have to pause. We'll get the full experience. I have it. There we go.

It's so funny seeing Vera as well. And I were talking about this too, because we've seen the name Vera on so many different like assistant tools across industries. Yeah. Very truthful.

We named Vera after so P6 is the scheduling software that we and many other construction companies use. That's short for Primavera Six. So we don't even get it from like the truth. Like the Latin. But maybe they did. But we get it from the actual software. It's like an Oracle software. So if anybody else has seen a Vera out in the wild, that's like an agent of some kind sharing discord. Oh, yeah, that would be cool. I would I would like to see there's more than just ours.

Yeah, okay. Let me load my project here. My project ID is 123456. Let me know if you can hear it. You should now see the schedule data filtered to project 123456. Let me know if you would like to update any of the tasks. So the idea now is I don't have this this PDF that I have to bring around with me. I just have this agent that I can just chat with and provide updates in real time. So I could go through some of these. Can you set that third task to complete? And then can you set the plumbing pressure test level one task to complete but it completed on 927 not 926.

I found both tasks you're looking for. The third task is overhead duct installation level one. The task plumbing pressure test level one is also found. Do you want to update both of these as requested? I'll proceed with marking them complete. And for the plumbing pressure test task, I'll set the completion date to 927. Please confirm if that's good to go. Yeah, that's perfect. Thank you.

Both tasks have been updated. The third task overhead duct installation level one is now marked complete. And the plumbing pressure test level one task is also complete with the adjusted completion date of 927. You should see these reflected in the schedule now. So I could keep going here, right? Like, I can start updating these tasks, go to the next page. Go to the next page, please.

Or I can even filter these down, say like only give me the tasks starting next week or this week, give me the tasks that are overdue, meaning they're in the past, but I haven't changed their status. And then I can keep updating this. So I'm sure people can see like the benefit of this over the original workflow. And if you're very astute, you can see the problems as well, which I'm going to talk about as well. Is there any questions about this so far?

Okay, one funny one from Connor. I would love if the bot had the voice of Scotty from Star Trek.

So Tan did really quickly. How did you pick the voice for this? Did this come pre-packaged?

So I'm using Shiny Realtime, which connects. It's a wrapper for GPT Realtime, which is a service that OpenAI has for voice agents. You have like a selection of I think about four voices. So there's not many. Yeah. Yeah. I think my idea was like, it was like that just sounded the most like Avera to me, I guess.

Oh yeah. You did have a name already. Okay. Well, there were a couple of legit ones that we can ask. Hubert was saying, I wonder about information density of what they need to see versus what those printed PDFs show. Yeah. Yeah. I think we were going to work on this.

And again, this is an older version of this product. I think that's probably a problem in two places. It's like a problem in the original workflow where, you know, there is probably too much information there that they might not need to see.

The PDFs. Yeah, exactly. And then that's probably something, you know, when you automate these workflows, in a lot of cases, you're going to bring some of the maybe inefficiencies into your version. But I actually don't know if that's necessarily a bad thing. Sometimes what you need to do is bring in these inefficiencies that may exist in this original workflow. So then you can kind of iterate on that process as well.

But now you own it from end to end, right before you might have this inefficiency with this density in the PDF. And if you port it over, so people are at least like in this first phase, they're kind of seeing what they're used to. You go, okay, now let's try to wean you off of this inefficiency when you already have that trust built up with whoever you're providing this product with.

I see that exact same thing. Yeah. You're like, we're going to bring over what looks familiar so that it's not so jarring to change the process. And then over time, maybe that can go away when people realize, oh, I don't actually need that. I was just used to seeing it. There was also one from Montesir who said, how come this is so blazing fast? That response was extremely fast.

Is it that fast when you were testing it in the field? Yeah, definitely. So GPT real-time is pretty fast. I didn't love the idea of having the voice aspect of it, the model aspect of it, and then the response all in one call. Because this is one call, you're just sending this request to GPT real-time and it's giving you a response back.

But it's just so fast. And it's using a WebSocket call to their service that it was hard to separate those out because it would just be slower going from one to the other. But that's all really GPT real-time. And then not using the WebSocket call to make it really, really fast.

Yeah. It would be really disjointed for my brain to think, okay, you can only ask one thing at a time. I think as a human, I'm used to just being like, here, do these three things.

Logan had said, is there a way to undo? He says, I'm the worst at talking to AI, so I always have to redo or undo. Yeah. So this is old now. I believe I did implement a way to just go undo the last thing I just did and then it would do that.

Yeah. There is a way to undo. But these are all tools. So really behind the scenes of this, I have essentially just doing kind of harness building. So I have my agent with its initial system prompt. And then I just have a bunch of tools that I give it access to. So every time it's updating something, so I go overhead duct installation level two, can you set that to complete? The task overhead duct installation level two has been marked as complete. You should see this reflected in the schedule now. So let me know if there's anything else you'd like to adjust.

Sorry, I didn't mean to interrupt you, Vera. But I'll kind of break down like what that just did, right? So I have a couple of tools that I give available to the agent. One, I'm letting it search through what's visible to it in this data set. And then I'm letting it search through what in this data set. And then I'm letting it kind of make the decision. You can see the task code, the task name, and everything you can see here. And a little more I give to the agent. And then its whole job is to search through the schedule and pinpoint the task. That's one tool that I provide to the agent. Secondly, switching between different statuses, end dates, start dates, et cetera. That's another tool. So this is just a lot of harness engineering that I'm doing.

To go from the question to the final result. Does that answer the question? I don't know if I went on a tangent there. I think so. Well, there's another question too. Actually, there's two. There's one from Notobeco and one from Connor. But I think they asked something similar. And so we can feed into that, which is Notobeco said, does this app also feed into the company's project BIMs, the building information models?

That's an awesome question. Yeah. And Connor's question was similar. Is this data being written back to some change history table via SQL?

I'll start with the BIM question. The answer to that is no. I work very closely with a BIM data engineer that works at Suffolk too. That's a world where I would like to get more knowledge about and understand it fully. Because if you work in construction, there's so many data points there. And it's so interesting what you can do with BIM. But no, this does not connect to any BIM data systems or models. But that would be something I would love to explore.

Second question. Yeah. So the idea here is that after a user would make all these updates, they would have the option to export. And so what that would do is it would send an additional schedule to P6 that the planner would get notified via their email that there's a new schedule there for them to look at. And it wouldn't update anything on their end. It would just give them an additional schedule. And then they could bring those updates back into the schedule that they had as the actual contractual document.

So we didn't want to overwrite anything, especially because it's a contractual document, but we wanted it to be digitized for the planner right after the update. So they would get an additional schedule that they could bring back into their current. That is amazing. And keep in mind, everybody, we're at the halfway mark. This is literally the V1 version, the first iteration of what Lucas made. And Lucas, you learned a lot from this about what was going to work and what was not going to work. So when you field tested this with people, how did it go?

Field testing and lessons learned

Yeah. Let me move back here so we can chat about it a little more with more of my really bad- Slidy slides? Yeah, my really bad slidy slides. I'm sorry.

I love that you said, this is the slidiest slide I have. And I'm like, oh, that is such a good adjective. So let me go back. So this bottom one, so here, I think what we did is we over-engineered the answer to this problem in a couple of ways. And I think if you're an astute observer, you could see some reasons why. So we actually found that when we were piloting this, the superintendents actually really loved the voice concept, like being able to just chat to your phone and give updates, that that really resonated with them. And they felt like that would be a really good way to speed up their process. But the one thing that they mentioned is talking to a voice agent, and you even saw it when it was interrupting me and it was being a little verbose, it's kind of annoying, just point blank. And it's time-consuming trying to find these tasks that you were writing on paper before, and then later on you were figuring out, trying to do it in the same space at the same time. And these walkthroughs are not just for schedule updates. You're also talking to trade partners, trying to make sure everything's going to plan. So it was a little bit jarring to try to do everything into one, and it was too time consuming for them to realistically do this on every walkthrough. So that kind of taught me two real product lessons.

One, kind of meet people where they are, which we did okay. We're putting this on your phone. You have this already available to you. But one thing I kind of want to harp on is friction is not always an obstacle. So this friction of having the separation of your walkthrough, where you're actually getting the updates in real time, versus this place to kind of put them all together, that's a piece of friction. Not necessarily a bad thing to have friction. Sometimes it's there for a reason. So whenever you're trying to automate a workflow, think about the friction points and think about, is this piece of friction there for a reason?

think about the friction points and think about, is this piece of friction there for a reason?

This separation of me going on my walkthrough and then providing updates later, maybe that's not necessarily something that needs to be automated away. So I think that leads us in to their notes pretty well. So I can pull up- Was it also too loud on a job site to have your sound parsed or during walkthroughs was it pretty chill and noise wasn't as big of a deal?

So it did okay, I will say. I think especially if we had some type of headphones happening, usually the headphones are okay at blocking out those sounds. I think there was, probably if we continued this path, we would have had to do some more exploration on if we needed to provide some type of headphones that are noise canceling, have a good microphone, things like that. But based on the feedback that I got, it did an okay job even with the noise.

Oh, you know what? Noor has a question that I hadn't even thought of. She said, how do you handle security with this type of app? Because you mentioned talking to trade partners while also doing walkthroughs. And she's like, I'm assuming you don't want it to record a trade partner meeting or somebody else talking, right?

Yeah. I think part of that is... So one, everything that we do, when we tie it back to, especially in LLM, we have agreements with the providers that they're not going to store that data for training. So that's one thing. And then two, we're not doing this perpetual recording here. It's more engaged by the actual user. Because you're pressing a button. Exactly. It's not listening serious, right?

Exactly. So I think that's part of the two pieces of security there, where we don't really save the recordings either. We just save the output. So what changes are being made, in this case, when you see the new Vera, the end note, not the actual recording.

Vera V2: notes and review modules

All right, cool. Well, let's see. New version. And is this new version deployed yet? Is it out in the field doing stuff already? I'd say it's a little bit pre-pilot and I'll explain why, but I've done a lot of presentations on this to people on site and they're pretty excited. They're like, can we get this today? And I'm like, let's give it a little bit of time. But the idea here is we're still on the phone, but now we've moved over to, and sorry, I need to pull up my script here. I have a little script.

Okay, nice. Okay, I got it. So we're still on the phone, but we've kind of moved into two modules. We have Vera Notes and Vera Review. So Vera Notes, I've taken away the assistant. Again, it was annoying. And it was not only annoying to use as a user, it was really annoying to develop. Because you can imagine me in the office working on the original version of Vera and having to have full-on conversations with an AI assistant just in my office at my desk and people looking at me like, what are you doing?

Especially when it would do wrong things and I had to be like, you're wrong.

So it was annoying on both fronts. I'm glad we kind of moved away from it. So this is more of a transcription service, a note recording service that we can then use in a separate way to have a lot of utility. So I'll just start recording some notes. So this would be me walking the job site and I'll start to record some notes. And this is Shiny Python instead of Shiny R.

The fence is all the way up around the site now. The crew has finished getting everything set up this morning. So I just recorded that note. It took that in, made it into text.

And now I have this review piece. Let me make sure it transcribed everything correctly. Or if it made a mistake, I can fix it up here just in text. This would be with my thumbs instead of my mouse and keyboard. And then I'm like, OK, got it correctly. Let me save that note. And so now this is tied in. This is what gets stored again. We don't save these recordings that you're making, just the output. And then I'll do another one. We haven't gone to the elevator pit forms yet. The crew should start October 1st and have them ready by October 5th.

OK. And I stuttered there and it still got me, which is nice.

Cool. I'll save that note too. And then I'll go, need to get some extra safety glasses in the trailer before the owner walks the job. And it got me there too. And I can save or discard. I'll save it. So any questions so far on this note portion?

No, but we got some feedback from Notobeco saying project managers would froth for this.

Oh, we did get one. Daniela says, does the voice assistant understand people with an accent? Yes. So we have used, like, we have a lot of different people from a lot of different backgrounds with a lot of different accents. And so we've had some testing with people with different accents. I think it does a worse job if you have an accent, maybe, but it does it pretty well still if you have an accent. And at least there's a reviews component, because you don't get stuck with just whatever it has decided you said. There was also a question

from Logan that I was going to ask too, because you had mentioned elevator pit forms, which is like not a thing that I think normal people say in everyday conversation. So Logan's question was, are there ever industry lingo terms that it doesn't understand and you have to put industry knowledge into it so that it will? Yeah. I think that is more important if you have an assistant. So if you have something that's talking back and forth to you, we had to have this kind of data dictionary of construction terms and what they meant, especially because if you think about the original version, that was essentially an agent that was an orchestrator for tools. So there's this term actualization. And that just means saying that the date that's there happened. So if I actualize a start date, that means that the task started. If I actualize an end date, that means that the task completed. So that was a big thing. Like if a superintendent were to go in there and say, I want to actualize the start date, that's not a status that we have assigned. So you have to be able to give the agent context to be able to route it to the right tool.

I love it. Okay. Also, we had a question that says, is it possible to edit those saved notes? Oh, that's actually a great point. Like before you save them. I didn't add the capability to edit saved notes, but I should have. So you already caught me on something I probably should have done. Product feedback.

Yeah. All right. So Nor says, how is this preferable to just importing voice notes? Is it because it's loaded directly onto the platform? Info Nor, we need some more context.

I think I see what you're saying. So like being able to just type these in and store them that way. I think that's another good point. Like this should be flexible, I think. I will probably add a a form where you can either record or you can just type these in and so I think it's better than the manual obviously because we're already digitizing this like this is the idea of like can we get away from the manual side of this operation and can we kind of just keep this digital throughout the whole life cycle. Yeah I think maybe also this could be a question that's like well what if somebody's just like walking through and like you're in a text platform or you're in Slack you can usually add voice notes right by just recording on your phone and I do like that this is loaded directly into the platform because I have noticed when I am doing things I often I have gotten spoiled and I've gotten used to voice notes and when a platform itself doesn't have the voice note capability I'm like oh you're gonna make me type something like I think that this is really really helpful.

Okay okay yeah yeah so nor said clarification most know most most point most phones have a voice note option right already so yeah I think it is that it's built into the platform right. Yeah it's built into the platform and I'll show you kind of like the the use case here like obviously if this was just you taking notes and this was this is not the end of this application if this was just you taking notes and saving them yeah it'd be nice because this is stored in in Databricks for us so it's protected by like where we have all of our other data. Sure like you could save it in Snowflake or whatever you use but we would probably just buy this service I'm sure there are a billion note-taking tools out there but there's more utility so you have this option now I've done my walkthrough I would probably do like five ten more notes and now I'm gonna push these so I'm gonna get this little pop-up so I'm pushing them which means I'm sending them to the review so you could do this on mobile as well but I'll send this to desktop so the idea would be maybe oh maybe if your workflow is okay I'm taking my notes as I'm doing my walkthrough then when I get back to my desk at the end of the day that's what I'm doing the review step so I would go into Vera review and then I have my notes here that I have previously so I can load the schedule in automap.

This mean you can open it on both mobile and laptop later? Exactly yeah this has a this has a mobile view too and I'll show a little bit like when I when we go over the code we're already coming up on time but I'll try to show the how I separate the desktop and the the mobile UI. Yeah I'm gonna save a question as well I saw one that was about the capability of the transcription tool you're using so we'll save that one for when you talk about tools. Sure cool so I have my notes on the left that I just pushed and I'm trying to map them now to my schedule and so what it's gonna do it takes two of my notes here one about the fence mapped it to the to a schedule component and then it made the changes that I said so I said the fences all the way up around the site now the crew has finished getting everything set up so the this task was originally in progress and so send this to an LLM it maps the the task maps my note to this individual task and then it makes the change to the status because I said it was complete and then it gives me this like confidence percentage just just so it gives me like an extra layer of protection to be like I'm pretty sure that this is what you're talking about and then I have the choice to be like okay is this correct or not I'll read its reasoning I'm like okay that that is right so I'm gonna confirm it.

And then now I haven't this this went from me going to a PDF reviewing my notes and then marking a PDF to send later to the planner I now just pressed a button like this was me just taking my notes which I was going to do anyway but in an easier platform and now I've mapped it to my task and I pressed one button and that that task has an update associated with it and then same here it maps it to the task and it moved the dates to October 1st and October 5th and I can either either undo this and just set it up to be changed in this in this format or I'll confirm this too and then for the rest of these that don't have a note I can still go through these and much easier than the PDF I can change this so maybe this is complete and then I can confirm it etc and so this is kind of the replacement for having this voice assistant that I kind of always have to chat with I can kind of separate these steps and do my note-taking in one component and then I can do my schedule updates in a second component that's now informed by this first step that I took.

This went from me going to a PDF reviewing my notes and then marking a PDF to send later to the planner I now just pressed a button like this was me just taking my notes which I was going to do anyway but in an easier platform and now I've mapped it to my task and I pressed one button and that that task has an update associated with it.

So any any questions there. Um we had a suggestion from Renato who was saying it could be that send to review on the button could be more helpful than push it to have to have something that's like more explicit for what you're gonna do we're giving you totally unsolicited product feedback I love it iterating we love it.

Zack had asked did you need to learn how to build something like this on the job and if so how did you find the time to do so were you a Shiny expert before you did this. I would say not really I I've known how to code for a long time and so I think this job was the first time I had started to develop apps and I think Shiny has been a very good way for me to understand how to build apps like the framework is is more robust than maybe some give it credit for but it's super easy to learn as well as iterate on top of so it's been a good experience for me to kind of get through like the life cycle of an entire app end to end through Shiny but I was not an expert before this I'm probably not still not an expert but yeah I kind of learned on the on the job. I love it find you a job where they're gonna let you learn as you go I think that you also have some pretty cool people that you work with and definitely a supportive environment which is amazing.

Walking through the code

All right so the question that I was hanging on to was a capability question about tools and I think you're about to talk about tools so I'm gonna let you keep going we have about 20 minutes left. Okay so I think this point maybe we should go into some of the code so if you have any questions here but these the same idea here is in the last one where after you do your updates they're already digital and then you can send them back out to to the planner already digitized the main idea being that you no longer have to like print something out mark it up make all those changes manually send it back to somebody who's just gonna like print it out again or whatever they're gonna do which is like having worked with clients who are industrial and I also have a sister who's worked for like 30 years for a construction company that's normal like for most people across the United States like everything is still really really manual there's not a whole bunch of automation and stuff in the in this space and so I think this is a really really cool thing that's happening. Yeah innovative you're innovating Lucas.

Yeah and what one thing I'll say like this is kind of why I say pre-pilot we're thinking about as I was building this tool and this was kind of based directly on feedback we got from the pilot now we're thinking of like can we expand this like we have so many source systems that they could this can be useful for and and schedule updates are not the only thing that superintendents are taking notes for they're taking notes to put in their logs they're taking notes to maybe inform safety observations that they have to note like how can we expand this and make this so you have one place to take your notes and then you can kind of push this out to all of these ends that you need in one centralized location instead of going to this source system and that's our system and that's our system so how can we make this an easier process so that's kind of what we're exploring here too. That's a very good business proposition to is like hey I'm building this tool that is gonna solve this problem but it has the potential to solve all these other ones and that's a great way to get buy-in.

Okay so I'm gonna this is going to be this is a lot of code in this code base I just want to say so the more questions you ask I'm gonna kind of go over this at a high level instead of going like really deep into it but I want to say a couple of things here you're gonna notice that I have a lot of modules when you're making Shiny apps I think it's very easy to put everything into your app file I think you should kind of think about it the opposite like this is a complicated Shiny app so I separate everything into modules and the way that I see modules is like every script that I have in here I should be able to explain what it does in one sentence and you're kind of understanding where you need to go if something goes wrong or if you want to add a new feature you need to you can understand where you need to go in this code base and so you'll see my app file is maybe 230 lines just not not big a lot of this is like some basic UI stuff for the home page and then just making sure all of my other modules are tied together so I kind of want to say that at this point if you if you don't modularize your your apps today I would really really recommend it.

Every script that I have in here I should be able to explain what it does in one sentence and you're kind of understanding where you need to go if something goes wrong or if you want to add a new feature.

And then I'm gonna move over first to one of the reasons I really like Shiny over like something like Streamlit is I just think it's so easy to tie in you know JavaScript and when you're talking about voice you really probably need to use some type of JavaScript I know that's scary for some but this is a very simple script and I don't think it's even that many lines and a lot of its comments is 141 lines and this does all of my recording for for for voice and I just use a library called MediaRecorder and then Shiny makes it very easy Shiny makes it very easy for me to tie in input values that I have within the Shiny application to my JavaScript files and so I have my recording all being done through MediaRecorder which just it goes through your browsers built-in microphone and recording services and then I can record that audio and send that back to Shiny specific objects so I can then keep this a Shiny app so this is just the the JavaScript that that makes everything go but all of the complicated things are really happening in my Shiny app outside of hey JavaScript files.

But this is a good place to ask the question that I think Amelia had asked on behalf of a student which was would it be possible to set the input to a different language altogether yes how many languages could you potentially have as your inputs. I think that can be really dependent on the tool that you're using so I'm using I'm using a service called Deepgram for this Deepgram is a they kind of specialize in voice agents as well as voice transcription I believe they have multiple languages available you'll have to go to their website I think at the end of this I have a link for that but they give you a I believe a free $200 API credit and then you can kind of play around with this as well I'm also going to share I have to like demo code bases that one of them is Shiny real-time base and the other one is Deepgram based and then you can kind of play around with it on your own it's gonna be public free for you guys so I'd say play around with it I know they have more languages than just English but I don't know the the breadth of those options.

Fantastic all right yeah I'm really loving that your app dot py file is so tiny because I sure a lot of people have worked on Shiny apps that are enormously long and monolithic. Yeah and then it makes it easier to like demo this type of thing because then I can just go to the file that I know things are happening in so yeah. I'll move over to transcription so so the JavaScript this is my tie from the JavaScript and back into to Python so I have these kind of functions that bring in the recording that was done for MediaRecorder that I tie into the

browser and then now I can kind of build in this API request to Deepgram and so they have these parameters available and I have the content type and then I'm just able to request transcription from Deepgram I send them the audio file essentially and then I just pull in the the response that they get and I just isolate the transcript so I can use that for the note-taking portion so it's pretty thin I mean this is 58 lines the JavaScript was I don't remember maybe a couple hundred or maybe it's less than 200 so pretty pretty thin and this is all the the recording aspect of it kind of done it's just that JavaScript file and then this this Python file when you were building this did you do anything while you were building it to chunk out the testing portion were you like let's just get transcription transcription alone to work like let's just make sure that I can speak and it can transcribe it correctly did you like yeah step by step do that stuff or was this like I'm gonna one-shot this with an LLM it's gonna build everything for me yeah don't one-shot things like yeah I you

you kind of have to when you're building a complicated application in your head you have to split these things up that's gonna be your way to get a MVP proof of concept as quickly as possible and as accurately as possible in my experience if you try to one-shot these things you're gonna always go back and have to tweak and have to tweak and have to tweak and then eventually you're gonna refactor the entire app because you didn't split it up originally and then you let an LLM make decisions for you so yeah I it was nice here because I had these already split out as modules in my head like the Vera notes was the first module so I'm like okay let me work on Vera notes so let me make sure that the audio works do the JavaScript files tied into the Python and then just make sure I can I can record notes and then I have that part done and then I move over okay now let's do the review portion so I build the second module so you kind of have to split these up in your head just like you're modularizing this application in the code base you have to kind of modularize it in your head and work on them in pieces and then also give yourself success criteria to be like okay this is what I'm done with this phase fantastic I love it okay I

if you try to one-shot these things you're gonna always go back and have to tweak and have to tweak and have to tweak and then eventually you're gonna refactor the entire app because you didn't split it up originally and then you let an LLM make decisions for you

Walking through the notes module

I don't see any more questions so we can awesome we can move on so then let me move over to the notes portion and I kind of just want to talk about the audio piece so this is where like you're gonna see this this input audio data this is where audio is kind of traveling I'm trying to I've kind of bring you on the journey in the code of how this audio travels to each individual piece so now I had this reactive event for when audio data updates which is coming from from Deepgram and then now I I'm using all of these different functions helper functions that I've built throughout the way to you know transcribe that audio using Deepgram and then I can isolate that text and then I can update the UI to provide that editable text box that has the transcript that I I just did so this this is kind of a big function for this entire the note the note side because it's wrapping everything together so the JavaScript that went to the transcription to update the reactive value that then I can do this whole chain of events kind of in this function

very cool I have so many questions I don't want to interrupt you okay you keep going sure I'm trying to go that I know we don't have too much time so so maybe I'll end it at this point but there's there's a lot to go over and then and then again there's a there's a mini version of this available on my github which will get a link to that that is one for shiny real-time and one for Deepgram so maybe this is the last part that I'll I'll go over and then here this is where I talk about like storing notes right now in this demo version it's just storing to like a JSON but this is where you can kind of like think about this as a class where you can set these different values for what a note is right and then you can structure these notes into something that you can then send to different modules and it's important to organize all of these pieces into things that you can then make usable by different modules especially if you're you know this is not just a note-taking tool there's a note-taking tool that I can then map out to a schedule I can map out to a safety observation so making sure that the note itself is is something that is a structured is really important it's not just free text right so this is where I start marking my notes pushing them out making sure they're they're available to each individual module and then yeah that's kind of this point so I might end I don't know we don't have too much time are there any questions

Questions: Streamlit vs Shiny, and why Python

there are yeah we hadn't go to those we had one that was did you consider a streamlit app initially for this or any other question I'll add that's a good question I will say to that I am NOT the biggest fan of streamlit if you guys are big fans of streamlit don't let my distaste stop you I think the answer that is no not really I think shiny is to me is a lot more robust of a framework than streamlit is but it lets you be flexible in a way that streamlit well it's it's like it's more flexible than streamlit but it's it's just as easy to kind of set up then streamlit in my opinion especially once you get to I think streamlit has a ceiling in terms of how complex you get until it's like impossible or it feels impossible to get to the next step I think shiny just doesn't really have that ceiling which is why I didn't go with streamlit

I think streamlit has a ceiling in terms of how complex you get until it's like impossible or it feels impossible to get to the next step I think shiny just doesn't really have that ceiling which is why I didn't go with streamlit

awesome and I think I might be wrong but I think y'all have posit connect right we're like you can do it could have been a streamlit app because you can host all kinds of things on connect but one question that I did have as you were going was why the shift to shiny for Python yeah so was it just to use those like Python libraries that you were like these are all just really good at this yeah so for me I I'm one of those people who just doesn't care what language something is in like I just couldn't care less I can do both like left arm right arm the reason why I switched to Python was because Deepgram had an SDK that was Python only and so I was like okay that's probably enough of a limitation for me to think that making this in Python will be easier but that is essentially the the only reason yeah

and so I put this up on the screen just because this this QR code is going to link you to a github page that is going to show you the the two kind of demos I have one of them is with Deepgram it's just a note-taking tool the other one is with real time and it's kind of a fun one it's like a choose your own adventure like Dungeons & Dragons style game so I want to put this up just so you have time to scan it and and then go through it a lot of those the practices that are in those demos are very similar to the practices I use to build out both vera versions so you're gonna get a lot of information from that that's hopefully going to be actionable after this conversation so I really implore you to go look at those and then if you want to for the Deepgram it'll be easy because you can get the free API credit if you if you have a GPT license or API key that you want to use for the real time one feel free just keep in mind real time is not cheap so okay I'm gonna throw that out there that makes sense any other caveats that you want to throw out I did copy this by the way the QR code and I put it into the discord for posterity so we've all got it there I know some people are probably hopping Hebert says he's hopping for his next meeting yay hey Marcos I don't have my phone up to get that link you guys are gonna have to do it okay here's what I need I need a really quick way to like take a screenshot of a QR code and have it read it so that I can then copy the link the short link I always need this does this exist in some way that like I don't know about it must there must be something like it's gotta like I use um what do I already use I use snag it for like everything hmm I wonder if snag it has a way to do this that I'm not already paying for but yeah I want it snag it are you listening text me are you listening okay I think I say this like to Isabella if you want to share it oh yeah I mean yes I have my phone and I could pull it up so yeah yeah that's sort of like dance that I do between like oh let me scan this with my phone and then I'll like slack it to myself and then I'll put it in discord or something yeah okay yeah thank you so much this was so cool

I know that this is such a huge show-and-tell this is why when Isabella and I saw this like Lucas showed us this in like 30 minutes right and we were like oh my gosh this is so cool this is like the coolest thing that I've seen somebody do with shiny and have it actually be like a deployable thing that people at work can use and not not just a like a cool fun side project and so I really really really am so glad that Suffolk has been willing to be open and talk about it and you're allowed to talk about it and you're allowed to share some version of code with us I feel like that never happens so thank you so much Lucas and thank you to Suffolk for being willing to do this if there are lingering questions for you is it okay if somebody puts them into the data science hangout or they say channel and discord yes first of all this has been so much fun like it this is just I'm such a big nerd so being able to talk to like-minded people not to call you guys nerds is so much fun so thank you for having me and yeah reach out to me in any way I'm in the discord reach out to me on LinkedIn whatever whatever you need I'd love to be a resource for any and all of you as you need so thank you I hope I don't know if you can see all of the clapping and the like the like oh my gosh amazing and like mind-blown faces and emojis that thank you so much in in zoom this has been so much fun if you are curious about any other stuff happening at Suffolk there are like several comp talks that Blake has done he also joined us on the hangout ask Isabella and I we can we can help you find links to stuff for past things but that's it we've reached the top of our hour we've got to go so we'll see you on discord we'll see you Thursday at the hangout if anybody wants to come p.o.s. it slash DSH if you want to join for that and Lucas we'll see you on the discord hey everybody thank you bye everybody