Transcript#

This transcript was generated automatically and may contain errors.

Hello everyone, thank you for tuning in to episode 12 of the Shiny Developer Series. My name is Eric Nance and I am the host of this series and thank you so much for joining me.

We are kicking off a multi-part series with RStudio software engineer and member of the Shiny team, Barret Schloerke. We talked about so many great ideas that it couldn't fit into just one episode. We lead off this series with learning more about Barret's background and his journey to the RStudio team, as well as learning about the excellent reactlog framework with some great demonstrations.

Let's not wait any longer. Let's connect with part one of our series with Barret.

Well, welcome everybody. We have another exciting episode of the Shiny Developer Series on tap. And I am pleased to be joined by RStudio's software engineer, Barret Schloerke. Barret, thank you so much for joining me today.

Ah, thank you for having me, Eric. I really appreciate it.

Yeah, we're really excited to talk about some really cool topics. So I thought first, why don't you give us a bit about your background and how you discovered R and then by proxy, the Shiny package.

Barret's background and journey to RStudio

Yeah, so my background, I am a computer engineer undergrad and I got that at Iowa State. And that is actually where I discovered R. I was in a freshman honors seminar and I just happened to be in a class taught by Hadley Wickham and Heike Hoffman about data visualization. And each week they offered up a new visualization tool to look at data. And we just, you know, talked about that. And at the very end of the seminar, he goes, hey, if anyone like a research position doing stuff like this, let me know if you're interested. And I was like, I can get paid to do this? This is great.

So I contacted him and said, this sounds great. I want to do this. And I joined up with the research group with Dai Cook, Hadley, Heike Hoffman, and Michael Lawrence, who were all there at Iowa State during that time. And it was a lot of fun. And I just did that as my side research while doing my undergrad there at Iowa State. To me, it was software engineering. But to them, it was statistics. And it, you know, just the application is towards statistics. And I think that is really useful in having that context of statistics. But in, you know, deep down for me, it was just coding at the heart of it.

But anyways, I finished up at Iowa State. And then I did a quick pit stop out in San Francisco for a year and a half. And I was a front-end engineer for a company called Metamarkets. They've since been dissolved. But I was writing mostly in JavaScript. And I was working on a project called DVL, which is Data Visualization Legos, which wrapped around D3 and just made it a little bit smaller interface rather than the very complex and complicated D3 interface. And it was nice. It was really good. But that kind of gave me that foundation of reactivity. And if I change this input, then I'll change all these outputs. And so that gave me a pretty strong foundation.

So after San Francisco, I continued on into grad school. I did a PhD in statistics at Purdue University. I was under the direction of Dr. Ryan Hafen and Dr. William Cleveland. Ryan is more known for his bokeh work in R. But I personally am very excited about his Trelliscope and Trelliscope.js work.

Me too. I'm actually using that for a very important app as I speak. And my VP that I showed it to absolutely loves it. So I'm so thankful that Ryan made that.

And that was actually, we could say, one of my first introductions to Shiny. Because Trelliscope was written in Shiny. And we had done the proof of concept and we weren't approaching things properly. And we said we need to spend the next 500 hours or next 100 hours optimizing this to try to turn it into an HTML widget. And that way, we can then have multiple Trelliscopes and it's a lot faster. And you can port it into just a regular document.

But one of the things that came out of that while I was there is I built a package called Autocogs and it created automatic cognostics, which are used inside Trelliscope. So if you have a ggplot object, there's no reason I can't inspect that object and say, oh, you're doing a linear model with the scatterplot. Let me give you the range of your X and Y values, your slope and intercept for your linear model and your R squared value. And then that's automatically added to your Trelliscope so that you can sort and filter and do all of that.

So if you're using it for your VP, check it out. I know my wife is using it for sorting different traffic data that she's using. So, yeah, it just makes my so much easier. I don't have to build all this up from scratch. It's just let them do the organizing they want and then, yeah, drill down if they want. It's very cool.

Yeah. And so those are some awesome people to learn from. I'm a bit jealous, you know, that's a lot of great knowledge in that group.

Yes. Thank you. It's also, as you know, happened to be at the right place at the right time. If I would have been at a different school or a different year, you know, I would have missed that seminar and who knows where I'd be. So, yep, yep. Life always works in a serious way sometimes.

And then, yeah, maybe tell us a bit about how you found your way to the RStudio team.

So since I had had tasted, you know, work life and I wasn't in school anymore, like I was really going to go back to school and finish that up. And but with the intent that I was going back to industry. And so I try to always position myself that even though the work that I'm doing is statistics for the PhD, it's going to have application towards industry and towards, you know, like this is a chance for me to build a two or three year resume that I can use to like say, hey, I am your candidate to work on on the RStudio team. And so I think it worked out, worked out well.

So, yeah, I had a conversation with Hadley and, you know, because we're good friends at that point and then got a got a conversation with Tarif and then had to earn my way through through Joe and the team.

Yeah. You've been a wonderful addition. And this actually is a nice lead into when I first met you. It probably wasn't that long after you joined, but you and I spoke at the same RStudio comp session. It was a shiny session where I was talking about the effective use of shiny modules because I've been a big proponent of modules and you came right afterwards. And you probably saw my face in the front row. I was kind of decompressing from my talk because that's the first time I spoke in front of so many people. But then you gave a great talk about some excellent work you've done working on a package called reactlog.

And so I want to start with maybe showcasing some of the work you've done with reactlog and why it's such an important tool for shiny users to leverage to understand their reactivity and their shiny apps. Maybe we could go to the demo now.

Introduction to reactlog

Absolutely. So reactlog is a way for users to look inside the black box of shiny, and I think it is so great because I've seen many different like napkin drawings by by shiny developers where they're like, I believe this input connects the output and it does it this way. Here's my napkin.

Absolutely. So reactlog is a way for users to look inside the black box of shiny, and I think it is so great because I've seen many different like napkin drawings by by shiny developers where they're like, I believe this input connects the output and it does it this way. Here's my napkin.

Guilty as charged on that one.

Yeah. And so I knew there's got to be a way that we can record this. And Joe had already implemented something beforehand. But it, you know, after about your toy example, it didn't really scale that well. And we needed something that was a full fledged network graphic and you could search, you could go backwards and forwards easily. And so I just was given a clean slate and said, hey, implement this.

So for reactlog, what it will do is that we can look at this and this is just a replaying GIF, but we can imagine an example like Pythagoras theorem, right, we have our input A and our input B, and as I change the values of A, in this case, it's calculating, it will calculate A squared and B squared, we can then calculate C squared and then, you know, we take square root to C and then we can display it in our output as text or verbatim, just so we have some multiple outputs.

And you can see here, if it's green, then the value is ready and ready to be used. If it's dark gray, it's invalidating, like it's trying to get rid of all the people that depend on it. If it's light gray, it hasn't been calculated yet, but exists. And if it's orange or this orange is yellow color, it's calculating its value.

Live demo: Pythagorean theorem app

So I will do a quick demo here. So I have that app up here in the RStudio IDE, and I'm going to go ahead and run the app. I'm going to do it in the browser just because the key bindings are just a little funny. And so I can change the value to 5 and 12 and we get, you know, the 5, 12, 13 triangle. Great.

And if I open up reactlog with command FN3 or F3, then I get the reactlog right away. This shows up at the first time that Shiny is idle. And in this case, all of our outputs and inputs have been calculated.

So, you know, this app isn't too exciting because everything is working as expected, but you can see when I change the value to 5, then now all the downstream dependencies will start to invalidate. B and B squared don't invalidate because they weren't changed. And then once everything's invalidated, we then calculate our outputs like the output C text and verbatim. And so they calculate upstream, retrieving the values as necessary and only calculating what's necessary. And then when we change B, the same pattern happened as well.

The part, though, where it becomes really useful is when you believe something should work and it doesn't. And reactlog can't really tell you that your app is wrong, but it can tell you what's going on.

Yeah, the first step is always knowing where the problem is. It's very hard to find.

Yeah. So we have our mental model of what things should be done. OK, so this one is where the Pythagoras theorem is broken. We have three, four and a five triangle. The C value here is not five. So that's not expected. Right. And if we change A to 5 and B to 12, we should have the 13 as our answer. And C is still wrong. And this one is not 13. So that's not good.

So let's open up a reactlog, the command FN3 on Mac. And we see a different shape than what we had last time. Last time it was two inputs and two outputs. Now it's kind of a cone and this one isn't even connected to anything. So there's a reason why that C verbatim is not being correct. And then we can also see that C squared pulls in A squared, B squared. Good. But B squared pulls in A that does not pull in input B.

So we could come to our code and validate that and say, ah, B squared does pull in A. That's a mistake. And C squared here is isolating on B squared. We didn't see that, but this answer is just five. So we should probably switch that to answer and then we can move on.

So I think you can you can kind of see where, yeah, in this example, you're able to fix them fairly quickly, but you might have in a production setting, React is a whole bunch of inputs and outputs connected. And this having this visual just helps cognitively figure out where are the disconnects where we're just looking in code.

It's it's hard enough to match up your input names of the right reactive references and all that. So this is certainly something that had this been around in this form when I was first learning Shiny, would have helped me understand reactivity a lot better.

Yeah. Having that visual tool of knowing that I change A and this is how it changes downstream is just invaluable. It's not just magic anymore. There's a thing to explain it.

Reactlog on a complex application

So talking about the larger applications, this is actually the reactlog for Joe Chang's Cranwhales application that he built. Yes. And so I think we can probably just kind of link to that at some point. But we'll have that in the show notes for this. Yep. And but what it does, it has like a plot and it also has some numbers and it tries to tell you who is the major offenders for downloading all of the CRAN packages and causing all the bandwidth issues.

And this is where we start to get a little bit more complicated. It's a lot taller than it was last time. And it has, you know, one, two, three, four, five, six, seven dependency layers. So it's quite a complicated application.

But one of the things that I like is that we have our plot object that we're putting into the downloads per hour. And if I double click on it, I can actually search into its family tree, both, you know, ancestors and descendants. So if I even calculate maybe our whale downloads, I can see all the things that are involved with that whale downloads.

This is really useful when you have many different reactive objects. I've seen some apps have like 2000 reactive IDs on load. Wow. And yeah, it is. And so when that happens, if hopefully everything has been labeled properly, but I like

to use the search bar in the top right. So I'll say like whale and now two or three things have been been hit by that. If I do whale underscore now, it's just that whale underscore downloads. Oh, that's amazing to save time. And so I'm just kind of like poking and hoping for the best that you find the right areas. Yeah, absolutely.

And when I move forward in the timeline, I'm using the arrow keys on my computer or I can use the buttons up top, depending on how far you want to skip. The jumps will only be for this particular subset of elements. So if you see the number up top, it goes from 201 directly to 207 because maybe 202 and three were some things that weren't involved. OK, but it's nice because then you only are looking at your subset and you can see what's going on. Right.

I guess this begs the question, obviously, yeah, the whales app is one of the bigger ones you could use this with. I mean, I know you've been building this with scalability in mind, but have you encountered apps that are almost too big for this yet? Or is it we're kind of in the cases that you've tried this with?

I have encountered a few apps and it's very encouraging to see that I think it's really neat. It's too big when it is not filtered. When you filter down just a little bit, you know, if I do this animation here and there's 2000 edges that you're moving around, that's a lot of computation time. And so maybe I've thought about opening up settings to say, you know, for this visualization tool that we're using, the library, maybe we add in the option of like do not animate because then it would just jump jittery for a user experience. It's not useful, but for scalability, it's really nice because then you just paint once rather than painting many, many times. Sure. Yeah.

But it's it's better than a very large napkin, that's for sure. Yes. Yes. And this one you can inspect at any time and not have to worry about your drawing skills to boot. Absolutely.

The anti-pattern demo

Very cool. Other, yeah, let's let's see what else you have for reactlog, did you want to?

Yes. So the last one that I want to show for reactlog is to show off, you know, show off a phrase that Zhou Cheng had had coined called an anti-pattern and users naively will do this. And it's not a bad thing. The way that memory management works within Shiny, things are being cleared up. However, reactlog really exposes this pattern.

So imagine we have an app that if I go ahead and run it in this, all it does is says click me. And then we get a histogram of the count of how many times the button has been clicked or, you know, our unit of the button clicks. So it's pretty simple app. The goal is to say when the button is clicked, draw a new graph.

Now, if I was first learning Shiny, I would probably jump on to the the function called observe event and say observe every time the button is clicked, and then after we, you know, once the button is clicked, we will then render a plot given the histogram of the button click and save it to the output value. I mean, logically, it makes sense. Um, I think the rule of thumb that we should try to enforce with ourselves, though, is that any render method should not be inside an observe method. Plus 100 on that one.

Um, I think the rule of thumb that we should try to enforce with ourselves, though, is that any render method should not be inside an observe method.

And so what we can do instead is have a temporary variable such as click count, which is an event reactive saying like, hey, listen to the button. Let's do some calculations here. I know this is, you know, just returning the identity, so it's not too complicated, but we could say do some calculations. Once we have a click count, then we can say, hey, on render plot, listen to that reactive value and, you know, redraw my histogram. And now we're only calling this function or this setup only one time. Right.

So if I open up this in the browser to look at the reactlog, so I will click once. And let's do the reactlog here. So I've done this one click and, um, let's, this is the final state here in that it finishes and the, the butt, the button is done. Awesome.

I'm going to mark this time point with a shit command shift, uh, FN3. So you can see the mark time point. Oh, interesting. And I'm going to click again. And now we have two clicks. If I refresh that reactlog, we'll start where we mark that time point. And now we can see there's this whole nother, you know, output bad pattern, output bad pattern, and, you know, things are getting filled out great, but the original output bad pattern is not being calculated. Only the new one. Yeah.

And this is showing off that anti-pattern. So I'm going to, um, refresh or sorry, not refresh. I'm going to click. I think many more times I'm going to, uh, uh, look at the reactlog again, and we will see that the graph has now exploded. And. Adequate description. Yeah.

And only one output bad pattern is actually getting the plot saved in the end. The, the prior ones have all been like, just, you know, non-existent anymore. So ours memory management will actually have cleaned those up, but the reactlog says, nah, no, you're not doing it. Right. You know? Yeah. You can't rely on that crutch forever or forever. And I, I, I remember my early days of Shiny. I was, you know, not cognitive enough to know that that was a bad idea to throw these different renders inside these observed blocks. And now I think it wasn't until after Shiny DevCon that I started to see the light a little bit from Joe Chang's reactivity talk back then, but like I said, this tool here, what a great way to kind of put this in a visual way that you can share with your, obviously yourself, but if you're working with a team of developers and maybe they did do some kind of hackery around a certain output and you can say, well, this is what's happening in terms of the backend of it, but we can't really reason with this as much as if you utilize reactivity, the, the optimal way where you minimize the time these things are invalidated and everything like that. So this is an excellent example. I definitely appreciate you sharing this.

Future improvements and closing thoughts

Oh, you're welcome. Uh, yeah, the, there's some things that I would like to like help out with this though, in, in that, um, the reactlog side, not necessarily the anti-pattern, but for reactlog, you know, we know that this output plot, you know, is a plot. And we know this plot object is a plot. Sure. So maybe it would be really nice if we could save these output values and I just don't have access to them right now. Okay. Yeah. That would be really handy to see. Yeah. What the actual changes are to these objects. Yeah. That's cool.

Yeah. Cause if you had like, you know, we know this is a ggplot2 object. We could just put the image in right there. Not, not a problem. The only thing I have the values for is the reactive values at the top level. Where I can say, oh, the pixel ratio is two. Uh, the input button is 11 clicks, you know, but I can't tell you the value for this guy. So.

Well, there's always future work for everything and that's certainly some, but it's excellent already. So I think it's a great way to learn more of what's happening behind the scenes. Thank you so much to Bear for taking us on this great journey through how reactlog can help give you insights into the reactive process of your Shiny applications. I know I will be using this feature much more often in my apps, big or small for that matter.

In our next installment of this series, we will go back to school. So to speak by examining the learn R package as well as new developments for the highly innovative plumber package. Until then, you can find detailed show notes for this episode on our main site at shiny dev series.com. And if you enjoyed the content from this series, we would greatly appreciate a thumbs up to this video on YouTube and be sure to subscribe so you can watch our future episodes a moment they are released. With that, we will close up shop for episode 12. Thank you so much for watching and we will see you next time. Until then, have fun creating Shiny apps. Bye, everybody.