Transcript#
This transcript was generated automatically and may contain errors.
Hello everyone and welcome to episode 13 of the Shiny Developer Series. My name is Eric Nance and as promised we are continuing our conversation with RStudio software engineer and member of the Shiny Development Team, Barret Schloerke. In this episode we're going to dive into one of the awesome techniques that you can use to separate out some bits of processing from your Shiny application into an API. And sometimes you'll have the good fortune of using an API that's already been made for you, perhaps from an additional R package. But what can you do if you're able to make the API from scratch and make it all within the confines of R? And that's what we're going to talk about today when we dive into the plumber package. And as I record this, by the time you're seeing this, plumber version 1.0 will be out on CRAN and we're going to dive into some great demonstrations of the really exciting features that are part of this brand new release. So let's not wait any longer. Let's have fun talking about plumber with our guest, Barret Schloerke.
You think about the evolution of a Shiny developer, even myself, I'm starting to live this, where the apps I'm creating are not just using things like R on the back end, but they may have to tab in some other system that's available via an API call. And you have been doing some really awesome work with plumber recently. I'd love for you to tell our audience a bit about plumber if they haven't heard about it and some of the things that are exciting you about plumber, especially how it relates to Shiny. That was a perfect introduction to plumber because it is a wonderful way to make your R code into an API. And as apps start scaling, you know, we need to outsource where the data is being hosted. And I think plumber just serves a wonderful use case of Shiny in that we move the data to a central location, central plumber API, and then that way you can have 20 different Shiny instances to handle all of your traffic, but they don't need to duplicate that memory 20 times. And if you have a counter, that counter can be shared in one central location rather than trying to coordinate between those different Shiny instances.
But, yes, plumber, it allows you to create web APIs that are, I think, are very useful for workflows as well. So, you can imagine, like, the Twitter API. So, when you query Twitter and you ask for data back, there's nothing to say that that's not actually done with plumber. I bet it's not done with plumber. But the user does not need to know how it's actually being executed. And I've heard of some fun stories where some statisticians were told to set up a database and have a front end or an API for it, and they said, oh, I just did a quick plumber API and, you know, it works and the boss doesn't need to know and it's very robust and it scales well. So, it's great. I'm going to keep using it. Let me jot that down for future work projects. That's a great idea.
Yeah. And I think it works so well, it's because the, you know, R is this, you know, it's that one second language. You know, there's that whole spectrum of how fast you can iterate on things and, like, JavaScript may be faster in certain cases or certain situations, but if the visualization takes you 10 hours to produce, you can't iterate on that. Whereas, ggplot, you can iterate every one, two seconds and come out with a new plot, look at it, change it, submit, do it again. And so, we can leverage that fast iteration possibility inside plumber to create web APIs that you're familiar with and it's also using comments that are very similar to the Roxygen style comments and so that we're in familiar territory and we don't need to relearn the wheel here.
Hello world example and the Swagger UI
All right. So, let's go ahead and copy this example in for a hello world, you know, plumber API. So, I have it here and functions, all the syntax highlighting highlights up. The new thing that we have to be aware of is the at post, at get, and at get, each takes a route and we've set our objects up. If we go ahead and plumb that using plumber's plumb method, it will come back with an API of what is going on. Ooh. All right.
So, in the new version, at PNG is being deprecated and we want you to be explicit in what's happening. So, we're saying, please make it a serializer with PNG. I'm going to save that. Let's rerun it. Cool. Now, I have my get route of echo, get route of plot, and post route of sum. And we can run that, typing it into our PR run, and the IDE will pop up with wonderful Swagger UI where we can look at our API and inspect it and test it. So, let's just do a quick echo test where I execute it. And the answer that I get back is the message is blank because I didn't provide any message. I think the neat one to see is we have A and B. So, I'll have, let's say, two and six. The answer should be eight. And the answer is eight. Perfect.
So, this is really cool. Like, you get started very quickly and that Swagger, you know, pop up is very helpful, too. I'll share a funny story from my end is that I was tasked with talking to a group at work about using Shiny and calling APIs from R. So, of course, I thought of plumber first. And I had not used plumber itself very much before that tutorial. I've always heard of it. Like, I've heard of it for a while. I mean, I've talked to Jeff Allen about it. I hadn't really put it in action. And it took me not too long to get up and running quickly. So, I definitely commend you for the examples and the documentation on the functions. And like you said, the R Roxygen-like syntax, as someone like me who's created some packages internally, it's easy to get into that workflow and to be able to test it out quite quickly. So, I was able to at least act like I've been using plumber for years. And really, it really was easy to get up and running. So, I definitely appreciate that.
The new pipeable API
Oh, you're welcome. Yeah, Jeff's work, it was really nice to step into that where, you know, we had this Roxygen style comments and everything was kind of set up. And I'm fortunate that I'm able to take it into, like, the next layer where we're starting to generalize it more and allow other people to opt into things. And one of the comments that people have been giving us is that the programmatic API is not that user-friendly. And working with James Blair, we've actually came up with a pipeable interface for plumber. Wow. I haven't seen this before. This is really cool. So, this will be coming out in the next release. So, we can start off with our default plumber object. And so, this used to be, like, plumber dollar new. And, you know, this is very non-familiar territory, like R6 behavior. And so, we wanted to kind of get away from that and say, okay, we're going to create a new plumber object. Let's add a get route of hi and hello. And this is the programmatic development of it. And then we can add a post route. And I could do this, you know, many times. I could have hi one, hi two. And, you know, that's great. I have multiple routes now. It's not just limited to one call. Yeah, so, it's great. I think it's a lot of fun doing that.
So, if we actually plumb this, it is an entry point. So, we'll do pipe. And we'll say directory equals this. And we'll run it. Ah, I didn't save my file. That's why it came up different. Happens to the best of us. There we go. So, I have hi one, hi two, echo. If I look at hi two, and I try it out, execute, I get hello world. And for echo, I'll try it out. I can execute. And there was no input. But that's correct. The post body was blank. So, perfect.
Speaking of the next release, one of the things that we worked on is plumber now has a new hex. So, your backdrop is going to have some wonderful legacy in that. It will have historical plumber hex sticker. And all the new ones will start having the pipes. Very cool.
New documentation and serializers
Also, involved with the new release for documentation, everything has been converted to pkgdown. So, we have searching available now. And also, we have a reference of all the functions that will be added. Especially the PR methods for the pipeable interface. Very cool.
Yeah. So, looking at these, all of a sudden, we have a lot of serializers. More than what is in the CRAN version. Yeah. CSVs, TSVs. Typical JSON and unboxed JSON, I believe. But now we can do RDS files and Feather files, YAML files. Yeah. It's really great on how we can do that. So, I'm just, you know, letting people know that there's more options out there. So, we'll just say, like, at serializer, JSON, at serializer, Feather, and it will be engaged. So, serializers, if we take a look at those, they're what is used to return the data back to the user. And so, if you want your data to be, you know, it's stored in R as a data frame, but I want the user to receive a CSV, then I will tell, hey, I need my serializer to be CSV. And it will be registered as CSV. Or TSV or JSON, you know, even YAML and Feather. And I think it's really powerful in how that can work, because it's just, you know, a one-line change in our hello, and that here I can say, at serializer, when this one's an image, but here I could say, at serializer, and we'll do, this isn't a data frame. Well, let's make a new route instead. So, at it slash demo, and I'm just going to return the at serializer and say, let's do YAML, right? YAML. And this will be, you know, inputs. And I will say, let's do the old faithful IRS, right? All right. So, if I run API, then the demo is here. And if I'll try it out, execute, then this is YAML code. Wow. Very nice. Very exciting.
I see a lot of opportunities just with that serializer alone, or maybe you have to have some kind of service that produces the YAML that goes as input into some other pipeline you have. I'm thinking R Markdown or something like as a YAML header for that. I mean, there's lots of, it sounds like, yeah, these serializers are now saving us developers more time to not have to worry about, okay, now I have to massage this output in a certain way, have it redownload or recompile it or something, but that the HTML response is the way we want it coming back, and saves us some bookkeeping on there. So, I could see lots of potential for this.
New parsers
So, speaking about chaining those pipelines together, we now have the ability for plumber to use parsers. Typically, plumber had not allowed for anything other than a form or JSON, and now we have opened up the floodgates. So, you can register your own, like, we have a multi-part parser, we have our regular JSON and form so that people submit, but we've also added in extra things like if we believe it's going to be YAML input or CSV or TSV, feather input, and we even have RDS as well if you're ingesting an RDS file. I know a lot of users had, you know, they had their own way to parse the data, and so now they can register parsers and add them into their routes and have them be very specific to a particular route as well. So, that way, not all routes are taking in YAML, but just the route that you want.
Yeah, I think it's going to be really great to have that. Yeah, I'll try this out, and when the new version's out, I'm going to give this a spin. Very cool. Whatever things are you excited about for the future of plumber?
So, with the serializers, I know I'm kind of jumping here, there's a method called as attachment, and all of the serializers right now work with as attachment, the plumber serializers, at least. And I think this is really exciting. As attachment allows to add a content disposition header to your return value. And what this means to, you know, users who want to ingest it, is that if I'm visiting this in the browser, the browser will try to download it as an attachment
rather than automatically displaying it in the browser. Oh, that could save us a lot of pain. If it's trying to display something that's totally cryptic to it, it'll just kind of bomb out. That's very cool.
To even take this one step further, the content disposition can also allow you to give a file name. So, even if I'm going to a route of like, slash time, I can save it as a file name that has like the date in the file name. Yeah, right. Something that doesn't... Generate that. That's very cool.
Yeah, something that doesn't agree with what's going on. Oh, I don't have that one.
Okay. So, there's a demo that I can do where we download things as attachments. So, before I run this line of code, I do want to expose this new function of plumb API and also available APIs.
So, within plumber, we will have the ability to have a consistent way to check for plumber APIs. And this is new. So, currently, there's only plumber. But if I were to run this, like, hopefully in like a year, I'll have multiple packages with different plumber APIs.
Oh, that's really slick. Okay. Yeah, I like it.
And so, we're asking that people put their plumber APIs inside the Inst Plumber folder, and then each subfolder is considered an independent API. And that allows you to be... Yeah, I could see this really advantageous for those of us in the enterprise. I want to start more building these APIs from plumber, but have like a way to index them somehow or assemble them nicely on somebody's installation. This seems like a good first step to that.
And you can also provide a package name as well that we will subset and look for that information rather than scouring your whole library. Yeah, very nice.
But using this, we can say, okay, I'm going to plumb an API from the package plumber. And the one that I'm going to plumb is 16 attachments. So, that's this name right here. And then let's go ahead and run it. Because this is so browser dependent, I'm going to pull it up because Swagger doesn't show off like the full aspect of it. But the routes that we have is name, no name, and inline.
So, if I come here and I go to inline, this is the one that we expect. I'm going to see the time in the browser. Okay. Then we have no name. And this downloads a file of no name. And I've done this once already. So, this is the second file that I've downloaded. And so, that's, you know, a little bit more useful. But I think the one that I find useful is that if I go to the name route, I should expect typically that it would be downloading as name.txt. But with the content disposition, I can download it as time.txt. Very cool. So, you have full control over that.
Full control for each route at runtime. Neat. All right. So, let me close up that demo.
Asynchronous execution in plumber
Another neat one coming through is the ability for asynchronous execution in plumber. Oh, this is right up my alley because I've been starting to interact with more than one API at a given time. And I'd love to see how this plays out. Okay. Let's see what you got.
So, plumber, you know, it's R, right? Underneath the hood. So, R is by default single threaded single core. And, you know, even if you're doing all the work that you could do, you'd only max out that one core. And if you had a route that would take, like, let's say, ten seconds to compute, you would be stopping everyone else from using even the routes that are basically instantaneous.
And so, we've done work in the internals to allow any of the hooks or the routes if they return a promise or future object, that we will execute them in asynchronous manner. But you don't need to know if your prior one was done asynchronously. And so, you just need to worry about your spot in the execution chain. So, if we look at this, if I go to slash future, this should sleep for ten seconds and then give me a result of the time. If I go to slash sync, it will just give me the time immediately.
In the current CRAN version of plumber, this would block for ten seconds and then this guy would solve immediately afterwards. So, the times would be the same. And now that we're doing it with future, this is actually offloaded to a different core and I'll get a different system process identifier and I'll get back with the time. But it will free up and allow me to do this multiple times.
So, okay. Let me go ahead and run this API here. So, I have plumb, plumb future. Great. And that's loaded up. It takes a second because future and it's multiprocess. It's kicking on the cluster. And let's open up these two routes and we'll try it out and try it out.
Okay. So, if I hit execute for the sync route, we should get the process time and, you know, the PID is ending in 31. If I hit execute again, still 31 and the time goes up.
If I come down here and we say execute, two, three, four, five, six, seven, eight, nine, ten. And it's a different PID. So, we know it's done in a different process.
So, now let's check to see if we can do one at the same time. So, if we do execute, this is loading. And I can come up here and say execute, execute, execute. And it's still loading. And it finished at 34. And this one is going to finish at response body. Sorry. This one finished at 36 seconds into the minute and this one finished at 40 seconds into the minute. So, there's definitely no way that this was held up by the future. And so, now all of a sudden plumber is multi-core and, like, it's great. You don't need to be blocking on everything.
Wow. I mean, the use case I'm thinking of right now and right now the APIs I'm thinking of aren't a plumber API, but if I built them like this or converted them, I would love to have some offloading of, say, a model fit as some other server that's running via plumber would take that and run a model fit. That's going to take maybe a long time. Who knows? But then you might have another that's running synchronously that's looking at a log of that model fit kind of periodically. But they can run at the same time instead of one blocking the other. That's what gets me thinking this would be a great workflow for.
That's what gets me thinking this would be a great workflow for.
Absolutely. Especially if you had a three-route system. I've talked about this in an issue in plumber. I like the result of a three-route system. One is submit, one is status, and the other one is result. Exactly. Yeah. I've been doing that kind of ad hoc in one of my production apps.
And when you submit, all you'll get back is, like, a thank you and an ID. And it just said process submitted, here's your ID. Then you'll take that ID and use a get route or a post route and say, what is the status for this ID? And using something like callR, you can read the logs of what's happening in that R process, in that background R process. And then also, for your results, you can then say, if the value has been finished, then you can do, like, callR retrieve value. That's a little different than how this is working, but I think both of them have very, like, direct impact on using all of the cores in the machine.
Yeah. I definitely want to explore all of that, because it would make my life easier as an app developer that's integrating these services and figuring out the best way to manage them, but not, you know, bog down the app experience at the same time. This is certainly something I'm going to do some homework on after this, for sure.
Custom API UIs
The last demo that I want to show off is talking about this UI. So, this UI here is done by Swagger. Thank you, Swagger, for getting the conversation started and having the Swagger specification, which got transferred into OpenAPI, and having OpenAPI be that open source collective initiative there. But, you know, there's no need to force Swagger to be the only one that can view this. So, similar to the register, or sorry, the parsers and the serializers, we can register a new API UI.
And so, if I have this one, and I look at this file, I can say library plumber library Rapidoc, and Rapidoc is a package that is created by the GitHub handle Meztes. His name is Bruno Tremblay, and he's been absolutely a great help for this last release of plumber. Done a ton of work. We can thank him for all the parsers that have been added in. You know, his contribution has been just immense for this last release. Another win for open source. That's awesome. Absolutely. Absolutely.
And so, if we go ahead and do this, so we could say install remotes, and then I say library plumber, Rapidoc, and then I'll just go ahead for the demo and say PR run, because I'm just running it from the command line, that now the UI is done using Rapidoc. And we get the plumber logo. It's just a real treat to use this one, I think. And, you know, the defaults that they have in here are a little bit nicer, I believe. You know, we can insert the default of 10 for the mean, and when we try it, you know, it's a little bit skinnier, so it's not as tall. So, it's just there's some nice aspects to it. But by default, because people are very familiar with Swagger, we're going to keep shipping Swagger, but now we can insert our own. We can even actually use another one called Redoc instead of Rapidoc. So, there's more.
Well, it's nice that you're giving flexibility to us on one that we think is most intuitive for whatever situation. Now it's not just Swagger. It's like the only one. So, this is really cool.
So, let's try this one with Redoc. So, I'll library plumber, library Redoc, and then I'll plumb the API, set the UI to Redoc, and we can run it. And it's captured it there. And I think this one looks a little bit more professional and sleek. Has a nice default. But, you know, it's a little bit more like, you know, thin typing. It's different. But my preference is towards Rapidoc.
Yeah. I mean, they're all great alternatives, but it reminds me a lot about how going to something like R Markdown in the ecosystem, you have so many templates you can choose from for your R Markdown docs. Now we have templates for our API documentation. That's great that we have these two additional alternatives and perhaps more down the road that we can use to share with our colleagues who maybe are used to API documentation, but it's just a different spin on it. Very nice.
Integrating plumber with Shiny
Yeah. I think, you know, certainly what I want our listeners or viewers to get from at least my perspective, you can chime in too, is that plumber itself, certainly you have now given us the ability to do any API with just our code, like no matter how complex or simple it is. To me, it still is quite a valid thing with Shiny itself, because now you have offloaded a lot of that business logic to a different service that maybe those in your company or your organization love to use your Shiny app and maybe have another department is like, well, we kind of like your Shiny app, but maybe we want to do our different UI to it.
But then you can have this be decoupled from the app. So you don't have to worry about messing up your app code to please somebody else. But it's more about just having one place to update the backend and then many different front ends can call it. So like, I definitely have plans to use plumber as API, you know, mechanisms for a Shiny app at work. But now I can say, well, it's saving me a lot of time from a development side to have that developed separately from the app itself. So certainly some of the advancements you're making there, that asynchronous stuff, especially I'm going to make.
Yeah, the async is a lot of help by allowing more routes to be calculated at once and not be bogged down by your slowest route. But as he said about connecting to other interfaces, James did a very good presentation on how plumber was a back-end for a mobile application to his friend who wrote a mobile app. He's like, all he knew about R was how to spell it.
But plumber was the data source that was backing that application.
But plumber was the data source that was backing that application.
Yeah, I remember watching that talk live. I was amazed by what was possible with that. So certainly I'm going to be playing more with plumber after this as well.
Well, as you can tell, I really enjoyed learning more about plumber and these awesome new features. And you can certainly see the enthusiasm and excitement that Barret has with this 1.0 release
that I think is going to be a very vast improvement even on an already great Plumber package. And I'm already excited to try out a lot of those new things that we saw. In particular, although they're all great features, but the one that had the most, you might say, light bulbs in my head thinking about trying next is the asynchronous processing. Because I've had multiple situations recently where I'm not just calling an API in one at a time. I'm dealing with multiple ones, sometimes at the same time. And I'd love to be able to manage that more effectively.
So this will be a very cool thing to try. And I'll be interested to see what I can come up with. Certainly, I'll be sharing that with all of you as I explore my journey on making my apps a lot better with these great improvements. So I hope you enjoyed that deep dive into Plumber. And we will have Barret back one last time, at least for this series next week, as we dive into some of the nuts and bolts around the LearnR package that has become such a mainstay in helping teach others how to use R or specific data science principles with an interactive interface. Barret's done a lot of excellent work on that front. And we're going to hear more about that in the next episode.
Closing remarks and how to stay connected
So until then, if you want to catch up on our previous recordings, both from the webinars that we conducted last year, as well as the recent interviews we've done in 2020, the best place to go is our home site at ShinyDevSeries.com. The YouTube channel will be linked directly there. And we definitely invite you to subscribe to our channel for future updates. And if you like the videos, please give us a little thumbs up to let us know that you're enjoying it. And we have a lot of big plans in the works for our next episodes. And I can tell you right now that I've been making a huge investment in both time and, in some cases, money with the production quality. And very soon, you're going to see the fruits of those labors in our upcoming guests that we have coming up for you in the next part of the series.
So until then, I'm Eric Nance. And feel free to drop me a line if you enjoyed this episode. I can be reached on Twitter with at the R cast. We also have a contact form on ShinyDevSeries.com if you want to leave your feedback on the series and any comments you have. So until then, we'll see you next time. Thanks a lot for joining. Bye-bye.
