Transcript#
This transcript was generated automatically and may contain errors.
Welcome to The Test Set. Here we talk with some of the brightest thinkers and tinkerers in statistical analysis, scientific computing, and machine learning. Digging into what makes them tick, plus the insights, experiments, and OMG moments that shape the field.
In this episode we talk with Trevor Manz. Trevor started out in chemistry and computational biology, before moving into visualization research, where he ran into a problem. Researchers could build these sophisticated tools, but getting them into the hands of people analyzing data was another story. That frustration eventually led him towards Python notebooks, and to creating AnyWidget, which makes it easier to build interactive tools that work across the computational notebook ecosystem. Today he's a founding engineer at Marimo, where he's pushing that idea in a new direction. If notebooks are such a useful place for humans to explore data, could they also be a better place for AI agents to do it? And what would AI agents need to use notebooks effectively? We're joined by self-proclaimed notebook skeptic Hadley Wickham, which I think makes for a fun discussion. So with that, Trevor Manz.
Trevor, welcome to The Test Set. So you're Trevor Manz, scientist, programmer, and as you say, Wisconsinite, which I'm excited to dig into. You created AnyWidget and you're a founding engineer at Marimo. Yeah. Thanks so much for coming on. So happy to have you.
Yeah. Thanks so much for having me.
And by way of introduction, I guess I'm, I'm Michael Chow, the host of The Test Set, and this is Hadley Wickham, co-host and chief scientist at Fawcett. Yeah, sure. I'm so excited. I know you've been doing a lot with Marimo and notebooks, and especially with Marimo pair. I'm, I think, yeah, I'm so curious to talk a bit more about Marimo and notebooks. Maybe just to kick it off, you could explain a little bit about what Marimo is and kind of like who uses it and what, what excites you about it?
What is Marimo?
Yeah. So Marimo, Marimo is like a new kind of... Marimo, I'm so sorry. Yeah.
So Marimo is a new kind of Python notebook and I like to describe them as a delightful reactive Python notebook. And the first piece is that we're a delightful notebook insofar as that we've been able to look a lot at how notebooks are used in practice and then really think about creating a very built-in user experience for many different ways that notebooks are used, like traditional notebooks are being used. And then the second piece there is that we are a reactive notebook. So similar to observable notebooks or Pluto, which are sort of from the Julia ecosystem, Marimo does not... Marimo understands the relationships between your cells, not based off of the ordering that you wrote the cells, but the actual data flow. So where your variables are declared and how they're used to find the execution order of your notebook. And that leads to eliminating this kind of hidden state that can come from having sort of a traditional Python REPL or like a traditional Python notebook. And then the third piece is that we are a Python notebook. So we're like only focused on Python. And that is all the way from a lot of the user interfaces oriented around data structures that are in the Python ecosystem. We have things like a built-in package manager. So for when you're trying to work on your data, we can sell packages for you. All the way down to the file format itself is not like a JSON file. It's actually a Python script.
Yeah. So, and I appreciate you helping with the pronunciation. I feel like this also came up at PyCon when I saw y'all that this is the perfect chance that I'm going to work Marimo in and Marimo out forever. So I'm glad that we have this to practice.
So we like to say that Marimo is like, you know, we're an open source computational notebook that's used across like industry research and education. And then beyond that, I think as well from being able to look at the different ways that people are trying to use traditional notebooks, we've also thought a lot about, you know, you build up a notebook, but like what are the directions that you might want to take it? So some notebooks look more like doing like data pipelines and those you can run as Python scripts and maybe have very few visual elements. There's some that maybe are more in this, you know, world of like literate programming where it's both like your code, your code is very forward. And then you have pros explaining and contextualizing the code. And then we have, you know, because we're a reactive Python notebook, the cells can react to user interactions. And so there's a mode within Marimo that we call like app mode that you can just hide all of your code and then you just have the outputs. And so for that set of users, it's more like presentational or, you know, showing people almost these like data apps, but without having to show the code.
And do you think, is it like mostly like data people broadly construed or are there people doing like non-data stuff? I'm sure there are people doing non-data stuff, but is that like a user base you think about?
Yeah, I guess one thing I've noticed by introducing like a reactive experience to an ecosystem like Python, we do have users that are, you know, trying to build, like they've sort of maybe learned a little bit of Python in their education and now they have sort of a toolkit or that background with that coding language and now a toolkit that allows them to make like programs a bit more interactive. So we do see things like games, I guess, being built inside of Marimo. But I'd say in terms of like the scope of where we are taking the project, it's definitely oriented around, you know, generally folks have some unstructured data, they're going to like load it into memory and like, yeah, do something with that data.
I guess like more, sorry, more concrete, just to remind me of like, do you know processing? It was like an art, really like a framework for doing kind of art explorations. And it feels like you could definitely drive a reactive notebook in that direction as well for doing computational art. That would be a cool application.
Yeah, we definitely have a community member, I'm forgetting his name. I can share a link, but he's a mathematics teacher who has all these really cool generative like art displays that he would normally, I think, historically in the past would write these like really these scripts in Matplotlib that, you know, demonstrate some sort of like a lot with like fractals and now like being able to grab that plotting library, add a slider and now you can like play around with that fractal.
Hadley's notebook skepticism
I will say I love the app mode. I feel like app mode is really killer to be able to go from a notebook where you're analyzing to this kind of like display more dash 40 setup has been super cool.
Hadley, do you use much notebooks? I know like in R it's a bit less common. I'd be curious maybe just for context to hear some of your experience.
I think like I've never really gotten notebooks and I kind of feel like there's this. Like when you start learning, like you either learn a tool like R with like a ripple and you learn something like R Markdown or Quarto where you like kind of render a document or you go down the notebook path where it's kind of like intermixed things that you run and text and it feels like once you go down one of those paths, like the other path just feels like weird and unnatural and like it's not like obviously like people can be successful on either path.
So, yeah, so like when I yeah, like I played a bit with Marimo and I just felt like really strange, especially the especially the fact that the the order of the chunks in the document does not matter. Like it's only the order of dependencies between them, which that did feel a lot more like this literate programming, which to be honest, I mostly have like negative feelings about based on LaTeX documentation manuals for LaTeX packages, which have either of you used LaTeX?
I think like the documentation packages for LaTeX and because I've written this literate programming, so which explains how the whole thing works and you're like like there's now a 400 page PDF that explains how this tool works and you're like, I don't care. I just want to know which option do I need to set to do the thing that I want. So I was like, I don't know, I felt like I got burned by literate programming on LaTeX package documentation.
Yeah, I would say like the the extension of or having like the like having reactivity and that the cell order does not matter, like introduces its own set of like new sort of tradeoffs and problems. But but those have been things that we've tried to address more through like having different debugging tools inside of the notebooks. If you're like focusing on a cell, we have this thing like actually inspired from Observable that's like a mini map that allows you to sort of jump around and trace back to like where like the dependencies of that cells are.
From chemistry to visualization research
I'm actually so curious to hear a bit, maybe going back to your, maybe to take a step back and go to your like journey into Marimo. And I don't know if being a Wisconsinite has something to do with it, but I actually like, I was so curious because I noticed you mentioned you're like from Wisconsin. As I understand, you went to Kenyon College in Ohio. And then you jumped to Cambridge in the UK and then back to the other Cambridge in the U.S. to do a PhD at Harvard. I'm so curious to hear a little bit about your like the places you've been and in some of that, because you've been so many different places, starting with Wisconsin.
Yeah. So I mean, I'm yeah, I grew up in the Midwest. I'm from Eau Claire, Wisconsin, a small-ish town. Like it's now I feel like as I'm zooming in on a map or if I'm flying over Wisconsin, you'll like see Eau Claire now. So we're big enough that like that, like it'll be one of the few, the few cities that like pop ups. But it felt very small, like growing up there. And then I went to like Kenyon College was in Ohio. It's like very small liberal arts college where I studied chemistry and I swam competitively. So I was like definitely in like the natural sciences, but did not really work with computers that much. I mean, I use computers. I made websites for fun, but I didn't for my own like research. I was very much in a wet lab doing like organic synthesis.
Yeah. So Kenyon College, it's in, it's about an hour outside of Columbus, Ohio. So there are about 2,000 students. Yeah, being a liberal arts college, like I got a bachelor of arts in chemistry, so I studied like humanities and things outside of like the natural sciences as well. And I think it was just a really nice place to have faculty that were like very hands on with like learning. So like, you know, in contrast to going to some of the places where I ended up for graduate school, like the priority at that university or at Kenyon College was like having faculty that were very like engaged and like trying to get folks into research. So I had, I think it was my academic advisor had a lab and he was like, hey, if you want to like do some research, you should you should try it out. And I didn't, it sounded interesting. So that's sort of where I started my research career, I guess.
Yeah. And then from there you went to Cambridge, is that right?
Yeah, I tell you, it was like a bit of a journey, I guess, like because it was such a small school, I would like apply for various like summer research internships. And I ended up taking like an internship in St. Louis, Missouri at the Danforth Plant Science Center. And I was doing a lot of like wet lab kind of like molecular biology stuff. But they ran this workshop with computer vision. And I was like the first time that I used like I think it was a Python program called like OpenCV. They gave us like a Raspberry Pi. And I remember I was like that summer I was imaging a lot of like Arabidopsis, which is this model organism. And I was measuring all the apertures by hand. And then like we ran this like computer vision program that like I saw like kind of automate a lot of that. And I think it was like kind of this like revelation of like, oh, my gosh, I think I should figure out how to use this stuff.
But also it was like kind of interesting because we like it was a very visual like output, like you could kind of see where it measured things on the actual images and being able to like as a human, like look and observe that you could kind of see where the algorithms would like fail, being able to actually like highlight a region. And so I think that's I mean, in retrospect, it's easy to like construct some narrative of when I started down this path into like visualization. But like that was probably somewhere we're having the somewhat like web background and like also this like science background and seeing like how the ways that, you know, visualizations can sort of surprise you or like confirm like some something that you suspected in the data.
visualizations can sort of surprise you or like confirm like some something that you suspected in the data.
Had you done a lot of computer programming up to that point?
Not really. Yeah, I mostly like web, HTML, CSS, JavaScript kind of stuff. And then after that summer, I think when I went back to doing more research, I got I started like trying to use actually R was the first programming language I like really sat down with to learn for data analysis. Never heard of it. Yeah, yeah, yeah. And and then I think I just got really interested in, yeah, like software generally for like for doing data analysis. And that sort of led to, I guess, not really like a total pivot, but switching into like looking more into doing like computational research rather than like sitting under the hood with a pipette was sort of the direction that things went.
And then from there, you went to Cambridge to do a master's. Were you, yeah, were you thinking of going like the academic route or kind of where were you at that point?
Yeah, I think at that point I was interested in doing a PhD, maybe at some point, but I, I had, but because because of my background, like I really wanted to spend a little bit of time, like doing some software stuff there, like, like data or spending time improving my maths and sort of like becoming more of a data person, maybe. So that that was an interesting, yeah, like avenue, I guess it was sort of like a one year master's. And then after that, I joined into like a bioinformatics computational biology program for the PhD.
Yeah, it was it was really interesting. I mean, like coming from a very small liberal arts college and then also just being like in the UK, like there are a lot of differences. Also like a lot more European in terms of like the the population that was like at the school and also in the program. So that was, I mean, probably the best part of that experience was making a, we had a very small cohort and just making like very close friends that now are sprinkled kind of like throughout Europe. But yeah, I think it was, it was a much different type of institution than like where I went to undergrad.
So I think it also like taught me a lot about, you know, priorities in terms of research or like like academic priorities for faculty at different universities. There's something interesting about, I feel like just all the places kind of you you bounce between. It seems like a big cut of both like American Midwest and like the UK. So it's interesting to hear about. I mean, I had to wear like a gown sometimes to like dinners at Cambridge, and that was very like, I remember not even like thinking I didn't need like a suit or something. And then I was really happy that I brought one because we oddly were wearing it a good amount.
So that actually reminded me, like my dad did his PhD at Cornell and he had like this part-time job, like checking that people were wearing their jackets to dinner. He was like jacket police. Like the jacket bouncer.
Visualization research at Harvard
Yeah, it's called the, we're called the High Dive Group now. When I joined, the lead of the lab is Nils Gallenborg. We rebranded at some point, I think to like, but we're like a visualization group. So we, the way that the, the research that we worked on where we have a lot of close collaborations with like experimental biologists in the department or outside of the department at Harvard. And then we kind of work with them to understand like their data problems and see if like visualization, like yeah, visualizations could help or like really connect folks to their data. I think a nice thing with a field like biology is that a lot of the data are very abstract. So there, and like new types of experiments come out in which you like acquire and like new information. And so it's like kind of this nice, interesting pairing for folks that are really interested in developing new visual representations because you have this really abstract data type that you're trying to, you know, come up with new ways to represent that are effective for helping understand information.
Yeah, I'd say like a pretty like classic example, example would be something like a genome browser. Like that's like a very, like the types of visual representations in a genome browser are like well known, like it's a lot of bars and you know, like dots. But then the type of data behind it and like the idea of like overlaying all these different tracks and it being like zoomable and like panable and like also just laying out the genome like linearly. Like that's like a chosen representation for that information.
Although what I will say is like from the visualization community, like when I joined, I was super excited about, you know, I'd gotten into working on, interested in visualization. Some of my first like open source contributions were in sort of like visualization libraries. And I think I was just motivated to have something that like kind of fit my background, but where I could study visualization and work on it. And it seemed like the focus of that community is really on these like novel representations or because that is where like interesting, I don't know, like that's kind of like where the field was. And so the types of like applications that we, or the types of software that we would work on would resemble like a lot more like monolithic like software applications. Like it's, hey, we built this like viewer for this.
And what I found over time was that like we ran into all these practical barriers where like that might make a really interesting like visualization paper, but like the field is sort of interested in these like new representations, but things that were like are kind of like boring to the field now, like still are not being used by like practitioners that like have their hands on data. And I think that that's where I really started to think like, or I took for granted like the cost of the tools that we use. Like if you are like, I use this tool and I work on my data every day and then I come to you and I say, I made another tool for you. It's like, okay, does that compliment my workflow? Are you asking me to like throw out my previous thing? Like, how do I get my data into that? Like, these are all just questions that you start to have to either answer with software in that application or like workflows to get data in there.
And I think I just got kind of frustrated in this feeling that we, the community that felt like it had solutions. And like, there was like kind of this disconnect back to like the folks that would actually benefit from using them and in understanding their workflows in real ways.
I remember going to a Viz conference a while ago and having this just like really strong feeling, like these people were like, this is like haute couture. Like they're delivering, they're creating these amazing kind of bespoke pieces of art that's tailored to one person. And I was coming in and I'm like, I'm like a buyer at Target. Like, if I can't get this on the back of a million people, like who cares? Like, it's just such an interesting like difference in perspective.
I was coming in and I'm like, I'm like a buyer at Target. Like, if I can't get this on the back of a million people, like who cares?
I think I started to think about it a lot more, too. I think the reason why — it's not like that community set off to build these only specialized applications, but in writing my PhD thesis in the end, I tried to think a lot about why visualization software is kind of different from other software. And I think a part of it is just that it touches so many pieces of the software stack, like it has a backend, and it has the actual UI layer, so you can shell out to different commands from a lot of different applications, but it's really hard to take this visual thing that was built one way and stuff it in another one if they're built with totally different technologies.
So yeah, I think I just started to focus — maybe there are ways that we could improve our software practices in such a way that we could try to standardize a little bit more how we're building these visual representations, such that we don't need to — one, we don't need to build the all-encompassing tool that does everything, but maybe we can layer on just the visualization layer into an environment where someone already knows what they're doing with their data. And that led me to looking at notebooks, because that was something that I used a lot for working with data, and also our closest collaborators use Python notebooks. And then yeah, also there's a way of extending Jupyter notebooks with something called Jupyter Widgets, and that was sort of how I started playing around with this idea of taking web-based visualizations and embedding them inside of the environment where people have their hands on data rather than saying, hey, here's a new application, and sort of giving that feeling of, hey, if there's this library, pip install it and get going, rather than like, oh, go to this website, download the software, run this script to export into a particular file format and then ingest it.
Building AnyWidget
Yeah, I'd say like previously, or the way that I'd read about using something like MCP is like, it's, you know, model context protocol is this way of like taking an application and like exposing it to like your agent. And then yeah, also there's a way of extending Jupyter notebooks with something called Jupyter Widgets.
I mean, the other thing that did change is you could create — now you can create high-performance visualizations in a web browser, and 10 years ago that required C++ code, or 20 years ago. That is another — that's one of the reasons that it's so much easier today to distribute this stuff.
I guess to the notebook point too, yeah, it's interesting because I — Trevor, you mentioned like all of these applications had to worry about how does data come in? And I guess if you look at it from the notebook side, it's like this kind of substrate can handle that across everyone. And it sounds like you're saying if you just focus on the visualization part, the notebook can kind of take care of all these activities that every single bespoke app repeats, essentially?
Yeah. The concrete example I like to give is like, imagine that you have an application that like renders points, like a scatterplot application. And someone's like, it's like, okay, how do I get my data into that? It's like, okay, we take a CSV as input. Okay, what are the columns of that CSV? And then it's like, but what if I had a TSV or what if I had like a parquet file or like, it starts to be that now that you're putting stress on that application to solve that layer of like, how do I get my data into memory such that I can put it into this visual representation versus in an environment like a notebook, you can kind of just say my data visualization takes a data frame as input, and it should have these columns. And then there's other libraries that someone is interested or that works in notebooks knows how to get their data into a data frame. And so you can really focus on like, the data types, like tabular versus maybe like multidimensional arrays, or I feel like are kind of like the core data types in the data science ecosystem.
And in Marimo, just to tie it up with a bow too, in Marimo, this is a widget. Is that right?
Yeah, exactly. So, you know, to complete this, I guess, this thread with like Jupyter widgets, I started working on them because they seem like this useful primitive for, you know, extending notebooks with these interactive elements. And just to be concrete, a widget is like a little bit of front end code. So something that runs in your web browser that can embed directly into the output area inside of a notebook. So if you have a cell that runs some code, now you can have like some user defined front end code that sits there. But the really important bit is that also that code has a way of like communicating back to your actual, where your Python objects live. And so it provides this really rich interface for actually like, I can move a slider and that changes the value that actually lives in Python, not just like in the front end, like that state isn't trapped.
And so, yeah, Jupyter widgets are the mechanism by which Jupyter define like, here's how you sort of coordinate these two worlds. And what ended up happening was at the, you know, when widgets first came out, you had this fragmentation of different front ends for Jupyter where you had like Google Colab or VS Code or JupyterLab or even classic Jupyter. And because the web browser didn't have like a standard module system, they all kind of rolled their own system for like how you load third party JavaScript code. And so what that meant was now every widget author like had to become an expert in like web bundlers and package managers to be able to like create something that could be used universally.
So I started working on a couple of Jupyter widgets. I'd been working on building like, you know, web applications because that's how we built a lot of the visualization tools. It just felt like a lot different type of development. Like it felt like it was like years before what I was familiar with, with kind of with modern web development. So AnyWidget, which is the project that came out of this, was originally just a way for me to be able to iterate very quickly on like building like more of these and make it a little bit more fun where I didn't have to like restart my browser and like refresh everything to like see the updates. It was trying to come up with unifying like the way that you wrote that front end code more around how modern browsers talk today and then like ingesting that in a bunch of different environments. So I could sort of solve the platform compatibility problem once and then like just author this thing that's a little bit more portable.
Just for some context too, I think people, like a lot of people think of AnyWidget today, but yeah, having lived through the shift to AnyWidget, yeah, I can't overstate how convenient it is to have something run across all these different platforms that I do feel like that juggling was real of figuring out like I can run in the Jupyter notebook. Now I've done the work to figure it out for Google Colab, but like to have it in one place and not even have to worry about that is so convenient, seems like.
Yeah. And I think also like, you know, when I've used notebooks for prototyping like Python code, like maybe to speak more concretely, like I use notebooks to prototype code because it's a way that I can run little snippets of code and try them out. But in order previously to like even just experiment with the idea of like a visual output, like a widget, like you'd have to like create a new Python project and like kind of like link and like start this like build process in Bundler. And so like you couldn't just like play around with, hey, maybe it'd be fun to like draw these things in the web and put them there. And that was probably the most like exciting thing to me originally with AnyWidget was just the fact that inside of a notebook, I could like start playing with both front end and my Python code together and then kind of get these proof of concepts very quickly without having to like start a whole new project.
Yeah, that's a great question. So part of this is that AnyWidget itself defines a specification, not only for like the end the people that author widgets, but also like what a host platform has to implement in terms of APIs to support widgets. So if you're using AnyWidget in a traditional Jupyter notebook, that protocol layer is like the host platform, AnyWidget, the library, like plugs that in so that you can just load them and like run in those environments. But inside of Marimo, we actually like natively implement an AnyWidget API to like support the AnyWidget protocol. So there is no like Jupyter machinery inside of Marimo, but they do support AnyWidget. So this thing that sort of started originally as a way for me to be able to author like traditional Jupyter widgets, and now is like if you author them there, there are even more platforms beyond Jupyter because if they can implement sort of the host side of that front end code, now you have this contract where basically something's going to load this and render these arguments.
And then you can expect these APIs to exist is basically the runtime contract. It's been interesting to see too that a lot of tools like used to have these like catches, like if I'm in Jupyter, do this. If I'm in Colab, do this. It's been interesting to see like a lot of big tools switch to just using AnyWidget like that from that kind of like if else block of like what environment am I in? But from what you're saying, Marimo also uses AnyWidget as kind of its native like widget implementation, is that?
Yeah, if you want to make custom interactive elements, any Marimo has special machinery to host AnyWidgets. And then like the widgets themselves sort of have this internal state on the Python side that is currently like based off of this library called traitlets. That's just the Python side is like the AnyWidget or the machinery. And what you can do is like pluck off individual pieces of state from those widgets to observe as reactive values inside of your notebook. So I'll declare my widget, I can use a decorator that's called like, it's like, or a function is called mo.ui.AnyWidget and wrap my widget in that. And that will just listen to every single value on that widget. Or you can take individual pieces of state and lift them off of the widget as well. And now you basically are like taking that internal reactivity and pulling those into Marimo's reactive graph. So now I can like reference a value in a different cell, and it plays nicely by Marimo's rules.
Marimo pair: agents and notebooks
I'm curious too, I know you mentioned you've been working more lately on Marimo pair. Maybe tell us a little bit about Marimo pair and the problem that solves.
Yeah, I'd say, you know, increasingly, we've started, I started using coding agents at work for doing like traditional software development. And we had like some initial sort of integrations with like coding agents for Marimo, but both from our users and like my own personal experience, like there felt just kind of something off when you tried to like do something in notebooks. Like it was like, and I couldn't really put my thumb on it at the time, but it was just like, okay, I have this data idea. And, you know, with my coding agent, I can just let it loose in like a folder of a bunch of script or on my file system and just like let it rip on a problem. And I'll come back and I'll have a bunch of scripts and some outputs.
But like if I tried to use like integrations I'd tried with other types of notebooks, or even with Marimo at that time, you'd like, the way that our integration worked then was sort of, we'd edit the like the Python file on disk and try to like play the different changes back inside the UI. And what would happen is the, you know, you'd let the agent go and it would just kind of say like, okay, good, I'm done. Like it works. And like, it like wasn't working. There would be some like issue in the UI. And then you really felt like you're kind of like babysitting this agent, which is like a much different user experience to like working on traditional software where it's like, hey, I have these tests, go run them until you like fix the problem. Or like it's going to encounter syntax errors and like keep going until it like finishes up.
So yeah, Marimo pair kind of spawned out of like taking a step back at how we were approaching, like trying to drive, like use notebooks with agents. And I really think our new framing of it is thinking about like notebooks, you know, notebooks and stateful environments for coding have been useful for humans to understand data. Is there a way that we can like actually make notebooks useful for agents as well for exploring data problems? And so it was not so much as like, can we use like a notebook as nice context for the model, but like, can the model actually just use a notebook to like reason and like work on a problem for a while? And that, that changed, that led to us like really changing how we went about like exposing our notebooks as a tool for agents.
Yeah. I demoed Marimo pair for the company at like a work week because the fact that in Marimo, it can just, the agent seems to be able to do anything, kind of like anything I could imagine clicking. It seems like the agent can do, I don't, there may be some limitations, but it's, it's been so interesting to interact with.
Yeah, I'd say like previously, or the way that I'd read about using something like MCP is like, it's, you know, model context protocol is this way of like taking an application and like exposing it to like your agent. And the way that we were using MCP before was kind of like allowing the agent to query the state of the notebook. So it'd be things like, you know, what are the cells? What are the contents of the cells? Edit, you know, what are my variables that are inside of the notebook? But then what you, what we realized was like, you know, when the agent would take an action like writing or mute, like doing something in the notebook, it was using its existing tools to edit a file on disk, but it never got feedback for like, Hey, did something go wrong? Like in that same execution channel. So it was kind of, so what the agent would have to get good at is like making a change and then querying the notebook of like, and making a bunch of serial queries to be like, did something go wrong?
And what we changed inside of Marimo pair is we basically got rid of like all of our tools. And we just have one tool that is like run Python code. And that just runs Python code in the Marimo kernel. And there's a hidden like Python API in there where the agent is able to control the Marimo notebook and ask about the Marimo notebook itself. And so when it does something like hallucinate a value or, you know, redefine a variable or something that doesn't play nicely by Marimo's rules or Python's rules, it just gets feedback in that same like turn that says, Hey, you need to change that. Like that's not working. There's an error. And then it can keep going.
And that ability to, I think, have both that read only access and like see the effects of the mutations, like a very similar way to like when you edit files on disk and then run the scripts to see if they, you know, exit code one or zero. Now the agent can do that at the cell level. And that, I think, allows for this ability for, you know, the usefulness of notebooks is that you can iterate very quickly. And then when you try to like give agents to them previously, it started to feel like, OK, but the agent isn't being very productive in this environment. And I think that that was sort of the missing piece was like, how do you give them feedback in that same execution channel to seeing, you know, what you're seeing so that you don't have to sit there and tell it that something went wrong.
It is interesting that like data exploration is like so stateful. When you think about like traditional software engineering, like you run your unit tests and those are all designed in such a way that you start from a fresh session, they run and you get the results back. But like that just doesn't work for data analysis because it's so iterative. And like, yeah, you want to be able to look at the state of the thing you just did and you don't have to rerun the whole thing from scratch. Every time we'll figure out some complicated like caching mechanism. You just you want to have this like long running persistent session that records like what have you done to the data?
So it sort of adds this extra layer of guardrails beyond just a normal Python REPL, or in order for the agent to like continue on a problem, it has to check cells into the notebook and define variables. And then each of those cells has to play by marimo's rules. So it's almost like, you know, yeah, traditional software development is very stateless, but your state is kind of like your file system, right? So you like version that over time, and then it's running these like scripts on top of this state that you version over time.
And I think very similarly, we want to provide an experience like that in a stateful environment with these guardrails around like, how you have to check those cells in. So, you know, in marimo pair, you can't check in a cell that redefines a variable or something. So it has to play nicely by marimo's rules. And so sort of as a side effect of the agent exploring a problem using marimo, you get like a reproducible kind of like execution flow.
There's a downside to that though, too, though, right? Like, like what happens if like the thing at the end of the very end of the reactive graph takes like a minute to run? Like, how do you, like, I assume marimo must have like tools to like, when you don't want things to run all the time to like opt out of opt out of that.
Yeah. So the contract in marimo is that what you see on the screen is a representation of the state in the kernel. And so you can turn off reactivity or disable individual cells, and they will just show as being stale. And then you have to like manually execute them. So if I turned on like lazy mode, then the agent would get feedback saying you need to rerun these cells, but then it would, it could go back and run those cells.
It's interesting. And what you're describing, I think for the agent, as I understand is marimo code mode. Is that right? The sort of secret backdoor into marimo that an agent can use to basically like create cells or edit them.
And I think another thing you mentioned is like the connection to marimo that like, is it right? If, if the agent like deletes a cell that defined a variable, it's like the variable itself is removed. Is that, that's kind of part of the maintaining like a fresh state. So it's like, in a way it kind of like locks in the agent into having like an up-to-date environment. And I don't know, I thought that was really intuitive that like, if I, as a human also deleted a cell that defined the variable, I would also kind of want it to go away. Like that's a really classic human blunder to like delete the ref, delete the code that made it, but still have it like floating around in the environment.
Yeah. Like it's, it's, it's a hidden Python API that the agent, like if you read the docs inside of marimo, it's like, do not use this like yourself. If you're a human being. If you're a human being. And part of that is that we, you know, we don't introduce in comparison to other APIs that we have inside of marimo that we want people to like depend on like code mode was something that we've like refined over time and we're still like changing. And that's part of that is because like, because it's used within these sessions, the agent learns about like what APIs it has available and how to call those in that session versus, and that's a much different contract between like a model and like software versus like two pieces of software where you have like versioning problems. And like, you really have to make sure that they yeah. Like if someone was building around code mode that was outside of using agents, like they would get very annoyed with us when we made breaking changes versus the model basically gets a new version of marimo and it has new capabilities or maybe we fix bugs and it just calls those APIs and can sort of chug along inside the notebook.
Presumably code mode is what the marimo UI uses behind the scenes.
So most of marimo's UI is driven through like the front end, which is like JavaScript code. So we've had a lot of like, you know, it's this message, I mean, to get technical, it's like this message patching system of like edits, going back to the kernel, you have some piece of state that you're updating. But really we had to introduce like a Python API for driving marimo notebooks because, which we didn't really have before. And that made us changing a little bit about how our document model works. So now you can like drive it from Python or you can drive it from the front end and they can all like sort of share that state.
But you know, really the reason we were able to do that originally was just because our kernel is in Python. So when I first worked on marimo pair, the first thing introduced was just the ability for the model to like run Python code in the kernel. And that let it do basically everything that was like read-only inside of marimo. So it's like, you know, we have this big set of MCP tools that were like list my data frame, list my variables. And now it's like, okay, get rid of all those. You just write the Python code you want to like cat out a table, like print the columns, like you can discover APIs, like just call Python the same way. If you're running an agent and you see it using Python from the command line, it's like running scripts like very quickly.
But then, and then I think the first thing I did was like, okay, show me a notification in the UI. And I saw it like, like try a bunch of different things. And then finally it found like one API where it like toasted me a notification. I was like, whoa, okay. Like, how'd you do that? And then like, we kind of like look back through the code. And so we started to then think about like what types of things would you want to drive from the kernel? And then we harden those behind like a Python API that we could test and such that, you know, you can have high confidence when the agent doesn't have to like get the right sequencing of notifications. It just calls an API. And then we replay those notifications to the front end and also make sure the kernel stays up to date.
Yeah, it's such an interesting approach. And also I do think it's interesting to hear it in contrast to like the MCP or tools. There's like a lot of ways to kind of carve this up. But I think we've like broad, like in the early phases of like MCP being a thing, everyone was like, let's make a bajillion MCP tools and then kind of discovered like, oh, actually LLMs are like way better at writing code in an existing language than trying to figure out some like bizarre novel set of tools that you have created. And now like, I think, I mean, MCP still has a, as it plays a role, but it's so much more powerful just to give it a few like tools that let it write code and do stuff like that's really, really powerful.
And I think the other thing that was non-trivial to me that when we changed that way of having many tools to like one very sharp tool is that programming languages have this, you know, ability to like compose different primitives as well. So like you don't have a bunch of different tool calls, you could actually in one execute code. It could like look at the notebook state. It could like create multiple cells, delete cells. Like if you don't have, those don't have to be individual transactions that you replay. And instead it's like, you can leave it up to the creativity of the model to figure out how to string these things together to accomplish the task that the user wants to do inside the notebook.
How Marimo pair works in practice
And I feel like just to set the stage too, for what, like when I use Marimo pair that there's a skill, like essentially it was so interesting to be like, okay, just use this skill. I was using Claude code from the command line, just like plug it in and that Claude code actually could, I mean, and this isn't a surprise, install Marimo, you know, essentially fire it up on my computer, tab opens in my browser, and then just starts driving Marimo. I feel like that was such a kind of eyeopening experience to realize like there's another world where Marimo said like, Hey, set us up, you know, run the agent from inside Marimo and it could do some stuff. But to watch it actually like run, realize that we needed a blob on the outside, like even running Marimo, opening it and orchestrating it, uh, was a really interesting experience to, to see.
And the other thing I'll flag in, in correct me if I'm wrong, is that Claude code is essentially using a CLI, like a bash tool. Is that right? To basically like ship this Python code in, this Marimo code mode in, to run in the kernel. Is that, is that right?
Yeah. So if you install the skill from like GitHub, which is the way that we recommend doing it, you'll get a bash script that the agent can call that's called execute code. And behind the scenes, what that does is it pings an API endpoint on Marimo that we have set up that then replays or kind of maps the, that HTTP response back to your standard end, standard out, um, standard error, and then exit with an exit code of zero and one. Um, and what's cool about that is if, you know, so we have a cloud hosted version of Marimo notebooks called Molab that runs like, um, where you have a kernel that's running in the cloud. And because it's all done over HTTP, you could actually point Marimo pair to like talk to a remote version of, um, of your Marimo that's running. And then you don't even need to have Marimo installed locally. You just need curl and bash scripts. So then you can be using cloud and like you're driving a remote notebook as well.
It's interesting. Cause we, I think we, in some, yeah, cause you can also do the opposite, right? You can provide a CLI, like you could have a Marimo CLI. And if you call Marimo agent, then that produces the skill file that then that tells the agent how to use it. But you like you distribute the skill and then it gets that kind of CLI from that as opposed to distributing the CLI and that gives the agent the skill.
Yeah, we've definitely found, I mean, part of it inside the Marimo UI, if you're on any Marimo page, you can click in the top corner and then it says pair with an agent. You can click in the top corner and then it says pair with an agent and we give you this little snippet that you can run in your terminal that like sets it says use the Marimo pair skill and like, you know Toast a notification and like here's where the notebook is running kind of thing. And that that is kind of the nicest experience I found to being able to launch it, is just like, you spin up Marimo, then you click that, and you can sort of be confident you're connected and keep going.
It's a subcommand in the Marimo CLI that, like, basically is like a prompt that says, like, check to see if the — like, we check to see if you have the skill installed. If you don't, then we'll, like, prompt Claude to, like, go get the skill and, like, have you install it.
But if you ever — if you look at the Marimo pair skill, it also is, like, very uncoupled to, like, the code mode internals. Like, it basically tries to orient the agent around, like, the things that Michael was talking about. Like, you know, if a user's trying to do something with Marimo, like, what's a nice path forward and, like, trying to help them, like, spin up Marimo. And then the second piece is basically telling it, you have one tool, and then there's this thing that's called code mode. And, like, when you start a session, go look and see what code mode is in that instance of Marimo that you have. And so because we can ship changes to code mode, baking in, like, those API invocations in the skill itself would kind of lead to this versioning problem. And instead, what we've been able to do is, like, version the two independently, where we just say, here's the skill, and then we can ship upgrades to Marimo, and we don't have to worry about the two sort of, like, drifting from one another.
It is fascinating to me how agents can change the API ergonomics. You know, like, one thing I think that's kind of interesting and surprising is, like, I would say on average, like, agents are way better at reading error messages and actioning them than humans. So like, good error messages have always been important, but they're almost more important now because you can assume that an agent will actually read it comprehensively, and if you have a good suggestion, it will follow it. And then that can make, like, APIs, like,