Big Data

sparklyr 1.3: Higher-order Functions, Avro and Custom Serializers

Sparklyr 1.3 is now available, featuring integration of Spark higher-order functions, and data import/export in Avro and in user-defined serialization formats

sparklyr 1.2: Foreach, Spark 3.0 and Databricks Connect

sparklyr 1.1: Foundations, Books, Lakes and Barriers

sparklyr 1.0: Apache Arrow, XGBoost, Broom and TFRecords

sparklyr 0.9: Streams and Kubernetes

SparkR preview by Vincent Warmerdam

How Open Source, Python and AI Are Shaping the Data Future with Wes McKinney

The future of analytics isn’t just about bigger models — it’s about building smarter, more interoperable data systems. Wes McKinney, Principal Architect of Posit PBC, Chief Scientist of Voltron Data and a General Partner at Composed Ventures, joins us to explore how the modern data stack is evolving and what it means for the future of analytics. Wes reflects on his journey building pandas and Apache Arrow, sharing how open-source ecosystems grow, transform and shape the way organizations work with data today. Wes also highlights the rising importance of semantic layers, agentic workflows and defensive coding practices as teams embrace AI-driven development. Key Takeaways: 00:00 Introduction. 02:32 Wes didn’t expect pandas to drive AI but he recognized Python’s unrealized potential. 05:09 A lucky convergence helped Python’s tools snowball into the AI standard. 10:40 Early big data focused on essentials, not the interoperable stacks we rely on today. 15:44 The composable data stack grew through bottom-up, grassroots open-source momentum. 21:56 Many “data science” roles ultimately became business intelligence and dashboard work. 25:24 Complex statistical work still depends on human judgment, not fully autonomous agents. 30:27 Frontier models retrieve table data reliably, while smaller models fail dramatically. 35:16 Better models and coding agents shifted Wes from an AI skeptic to an adopter. 40:07 AI-driven code demands stronger testing and review to avoid costly failures. 45:14 An AI-built finance project ballooned, revealing how agents inflate codebases. Resources Mentioned: Wes McKinney https://www.linkedin.com/in/wesmckinn/ Posit PBC | LinkedIn https://www.linkedin.com/company/posit-software/ Posit PBC | Website https://posit.co/ Voltron Data | LinkedIn https://www.linkedin.com/company/voltrondata/ Voltron Data | Website https://voltrondata.com/ Composed Ventures | LinkedIn https://www.linkedin.com/company/composedvc/ Composed Ventures | Website https://composed.vc/ pandas https://pandas.pydata.org/ Apache Arrow https://arrow.apache.org/ DuckDB https://duckdb.org/ DataFusion https://datafusion.apache.org/ Jupyter Notebook https://jupyter.org/ Parquet https://parquet.apache.org/ Iceberg https://iceberg.apache.org/ Delta Lake https://delta.io/ Thanks for listening to the “Data Masters Podcast.” If you enjoyed this episode, be sure to subscribe so you never miss our latest discussions and insights into the ever-changing world of data. #DataStrategy #DataManagement #DataMastersPodcast

Neal Richardson | Bigger Data With Ease Using Apache Arrow | RStudio

The Apache Arrow project enables data scientists using R, Python, and other languages to work with large datasets efficiently and with interactive speed. Arrow is so fast at some workflows that it seems to defy reality--or at least the limits of R's capabilities. This talk examines the unique characteristics of the Arrow project that enable it to redefine what is possible in R. The talk also highlights some of the latest developments in the arrow R package, including how you can query and manipulate multi-file datasets, and it presents strategies for speeding up workflows by up to 100x. About Neal: Currently Director of Engineering at Ursa Labs / RStudio. Previously led product and engineering at Crunch.io. Ph.D. in Political Science from the University of California, Berkeley

Neal Richardson