Over the course of a week, I’ll be hosting two good webcasts featuring Spark release manager Patrick Wendell and Spark committer Kay Ousterhout. Register now! Patrick Wendell: Spark 1.3 and Spark’s New Dataframe API (March 25th at 9 a.m. California time) In this webcast, Patrick Wendell from Databricks will be speaking about Spark’s new 1.3Continue reading “Apache Spark 1.3, the new Dataframe API, and Spark performance”
Category Archives: Data Engineer
Turning Ph.D.s into industrial data scientists and data engineers
[A version of this post appears on the O’Reilly Radar blog.] Editor’s note: The ASI will offer a two-day intensive course, Practical Machine Learning, at Strata + Hadoop World in London in May. Back when I was considering leaving academia, the popular exit route was financial engineering. Many science and engineering Ph.D.s ended up inContinue reading “Turning Ph.D.s into industrial data scientists and data engineers”
Time-turner: Strata San Jose 2015, day 1
[Our friends at Dato created an interesting content-based, Strata session recommender. Check it out here.] There are so many good talks happening at the same time that it’s impossible to not miss out on good sessions. But imagine I had a time-turner necklace and could actually “attend” 2 (maybe 3) sessions happening at the sameContinue reading “Time-turner: Strata San Jose 2015, day 1”
Hardcore Data Science: 2015 California
Ben Recht and I hosted another great edition of Hardcore Data Science yesterday. From the very first talk, the room was full, the audience was attentive, and the energy in the room was high. It remained that way throughout the day. This time around, I spent more time documenting the day on Twitter – enjoy!Continue reading “Hardcore Data Science: 2015 California”
Spark 1.2 and Beyond
Next week I’ll be hosting a webcast with Spark’s release manager – and Databricks co-founder – Patrick Wendell. (Full disclosure: I’m an advisor to Databricks.) In this webcast, Patrick Wendell from Databricks will be speaking about Spark’s new 1.2 release. Spark 1.2 brings performance and usability improvements in Spark’s core engine, a major new APIContinue reading “Spark 1.2 and Beyond”
Apache Spark’s journey from academia to industry
[A version of this post appears on the O’Reilly Radar blog.] Three projects from UC Berkeley’s AMPLab have been keenly adopted by industry: Apache Mesos, Apache Spark, and Tachyon. As an early user, it’s been fun to watch Spark go from an academic lab to the most active open source project in big data. InContinue reading “Apache Spark’s journey from academia to industry”
Building Apache Kafka from scratch
[A version of this post originally appeared on the O’Reilly Radar blog.] In this episode of the O’Reilly Data Show Podcast, Jay Kreps talks about data integration, event data, and the Internet of Things. At the heart of big data platforms are robust data flows that connect diverse data sources. Over the past few years,Continue reading “Building Apache Kafka from scratch”
Spark + Cassandra: Technical Integration Details
I’ll be hosting a Nov 12th webcast on two of the most popular components in the big data ecosystem: Apache Spark and Apache Cassandra. As highlighted in a recent Databricks blog post, recent improvements to Spark’s shuffle have led to significant speedups (Spark is faster than Hadoop MapReduce, even on disk). While Spark has longContinue reading “Spark + Cassandra: Technical Integration Details”
Anomaly Detection with ElasticSearch
One of the technologies that I’m hearing more about is ElasticSearch. In particular the combination of ElasticSearch, Logstash, and Kibana (the ELK stack) has proven to be a popular platform for real-time analytics on both structured and unstructured data. I’ll be hosting a webcast on October 30th on the ELK stack featuring Mark Harwood, softwareContinue reading “Anomaly Detection with ElasticSearch”
Time-turner: Strata NYC 2014, day 2
There are so many good talks happening at the same time that it’s impossible to not miss out on good sessions. But imagine I had a time-turner necklace and could actually “attend” 2 (maybe 3) sessions happening at the same time. Taking into account my current personal interests and tastes, here’s how my day wouldContinue reading “Time-turner: Strata NYC 2014, day 2”