Notes from the 2020 Spark+AI Summit

Thread: Celebrating the tenth year of Spark, and the release of Spark 3.0 while listening to @matei_zaharia's #SparkAISummit keynote ✍🏼 I found the first post I wrote about Spark, roughly eight years ago (dated 2012-08-12) 👉🏽 https://t.co/bPTQJfMSNS pic.twitter.com/1KX7cWluZs

— Ben Lorica 罗瑞卡 (@bigdata) June 24, 2020

Thread: @alighods #SparkAISummit keynote on Lakehouse, a data management paradigm we wrote about early this year https://t.co/CMExlP3wi3

— Ben Lorica 罗瑞卡 (@bigdata) June 24, 2020

Thread: @rxin announces Delta Engine at the #SparkAISummit 🙌 Delta Engine is designed to accelerate SQL and dataframe workloads with two components: a query optimizer & a vectorized execution engine. pic.twitter.com/E5hOPmxaFH

— Ben Lorica 罗瑞卡 (@bigdata) June 24, 2020

1/ Day Two of Live tweeting at the #SparkAISummit 💻📱 @ClemensMewald keynote ↦ Tooling for #DataTeams is still a challenge for most companies ↓ pic.twitter.com/rc9j7n0p2K

— Ben Lorica 罗瑞卡 (@bigdata) June 25, 2020

Thread: @matei_zaharia is giving updates on MLflow at #SparkAISummit. He describes MLflow as an open source *Machine Learning Platform* that now has four main components: experiment management, reproducible runs, packaging & deployment, and a model registry pic.twitter.com/xQaXEiM45F

— Ben Lorica 罗瑞卡 (@bigdata) June 25, 2020

Thread: live tweeting @DrKimHazelwood @Facebook keynote at #SparkAISummit: the share of Facebook’s structured data being used machine learning models grew from 30% last year to 50% this year. The number of engineers who train ML models has doubled ↑ # of models has quintupled pic.twitter.com/2R7tadBtW0

— Ben Lorica 罗瑞卡 (@bigdata) June 25, 2020

Thread: Next up at #SparkAISummit is Hany Farid of @Berkeley_EECS, widely considered to be the father of digital forensics, a field that finds itself in the middle of the battle against deepfakes, the use of models to synthesize realistic images & video https://t.co/XDoCpgYKdy pic.twitter.com/DeITmb73gC

— Ben Lorica 罗瑞卡 (@bigdata) June 25, 2020

1/ Opening this morning’s #SparkAISummit ↦ @apaszke of @Google author of #PyTorch. As Adam notes, PyTorch continues to be the favorite library used by ML researchers: over half of ML papers were implemented with PyTorch. This is trend I observed last year https://t.co/V0BAVE7tTN pic.twitter.com/YZjwYlZcjU

— Ben Lorica 罗瑞卡 (@bigdata) June 26, 2020

Thread: Closing #SparkAISummit keynote is @aheineike @primer_ai, she is describing one of the best sites on #COVID19 ↦ COVID19Primer uses advanced #NLproc to summarize, surface trends in research papers, media articles, & social networks. pic.twitter.com/yJCIsFEmsr

— Ben Lorica 罗瑞卡 (@bigdata) June 26, 2020