Hot technology job market is losing its sizzle
If you have the right skills you’re probably still getting inundated with messages from recruiters. But beware that the overall job market is softening. We’re seeing large percentage declines (year-over-year) across a variety of data and AI keyword searches. Exceptions include “data governance” (essentially flat), DataOps, and MLflow. The number of MLflow job postings have grown from about one-eleventh of TensorFlow to about one-sixth. The chart is based on unique job postings for the same periods (Aug/Sep/Oct) in 2021 and 2022, and is limited to select technology hubs in the U.S.
Data Exchange podcast
- The Unreasonable Effectiveness of Speech Data: Piotr Żelasko is Head of Research at Meaning, a startup building an AI platform based on speech technologies. We discuss Whisper, a deep learning model (from OpenAI) that approaches human level robustness and accuracy on English speech recognition.
- Project Lightspeed – Next-generation Spark Streaming: Karthik Ramasamy is the Head of Streaming at Databricks. He has extensive experience in streaming, having led teams at Twitter (Apache Heron), Splunk, and Streamlio (Apache Pulsar). We do a deep dive into some of the key areas addressed by Lightspeed (next-gen Spark Streaming).
- A new storage engine for vectors: Ram Sriharsha is VP of Engineering and R&D at Pinecone, a startup that offers a fully managed vector database (not just an index). We discuss Pinecone’s migration from RocksDB to a new proprietary storage engine written in Rust.
K1st World
I’m co-chair of K1st World, a fantastic symposium and networking event slated for November 16th. Leading researchers and companies will examine key lessons in combining data, machine learning, & knowledge for life-critical applications. To be more specific, the symposium explores how we combine human experience alongside data and promote powerful, yet safe, reliable and trustworthy ML/AI systems. Use the discount code GRADIENTFLOW60 to attend in person or online.
Data & Machine Learning Tools and Infrastructure
Is the data mesh right for you? Ahem. No. On the other hand the lakehouse addresses one of the fundamental concerns with data lakes that led to the data mesh – that monolithic lakes can become unmanageable.
Four Reasons Why Leading Companies Are Betting On Ray. In a new post with Ion Stoica and Zhe Zhang of Anyscale, we list the top technical reasons why engineering leaders and alpha geeks are putting their faith in Ray.
Reimagining Data Discovery with a Modern Metadata Platform:
Here’s what we need to do to fix AutoML
With AutoML, domain experts can create machine learning applications without much statistical or machine learning knowledge, thereby accelerating product development while simultaneously reducing the need for data scientists and machine learning engineers. While AutoML remains a relatively new and very small market, it has enormous potential. From $270 million in 2019, it is estimated that the AutoML market will generate $14.5 billion in revenue by 2030.
Existing AutoML tools have primarily focused on model generation and model building. In a new post with Assaf Araki of Intel Capital, we offer seven suggestions to make AutoML more effective.
If you enjoyed this newsletter please support our work by encouraging your friends and colleagues to subscribe: