The evolution of data science, data engineering, and AI

[A version of this post appears on the O’Reilly Radar.]

The O’Reilly Data Show Podcast: A special episode to mark the 100th episode.

This episode of the Data Showmarks our 100th episode. This podcast stemmed out of video interviews conducted at O’Reilly’s 2014 Foo Camp. We had a collection of friends who were key members of the data science and big data communities on hand and we decided to record short conversations with them. We originally conceived of using those initial conversations to be the basis of a regular series of video interviews. The logistics of studio interviews proved too complicated, but those Foo Camp conversations got us thinking about starting a podcast, and the Data Show was born.

To mark this milestone, my colleague Paco Nathan, co-chair of Jupytercon, turned the tables on me and asked me questions about previous Data Show episodes. In particular, we examined the evolution of key topics covered in this podcast: data science and machine learning, data engineering and architecture, AI, and the impact of each of these areas on businesses and companies. I’m proud of how this show has reached so many people across the world, and I’m looking forward to sharing more conversations in the future.

Here are some highlights from our conversation:

AI is more than machine learning

I think for many people machine learning is AI. I’m trying to, in the AI Conference series, convince people that a true AI system will involve many components, machine learning being one. Many of the guests I have seem to agree with that.

Evolving infrastructure for big data

In the early days of the podcast, many of the people I interacted with had Hadoop as one of the essential things in their infrastructure. I think while that might still be the case, there are more alternatives these days. I think a lot of people are going to object stores in the cloud. Another examples is that before, people maintained specialized systems. There’s still that, but people are trying to see if they can combine some of these systems, or come up with systems that can do more than one workload. For example, this whole notion in Spark of having a unified system that is able to do batch in streaming caught on during the span of this podcast.

Related resources:

Hong Kong 2016

We just spent the past week in Hong Kong and below are some tweets from our trip. The weather was perfect – the temperature was just right. Their metro/train system is world class and easy to use and navigate. I’m looking to returning soon!

Wan Chai

We stayed in the vibrant Wan Chai district which among other things, has a very active market scene. This is a typical evening just off of the Wan Chai market, and right on Wan Chai road itself:

Strata NYC 2016 is next week

Complete schedule is HERE.

Beijing Restaurants: Strata 2016

Here’s a partial list of the many memorable restaurants we visited in Beijing during the week of Strata 2016:

Tricycle Diaries

On a recent visit to the Philippines, I found myself gawking at two iconic modes of (public) transportation, the tricycle and the jeepney. They remain major sources of gridlock, chaos, and pollution, and many local residents would love to see them banned from the streets of Metro Manila. I doubt that will happen anytime soon, as they remain a cheap mode of public transportation, for a city lacking alternatives – save for a train system that doesn’t really cover large swaths of the metropolis. (For a good overview of traffic in Metro Manila, see this recent Economist article.)

Tricycles

Hanging on to Jeepneys in the manner above, is technically illegal (citation + fine) but it doesn’t seem to deter passengers much. Apparently people indulge in an even riskier practice with tricycles (also illegal). The photos below were taken while I was an Uber passenger in cars trailing these tricycles:

tricycle-back

While most tricycles are motorized, many in the Intramuros and “Old Manila” area were pedal powered:

tricycle-bike

Jeepney (“the king of the road”)

The Jeepney as a “school bus”:
jeepney-school-bus.jpg
 

Packed Jeepney, and it’s not even rush hour:
jeepney-packed2

Kuala Lumpur and Malacca City

The week before Strata+Hadoop World in Singapore, we snuck up the Malaysia for a quick vacation. The food in Malaysia – particularly at hawker (food) courts – was outstanding. We stayed in Petaling Jaya (“PJ”) an area that young professionals and families seem to favor. Below are some highlights:

Selena Malam SS2 Food Court
kl1

Wong Ah Wah (or W.A.W.) on Jalan Alor street – the chicken wings are simply amazing:
kl-waw

Other things worth noting: We enjoyed the bird park and the Islamic Arts Museum of Malaysia.


Malacca City

Restoran Asam Pedas Selera Kampung
malacca-lunch

Restoran Ole Sayang Sdn. Bhd.
malacca-dinner

Kopi Luwak (we found a distributor who carried the wild/”free range” variety)
malacca-coffee1