Introducing RLlib: A composable and scalable reinforcement learning library

[A version of this post appears on the O’Reilly Radar.]

RISE Lab’s Ray platform adds libraries for reinforcement learning and hyperparameter tuning.

In a previous post, I outlined emerging applications of reinforcement learning (RL) in industry. I began by listing a few challenges facing anyone wanting to apply RL, including the need for large amounts of data, and the difficulty of reproducing research results and deriving the error estimates needed for mission-critical applications. Nevertheless, the success of RL in certain domains has been the subject of much media coverage. This has sparked interest, and companies are beginning to explore some of the use cases and applications I described in my earlier post. Many tasks and professions, including software development, are poised to incorporate some forms of AI-powered automation. In this post, I’ll describe how RISE Lab’s Ray platform continues to mature and evolve just as companies are examining use cases for RL.

Assuming one has identified suitable use cases, how does one get started with RL? Most companies that are thinking of using RL for pilot projects will want to take advantage of existing libraries.

RL training nests many types of computation. Image courtesy of Richard Liaw and Eric Liang, used with permission.

There are several open source projects that one can use to get started. From a technical perspective, there are a few things to keep in mind when considering a library for RL:
Continue reading “Introducing RLlib: A composable and scalable reinforcement learning library”

How machine learning can be used to write more secure computer programs

[A version of this post appears on the O’Reilly Radar.]

The O’Reilly Data Show Podcast: Fabian Yamaguchi on the potential of using large-scale analytics on graph representations of code.

In this episode of the Data Show, I spoke with Fabian Yamaguchi, chief scientist at ShiftLeft. His 2015 Ph.D. dissertation sketched out how the combination of static analysis, graph mining, and machine learning, can be used to develop tools to augment security analysts. In a recent post, I argued for machine learning tools to augment teams responsible for deploying and managing models in production (machine learning engineers). These are part of a general trend of using machine learning to develop and manage the software systems of tomorrow. Yamaguchi’s work is step one in this direction: using machine learning to reduce the number of security vulnerabilities in complex software products.

Here are some highlights from our conversation:
Continue reading “How machine learning can be used to write more secure computer programs”

Responsible deployment of machine learning

[A version of this post appears on the O’Reilly Radar.]

We need to build machine learning tools to augment our machine learning engineers.

In this post, I share slides and notes from a talk I gave in December 2017 at the Strata Data Conference in Singapore offering suggestions to companies that are actively deploying products infused with machine learning capabilities. Over the past few years, the data community has focused on infrastructure and platforms for data collection, including robust pipelines and highly scalable storage systems for analytics. According to a recent LinkedIn report, the top two emerging jobs are “machine learning engineer” and “data scientist.” Companies are starting to staff to put their data infrastructures to work, and machine learning is going become more prevalent in the years to come.

As more companies start using machine learning in products, tools, and business processes, let’s take a quick tour of model building, model deployment, and model management. It turns out that once a model is built, deploying and managing it in production requires engineering skills. So much so that earlier this year, we noted that companies have created a new job role—machine learning (or deep learning) engineer—for people tasked with productionizing machine learning models.

Modern machine learning libraries and tools like notebooks have made model building simpler. New data scientists need to make sure they understand the business problem and optimize their models for it. In a diverse region like Southeast Asia, models need to be localized, as conditions and contexts differ across countries in the ASEAN.
Continue reading “Responsible deployment of machine learning”

What lies ahead for data in 2018

[A version of this post appears on the O’Reilly Radar.]

How new developments in algorithms, machine learning, analytics, infrastructure, data ethics, and culture will shape data in 2018.

1. New tools will make graphs and time series easier, leading to new use cases

Graphs and time series have been a crucial part of the explosion in big data. 2018 will see the emergence of a new generation of tools for storing and analyzing graphs and time series at large scale. These new analytic and visualization tools will help product groups devise new offerings, especially for use cases in security and fraud detection.

2. More companies will join data partnerships to share data

In 2016, I started hearing companies express interest in data sharing platforms, and startups have now begun to build data exchanges to allow companies to share data across organizational boundaries, while protecting privacy and IP. Ideas from the blockchain world have inspired some of these initiatives, particularly crypto and distributed control. Data partnerships are taking hold in financial services companies, and I anticipate this trend to spread into other industries this year. 
Continue reading “What lies ahead for data in 2018”

5 AI trends to watch in 2018

[A version of this post appears on the O’Reilly Radar.]

Expect substantial progress in machine learning methods, understanding, and pedagogy

As in recent years, new deep learning architectures and (distributed) training algorithms will lead to impressive results and applications in a range of domains, including computer vision, speech, and text. Expect to see companies make progress on efficient algorithms for training, inference, and data processing on edge devices. At the same time, collaboration between machine learning experts will produce interesting breakthroughs—examples include work that draws from Bayesian methods and deep learning and work on neuroevolution and gradient-based deep learning.

However, as successful as deep learning has been, our level of understanding of why it works so well is still lacking. Both researchers and practitioners are already hard at work addressing this challenge. We anticipate that in 2018 we’ll see even more people engage in improving theoretical understanding and pedagogy.

New developments and lowered costs in hardware will enable better data collection and faster deep learning

Deep learning is computationally intensive. As a result, much of the innovation in hardware pertains to deep learning training and inference (on both the edge and the server). Look for new processors, accompanying software frameworks and interconnects, and optimized systems assembled specifically to allow companies to speed up their deep learning experiments to emerge from established hardware companies, cloud providers, and startups in the West and in China.

But the data behind deep learning has to be collected somehow. Many industrial AI systems rely on specialized sensors — LIDAR for instance. Costs will continue to decline as startups produce alternative sensors and new methods for gathering and using data, such as high-volume, low-resolution data from edge devices and sensor fusion.
Continue reading “5 AI trends to watch in 2018”

8 fintech trends for 2018

[A version of this post appears on the O’Reilly Radar.]

AI, blockchain, payment regionalization, and other fintech trends to watch.

2017 saw big changes, a lot of investment, and some regulatory challenges in fintech. What will 2018 bring? Here’s what we’ll be watching in the coming year.

1. AI will be implemented across the stack

AI is sweeping across all industry sectors, including financial services. AI touches customer interactions (voice services like Siri and dialog systems), fraud detection, trading, and risk management (machine learning), and is being used to automate many back-office tasks (robotic process automation). AI technologies are also giving rise to new fintech startups that use techniques like computer vision to unlock new datasets (e.g., aerial images).

2. New products will make advanced analytics easier

Talk to any vendor or startup in big data analytics or cloud computing and they probably have key customers in financial services. This means that many technology providers will create products tailored for finance (most likely products that comply with existing regulations), which lowers the barrier to using advanced analytics.
Continue reading “8 fintech trends for 2018”

Bringing AI into the enterprise

[A version of this post appears on the O’Reilly Radar.]

The O’Reilly Data Show Podcast: Kris Hammond on business applications of AI technologies and educating future AI specialists.

In this episode of the Data Show, I spoke with Kristian Hammond, chief scientist of Narrative Science and professor of EECS at Northwestern University. He has been at the forefront of helping companies understand the power, limitations, and disruptive potential of AI technologies and tools. In a previous post on machine learning, I listed types of uses cases (a taxonomy) for machine learning that could just as well apply to enterprise applications of AI. But how do you identify good use cases to begin with?

A good place to start for most companies is by looking for AI technologies that can help automate routine tasks, particularly low-skill tasks that occupy the time of high-skilled workers. An initial list of candidate tasks can be gathered by applying the following series of simple questions:

  • Is the task data-driven?
  • Do you have the data to support the automation of the task?
  • Do you really need the scale that automation can provide?

Continue reading “Bringing AI into the enterprise”