Machine Learning Trends You Need to Know

Insights and trends that will help you navigate the AI landscape.

By Assaf Araki and Ben Lorica.

Automation and democratization are on the rise

AutoML tools are designed to automate the process of training and deploying machine learning. Such tools have progressed to the point where they can produce adequate models for many use cases. Moreover, in domains where model hubs and foundation models (e.g. language models) are available, pre-trained models and embeddings reduce the need to train models from scratch. The presence of pre-trained models drives attention towards tools for tuning, customization, and transfer learning, according to our annual NLP surveys.

Figure 1: Common types of low-code and no-code ML tools.

There will also be more tools developed to make machine learning and artificial intelligence more accessible. We expect a range of solutions including those available though simple declarative programming interfaces, and tools that utilize graphical user interfaces designed for domain experts and business users. With the global talent pool of analysts dwarfing that of data scientists and data engineers, we will likely see more ML tools designed specifically for analysts.

Figure 2: Size of global talent pool for some job categories.

Renewed focus on data (but with a twist)

The rise of model hubs and foundation models shifts the focus from gathering massive amounts of data to collecting data for specific use cases and applications. As a result, there is growing interest in “data-centric AI” – a collection of tools and techniques for cleaning, augmenting, and enhancing datasets to improve the accuracy of machine learning models. Practitioners have long known that better data (not just more data) leads to better models. Data augmentation is key even in state-of-the-art models. According to the 2022 AI Index, nine state-of-the-art AI systems out of the ten benchmarks are trained with extra data.

Figure 3: Data tools and initiatives are back in the spotlight.

Targeted applications will continue to lead the way

Within the community of machine learning researchers and engineers, the best known startups and projects are ones focused on infrastructure and horizontal platforms. But as it turns out, startups that focus on specific problems and use cases tend to be more successful. In our recent post on AI pegacorns (startups with ≧ $100M annual revenue) we found that there are more successful AI application startups compared to successful AI infrastructure startups.

There are numerous research breakthroughs in computer vision, language models, speech processing, and multi-modal models that are just beginning to surface in real-world applications. Our initial list of AI pegacorns has shown that there are many more use cases in the application layer than in the infrastructure layer where a single company can serve the same need across different companies.

Figure 4: The AI Pegacorn Club.; from “The AI $100M Revenue Club”.

Over the long-term, teams will favor platforms over point solutions

There are dozens of solutions that address various aspects of ML including data processing, model building and experiment management, model deployment/observability/governance, and orchestration. There’s a similar explosion in the number of tools for data management and data infrastructure: data engineering teams are now able to build modular data platforms composed of best-in-class tools (“modern data stack”).

Figure 5: Representative sample of companies that address different stages of a machine learning pipeline.

Piecing together multiple ML and MLOps solutions will be too complex for most teams. Consequently, we believe ML is a platform play and companies will use at most two platforms to manage the entire pipeline: one will manage the exploration phase, while another platform will manage deployment and operations. ML training and inference workloads have different characteristics, and different end-users. The exploration phase is performed by data scientists and includes data preprocessing, experiment tracking, dataset versioning, and model management. The deployment phase is handled by production engineers and is focused on deployment efficiency, metrics and monitoring, and incident response.

Practitioners have long known that better data leads to better models

We are seeing companies that adopt different platforms and projects for exploration (MLflow, Weights & Biases, Comet, Aim) and production (Verta, Seldon). Some platforms like Databricks can handle both exploration and deployment. Teams who choose to build their own ML platforms will gravitate towards components that are tightly integrated (Ray AI Runtime; or managed services from major cloud providers).

AI is becoming more affordable and more efficient

Training large neural models, such as foundation models, requires an enormous amount of data and computational resources, so they are dominated by a few research groups. But there are a few hopeful trends worth highlighting. First, pre-trained and foundation models shift the focus from comprehensive model training towards less computationally intensive approaches (fine tuning and transfer learning). Second, algorithmic and systems improvements continue to help lower costs and carbon footprints:

❛Since 2018, the cost to train an image classification system has decreased by 63.6%, while training times have improved by 94.4%. The trend of lower training cost but faster training time appears across other MLPerf task categories such as recommendation, object detection and language processing, and favors the more widespread commercial adoption of AI technologies.❜

Finally, efficiency has become a primary concern. A growing number of researchers are working on Green AI – research that aims to reduce computational costs while achieving novel results. Meta AI released a large language model that required much less training resources than some of its predecessors, and Google Brain published a series of papers on the efficient scaling of language models. Thought leaders from Google hinted at potential efficiencies yet to be unlocked when they noted that “If the whole ML field were to adopt best practices, total carbon emissions from training would reduce.”

Figure 6: Recent growth in interest in Green AI among researchers.

The pipeline from research to industry will remain robust

Figure 7: Top topics found in machine learning papers posted on arxiv.org (cs.LG).

There will continue to be a steady stream of research tools that eventually translate into solutions for real-world, production grade applications. Here are just a few examples of active research areas that are leading to real-world tools and applications:

Graph Neural Networks – Companies like Pinterest, Uber, and Google have reported significant improvements in recommendations, fraud detection, and forecasting models after incorporating GNNs.
Transformers – Transformers are starting to make an impact in domains other than NLP (e.g., in computer vision).
Multi-modal models – SageMaker Healthcare example; Google MUM; OpenAI DALL-E; LAION-5B; and DeepMind’s Gato.
Reinforcement Learning – While RL remains challenging for most teams, we are starting to see a variety of use cases in financial services, retail and ecommerce, security, semiconductors, and beyond.
Robotics – The price of robotic arms decreased by 46.2% in the past five years.

Figure 8: Recent examples of graph neural networks and machine learning from technology companies; from “What is Graph Intelligence?”

Responsible AI is gaining traction

Many teams are beginning to integrate audits and checks for various aspects of Responsible AI (RAI) – an umbrella term used to describe a variety of risks including bias, safety and reliability, privacy and security, and trust and interpretability. While companies in regulated sectors like health care and finance are further along than others, many companies are now in the early stages of developing their RAI strategy. Awareness is high and we expect steady progress in RAI adoption in the years to come.

Figure 9: Analysis of recent job postings that mention aspects of Responsible AI.

Deploying AI ethically and responsibly will involve cross-functional team collaboration

Note that the availability of RAI tools and best practices depends on the domain, type of model, and use case. As an example, detoxification of large language models remains quite challenging and is very much audience and domain specific. Here are specific examples of RAI tools and initiatives:

Fairness – Algorithmic fairness and bias have evolved from academic pursuits to a practical concern for data and machine learning teams. The US National Institute of Standards and Technology (NIST) recently published a new framework on bias. Judging by its track record in cybersecurity, this new NIST framework will influence how companies approach fairness in machine learning.
Interpretable ML – A recent ACM survey paper sketched out ideas on how to foster more widespread adoption of tools for interpretable ML.
Safety – The EU is on the verge of adopting a regulatory framework for AI designed to “ensure that AI systems placed on the EU market are safe and respect existing EU law”.
Data – A number of books, guidelines, and tools have been published on how to measure and mitigate risks that stem from the construction of training data.
Privacy-preserving ML – In a previous post, we described the vibrant ecosystem of tools focused on protecting data while it’s being used for AI and analytics.

Figure 10: Representative sample of Confidential Computing companies and solutions; from “Get Ready For Confidential Computing”.

Closing Thoughts

While algorithms remain the core of AI products, organizations need more than algorithms to build applications. Most AI innovations have focused on algorithmic development. Algorithms used in AI are maturing, and innovation is spreading to other stages of the pipeline. More companies will create platforms that solve specific business problems or enable users to deploy end-to-end solutions rather than tackling one building block at a time.

Assaf Araki is an investment manager at Intel Capital. His contributions to this post are his personal opinion and do not represent the opinion of the Intel Corporation. Intel Capital is an investor in Verta, Anyscale, Opaque, and DataRobot. #IamIntel

Ben Lorica helps organize the Data+AI Summit and the Ray Summit, is co-chair of the NLP Summit, and principal at Gradient Flow. He is an advisor to Databricks, Anyscale, and other startups.

Related content: Other posts by Assaf Araki and Ben Lorica.

FREE Report:

Download