Microsoft has discovered a new, widespread, ongoing threat that aims to infect Kubernetes clusters running Kubeflow instances with malicious TensorFlow pods that mine cryptocurrencies. Kubeflow is a popular open-source framework for conducting machine learning (ML) tasks in Kubernetes, while TensorFlow is an end-to-end, open-source ML platform.
Microsoft security experts cautioned on Tuesday that they noticed a rise in TensorFlow pod deployments on Kubernetes clusters at the end of May — pods that were running legal TensorFlow images from the official Docker Hub account. However, a closer examination of the pods' entry point revealed that they are used to mine cryptocurrency.
In a post on Tuesday, Yossi Weizman, a senior security research software engineer at Microsoft's Azure Security Center, said that the "burst" of malicious TensorFlow deployments was "simultaneous," implying that the attackers scanned the clusters first, kept a list of potential targets, and then fired on all of them at the same time. The attackers used two distinct images, according to Weizman. The first is the most recent version of TensorFlow (tensorflow/tensorflow:latest), and the second is the most recent version with GPU support (tensorflow/tensorflow:latest-gpu).
According to Weizman, using TensorFlow images in the network "makes a lot of sense," because “if the images in the cluster are monitored, usage of a legitimate image can prevent attackers from being discovered.” Another rationale for the attackers' decision is that the TensorFlow image they chose is an easy way to conduct GPU activities using CUDA, which "allows the attacker to optimize the mining gains from the host," according to him.
The newly found vulnerability is comparable to a cryptocurrency mining attack revealed by Microsoft in June. That previous campaign also targeted Kubeflow workloads, launching a broad XMRIG Monero-mining campaign by exploiting misconfigured dashboards. The most recent campaign includes the following changes: According to Weizman, the attackers abused their access to the Kubeflow centralized dashboard to establish a new pipeline this time.
Kubeflow Pipelines is a framework for creating machine learning pipelines based on Argo Workflow, an open-source, container-native workflow engine for coordinating parallel jobs. A pipeline is a collection of steps, each of which functions as its own container, that together creates an ML workflow.
Users of Kubeflow should ensure that the centralized dashboard is not insecurely exposed to the internet, according to Microsoft.
