Manage multiple clusters in Hadoop OR Distributed Computing Framework Manage multiple clusters in Hadoop OR Distributed Computing Framework hadoop hadoop

Manage multiple clusters in Hadoop OR Distributed Computing Framework


You should used YARN for manage multiple clusters or resources

YARN is the prerequisite for Enterprise Hadoop, providing resource management and a central platform to deliver consistent operations, security, and data governance tools across Hadoop clusters.

Reference


It seems that you have already stored the data on each of the nodes, so you have already solved the "distributed storage" element of the problem.

Since each node's dataset is different, this isn't a parallel processing problem either.

It seems to me that you don't need Hadoop or any other big data framework. However, you can embrace the philosophy of Hadoop by taking the code to the data. You run the clustering algorithm on each node, and then handle the results in whatever way you need. A caveat would be if you also have a problem in loading the data and running the clustering algorithm on each node, but that is a different problem.