Getting existing mapreduce job from cluster (the job could be running or completed) Getting existing mapreduce job from cluster (the job could be running or completed) hadoop hadoop

Getting existing mapreduce job from cluster (the job could be running or completed)


The problem was with with a recent yarn upgrade that required enabling MR history server on my system. This fixed the issue. I recently upgraded from MR v1 to v2 and in that upgrade, all completed jobs are now moved to the history server.


You look for getAllJobStatuses() that return JobStatus[]:

  List<JobStatus> runningJobs = new ArrayList<JobStatus>();  List<JobStatus> completedJobs = new ArrayList<JobStatus>();  for (JobStatus job : cluster.getAllJobStatuses()) {    if (!job.isJobComplete()) {      runningJobs.add(job);    }    else {      completedJobs.add(job)    }  }  // list of running JobIDs  for (JobStatus rjob : runningJobs) {        System.out.println(rjob.getJobID().toString());  }  // list of completed JobIDs  for (JobStatus cjob : completedJobs) {        System.out.println(cjob.getJobID().toString());  }  // to print out short report on running jobs:  // displayJobList(runningJobs.toArray(new JobStatus[0]));