Dependency Injection in Hadoop Mapper Dependency Injection in Hadoop Mapper hadoop hadoop

Dependency Injection in Hadoop Mapper


Went through a couple of books on Hadoop. Seems like, 'configure()' method is the only way, to do this.

Already added the code in the Question


This is a common dilemma on Hadoop because the Mapper and Reducer are handed to you by the framework. I found it best to call out to a lightweight DI framework from the setup() methods. Read my blog post about Dependency Injection on Hadoop. I wrote a single class to handle DI called Spit-DI which is available on github and uses the JSR-250 @Resource annotation for injections.

It ends up looking like this:

class MovieMapper extends Mapper {   @Resource   private Movie movie;   @Override   protected void setup(Context context) {      DependencyInjector.instance().using(context).injectOn(this);   }}class Movie {   @Resource   private Counter numMoviesRequested;      public Integer getYear(String title) {      numMoviesRequested.increment(1);     // more code...   }}/** * You can have a wrapper class around Spit-DI for all your configuration. * (We have a TestDependencyInjector as well for the context of unit testing.) */class DependencyInjector {   private SpitDI spit = new SpitDI();   public void injectOn(Object instance) {      spit.inject(instance);   }   public DependencyInjector using(final Mapper.Context context) {      spit.bindByType(Movie.class, new Movie());      spit.bindByName(Counter.class, "numMoviesRequested", context.getCounter("movies", "numMoviesRequested");      return this;   }}