How to write Rake task to import data to Rails app?

I wouldn't delete the products and vendors tables on every cycle. Is this a rails app? If so there are some really nice ActiveRecord helpers that would come in handy for you.

If you have a Product active record model, you can do:

p = Product.find_or_initialize_by_identifier(<id you get from file>)p.name = <name from file>p.size = <size from file>etc...p.save!

The find_or_initialize will lookup the product in the database by the id you specify, and if it can't find it, it will create a new one. The really handy thing about doing it this way, is that ActiveRecord will only save to the database if any of the data has changed, and it will automatically update any timestamp fields you have in the table (updated_at) accordingly. One more thing, since you would be looking up records by the identifier (id from the file), I would make sure to add an index on that field in the database.

To make a rake task to accomplish this, I would add a rake file to the lib/tasks directory of your rails app. We'll call it data.rake.

Inside data.rake, it would look something like this:

namespace :data do  desc "import data from files to database"  task :import => :environment do    file = File.open(<file to import>)    file.each do |line|      attrs = line.split(":")      p = Product.find_or_initialize_by_identifier(attrs[0])      p.name = attrs[1]      etc...      p.save!    end  endend

Than to call the rake task, use "rake data:import" from the command line.

Since Products don't really change that often, the best way I would see is to update only the records that change.

1. Get all the deltas
2. Mass update using a single SQL statement

If you are having your normalization code in the models, you could use Product.create and Vendor.create or else it would be just a overkill. Also, Look into inserting multiple records in a single SQL transaction, its much faster.

• Create an importer rake task that is cronned
• Parse the file line by line using Faster CSV or via vanilla ruby like:

file.each do |line| products_array = line.split(":")end

• Split each line on the ":" and push in into a hash

• Use a find_or_initialize to populate your db such as:

  Product.find_or_initialize_by_name_and_vendor_id("foo", 111)