How to efficiently feed FAST ESP with gigabytes of data with .NET

It sounds like you're working with a set of issues more than a specific C# feeding speed issue.

A few questions up front - is this 60gb data to be consumed every weekend or is it an initial backfill of the system ? Does the data exist as items on the filesystem local to the ESP install or elseware ? Is this a single internal ESP deployment or a solution you're looking to replicate in multiple places ? Single node install or multiple (or rather ... how many - single node cap is 20 docprocs) ?

ESP performance is usually limited by number of documents to be handled more than the number of files. Assuming your data ranges between email size 35k data and filesystem size 350k data you 60gb equates to between 180k docs and 1.8mil docs, so to feed that over 48hrs you need to feed between 3750 and 37500 documents per hour. Not a very high target on modern hardware (if you installed this on a VM ... well... all bets are off, it'd be better off on a laptop).

For feeding you have a choice between faster coding & more control with either managing the batches fed yourself or using the DocumentFeeder framework in the api which abstracts a lot of the batch logic. If you're just going for 37.5k docs/hr I'd save the overhead and just use DocumentFeeder - though take care in its config params. Document feeder will allow you to treat your content on a per document basis instead of creating the batches yourself, it will also allow for some measure of automatically retrying based on config. General target should be for a max of 50mb content per batch or 100 docs, whichever comes first. Larger docs should be sent in smaller batches... so if you have a 50mb file, it should ideally be sent by itself, etc. You'd actually lose the control of the batches formed by document feeder... so the logic there is kinda a best effort on the part of your code.

Use the callbacks to monitor how well the content is making it into the system. Set limits on how many documents have been fed that you haven't received the final callbacks for yet. Target should be for X batches to be submitted at any given time -or- Y Mb, pause at either cutoff. X should be about 20 + # of document processors, Y should be in the area of 500-1000Mb. With document feeder it's just a pass/fail per doc, with the traditional system it's more detailed. Only wait for the 'secured' callback ... that tells you it's been processed & will be indexed... waiting for it to be searchable is pointless.

Set some limits on your content... in general ESP will break down with very large files, there's a hard limit at 2gb since it's still 32bit procs, but in reality anything over 50mb should only have the metadata fed in. Also... avoid feeding log data, it'll crush the internal structures, killing perf if not erroring out. Things can be done in the pipeline to modify what's searchable to ease the pain of some log data.

Also need to make sure your index is configured to well, at least 6 partitions with a focus on keeping the lower order ones fairly empty. Hard to go into the details of that one without knowing more about the deployment. The pipeline config can have a big impact as well... no document should ever take 5-8 hours. Make sure to replace any searchexport or htmlexport stages being used with custom instances with a sane timeout (30-60 sec) - default is no timeout.

Last point... odds are that no matter how your feeding is configured, the pipeline will error out on some documents. You'll need to be prepared to either accept that or refeed just the metadata (there are other options, but kinda outside the scope here).

good luck.

First of all, you should use the tasks for such problem.
They can be started sync, async, in thread pool etc, and much more cheaper on memory than models with thread-locking.

I think, the Task.ContinueWith fits perfectly for your problem.

Algorithm will looks like:

1. Gather a queue with data you need to publish.
2. Start a task (or tasks, if you are risky :) which takes the heavier object from queue.(and the smallest object from other side), and start upload it.
3. Create a method for the end of uploading, which will start new task for new queue item.
4. You can use Cancellation tokens for the timeouts.
5. Every time you can define on what item the system get error.

Can you simply use BULK INSERT directly on the database? If not, I suggest you work with the provider of the third-party product so that together you can formulate a workable solution.