How can I send data in csv format from memory to a database without saving the csv to disk? How can I send data in csv format from memory to a database without saving the csv to disk? json json

How can I send data in csv format from memory to a database without saving the csv to disk?


LOAD DATA INFILE will be, by far!, the fastest way to go. But it does require you to put the CSV data into a file system. You may have a temporary, even a RAM, file system in your setup for doing this.

In the dotnet world, there's a robust module for reading CSV data from streams. Files are a special case of streams. The module is called, for historic reasons, Microsoft.VisualBasic.FileIO.TextFieldParser. (It works fine outside Visual Basic, it just has a name from long ago.)

If you use this approach, you can improve performance by inserting multiple rows of the CSV in each transaction. There are two ways to do that.

One is multirow inserts, like so

     INSERT INTO tbl      (col,col,col)     VALUES      (val, val, val),     (val, val, val),     (val, val, val),     ...     (val, val, val);

The other is to use START TRANSACTION, then do a couple of hundred inserts, then do COMMIT, then repeat that until you're done. Experience teaches that will make your insertion reasonably fast.

Parsing JSON in a MySQL stored procedure? Absurdly hard to debug. And, you'll still have to manage the transactions as I mentioned.