Do I need multiple cursor objects to loop over a recordset and update at the same time? Do I need multiple cursor objects to loop over a recordset and update at the same time? database database

Do I need multiple cursor objects to loop over a recordset and update at the same time?


I think you have roughly the right architecture -- presenting it in terms of "cursors" WILL confuse the "old SQL hands", because they'll be thinking of the many issues connected with DECLARE foo CURSOR, FETCH FROM CURSOR, WHERE CURRENT OF CURSOR, and other such beauts having to do with SQL cursors. Python DB API's "cursor" is simply a convenient way to package and execute SQL statements, not necessarily connected with SQL cursors -- it won't suffer from any of those problems -- though it may present its (completely original) own ones;-) But, with the "batching" of results you're doing, your proper commits, etc, you have preventively finessed most of those "original problems" I had in mind.

On some other engines I'd suggest doing first a select into a temporary table, then reading from that temporary table while updating the primary one, but I'm uncertain how the performance would be affected in sqlite, depending on what indices you have (if no index is affected by your update, then I suspect that such a temporary table would not be an optimization at all in sqlite -- but I can't run benchmarks on your data, the only real way to check performance hypotheses).

So, I'd say, go for it!-)


Is it possible to create a DB function that will process your content? If so, you should be able to write a single update statement and let the database do all the work. Eg;

Update data_tableset processed_col = Process_Column(col_to_be_processed)


Cursors are bad bad bad for a multitude of reasons.

I'd suggest (and a lot of others will definitely chime in) that you use a single UPDATE statement instead of going the CURSOR route.

Can your Processed_Content be sent as a parameter to a single query that does set based operations like so:

UPDATE data_tableSET processed_content = ?WHERE processed_content IS NULLLIMIT 1000

Edited based on responses:

Since every row has a unique value for Processed_Content, you have no option but to use a recordset and a loop. I have done this in the past on multiple occasions. What you are suggesting should work effectively.