Python is slow when iterating over a large list

This should not be slow with Python native lists - but maybe ODBC's driver is returning a "lazy" object that tries to be smart but just gets slow. Try just doing

allIDRows = list(clientItemsCursor.fetchall())

in your code and post further benchmarks.

(Python lists can get slow if you start inserting things in its middle, but just iterating over a large list should be fast)

It's probably slow because you load all result in memory first and performing the iteration over a list. Try iterating the cursor instead.

And no, scripts shouldn't be that slow.

clientItemsCursor.execute("Select ids from largetable where year =?", year);for clientItemrow in clientItemsCursor:    aID = str(clientItemrow[0])    count = count + 1

More investigation is needed here... consider the following script:

bigList = range(500000)doSomething = ""arrayList = [[x] for x in bigList]  # takes a few secondsfor x in arrayList:    doSomething += str(x[0])    count+=1

This is pretty much the same as your script, minus the database stuff, and takes a few seconds to run on my not-terribly-fast machine.