Why is TF Keras inference way slower than Numpy operations?
A little late, but maybe useful for someone:
Replace model.predict(X) with model.predict(X, batch_size=len(X))
That should do it.
Are you running your Keras model (with TensorFlow backend) in a loop? If so, Keras has a memory leak issue identified here: LINK
In this case you have to import the following:
import keras.backend.tensorflow_backendimport tensorflow as tffrom keras.backend import clear_sessionFinally, you have to put the following at the end of every iteration of a loop after you're done doing your computations:
clear_session()if keras.backend.tensorflow_backend._SESSION: tf.reset_default_graph() keras.backend.tensorflow_backend._SESSION.close() keras.backend.tensorflow_backend._SESSION = NoneThis should help you free up memory at the end of every loop and eventually, make the process faster. I hope this helps.
The memory leak issue still seems to persist in Keras. The following lines of code mentioned in that issue did the trick for me:
import ... as Kimport gcmodel = ....del modelK.clear_session()gc.collect()