Is there a parallelized version of scipy.optimize.minimize?
We implemented a parallel version of scipy.optimize.minimize(method='L-BFGS-B') in the package optimparallel available on PyPI. It can speedup the optimization by evaluating the objective function and the (approximate) gradient in parallel. I have not tested it with 5000 parameters. But, with fewer parameters we observe good parallel scaling.
The Python package is a Python implementation of the R package optimParallel. The method is documented in this R journal article.
Here is an illustration of the possible parallel scaling: