Tensorflow GPU Could not load dynamic library 'cusolver64_10.dll'; dlerror: cusolver64_10.dll not found
Move to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\binRename file cusolver64_11.dll To cusolver64_10.dll 
cusolver64_10.dll 
TL;DR For TensorFlow ver >= 2.4.0 on Windows, install exactly those versions of CUDA Toolkit and cuDNN highlighted below i.e. those listed in the official requirements.(v11.0 as opposed to v11.2)
On Windows, the TensorFlow^ install requirements at the time of writing are as stated here
NVIDIA® GPU drivers —CUDA® 11.0 requires 450.x or higher.
CUDA® Toolkit —TensorFlow supports CUDA® 11 (TensorFlow >= 2.4.0)
CUPTI ships with the CUDA® Toolkit.
cuDNN SDK 8.0.4.
(Optional) TensorRT 6.0 to improve latency and throughput for inference on some models.
The problem you are facing has probably to do with the version of CUDA® Toolkit. Tensorflow is picky about the version of dependencies. Have a look inside C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\bin**. You should be able to find most^^ of the dlls needed by TensorFlow there. You may notice that it contains cusolver64_11.dll as opposed to the expected cusolver64_10.dll as stated in the output.
Though the renaming hack mentioned in an answer above works, it's not guaranteed to work reliably all the time. The simple and correct solution is to install the correct dependencies, to begin with.
At the time of writing the compatible versions of CUDA Toolkit and cuDNN are
CUDA Toolkit 11.0 (May 2020)cuDNN v8.0.4 (September 28th, 2020), for CUDA 11.0 from among the plethora of available versions of both, listed here & here.
More recent versions (I tested v11.0 onwards) aren't yet supported. I remember having the same problems with an earlier version of TensorFlow a few years back.
^ For ver >1.15, TensorFlow has GPU support included by default hence the CUDA requirements. When unavailable, TensorFlow works fine - it just reverts to CPU execution.
** Or wherever you installed the toolkit
^^ cudnn64_8.dll comes with cuDNN SDK
I had the same problem. It turns out that CUDA 11.0 contains cusolver64_10.dll (that's probably why they indicate CUDA v11.0 in the tensorflow build guide here https://www.tensorflow.org/install/source_windows). Make sure to download cudnn as well!