The file /etc/inputrc or ~/.inputrc controls the library readline. Readline is used by many many programs (including bash).
/etc/inputrc
~/.inputrc
readline
The second one, set -o vi controls only bash. vimrc has no effect on bash.
set -o vi
vimrc
Only the first line is necessary:
set editing-mode vi
The second line defines how future manual key mappings will be applied:
set keymap vi