Possible to change tab completion behavior in fish shell?

I commend you for writing up this detailed and thoughtful post, and it deserves an equally detailed and thoughtful response!

The tab completion behavior has been rewritten in fish top-of-tree (not yet released), and is referred to as the "new pager." You can see the design goals and discussion here. I put a note at the bottom of this reply for how to get it.

Shells are personal, and like anything personal, rationalizations and justifications aren't worth anything: you either like it, or you don't, and we may not even be conscious of the factors influencing our feelings. So all I can really say is try it, see how you feel, and (please) report back.

I put up a short little screencast of the new pager on YouTube. Things to notice: 1. the menu is dismissed just by typing more, 2. it "unfurls" progressively (requires a few tabs to become fully visible), never modally takes over your screen even when there's a huge number of completions, and is easily searchable and navigable, and 3. escape will always dismiss it and put your command line back to just what you typed.

Let me go through your concerns individually:

"I have to hit tab an unknown number of times to get the value I wanted". With the new pager, the selected item is highlighted in the menu. This sounds minor, but personally I believe this makes a huge difference: the number of additional times to hit tab becomes known, and since your finger is over tab, it's often easier to just hit it a few more times than to type additional letters. You can also use the arrow keys to navigate.

"I have no way of getting the entry context back to only the letters that I've actually typed up to this point". With the new pager, the escape key does exactly that. It's easy to press since escape is right above tab, where your finger is.

"What if there happen to be a whole bunch of things that start with ba in this directory -- I'm totally screwed is what happens". Neither bash nor old-pager-fish handles large numbers of completions well. fish would drop you into this modal paging environment, while bash breaks your flow with the modal "Display all 1002 possibilities? (y or n)" dialog that forces you to stop what you're doing and hit 'n'.

I think you'll love how the new pager handles this. Initially you get a short menu, that fills a maximum of five lines below your prompt (not above, and not replacing). This menu is non-modal, and is dismissed by typing more or hitting escape. If you hit tab again, the menu grows to show more completions, but is still non-modal. There's never a jarring transition.

"it does not allow using tab to complete nested paths" Sorry, I'm not sure what you mean by this. Both bash and fish append a / when tab completing a directory.

"much more difficult to discover disambiguation sequences when in large directories" With the new pager, you can hit escape, type some more, and then tab again. Or you can search the menu: put the focus in the menu and type something, and it's filtered. See the screencast above.

"in general requires you to 'be careful' before you hit tab which makes you hit it less often and decreases its utility" A very valid point, which the new pager addresses in a few ways. First of all, it uses a notion of progressive disclosure, which means that it takes "work" to output a lot of data. Second, it never "takes over your screen" like the old modal pager. And lastly, you can hit escape to get back to just what you typed, and since the pager appears below the prompt, it won't leave little turds in your scrollback like bash does.

If you're using homebrew, you can install from master via brew install fish --HEAD. There's also nightly builds for Linux. And lastly, feel free to open an issue at https://github.com/fish-shell/fish-shell/issues with any ideas for improvements you have.