Within vim's regex engine, why are some metacharacters escaped and some are not?

This is because vim (actually vi) created their own regex flavor long before perl did. Even POSIX BRE and ERE came after vim^wikipedia. They are still maintaining their own flavor so it's quite different.

To make the answer more resourceful here is a quote from ed's wiki.

The editor was originally written in PDP-11/20 assembler in 1971 by Ken Thompson. Many features of ed came from the qed from his alma mater University of California at Berkeley3 Thompson was very familiar with qed, and had reimplemented it on the CTSS and Multics systems. His versions of qed were the first to implement regular expressions. Although regular expressions are part of ed, their implementation is considerably less general than that in qed.

Aspects of ed went on to influence ex, which in turn spawned vi. The non-interactive Unix command grep was inspired by a common special uses of qed and later ed, where the command g/re/p means globally search for the regular expression re and print the lines containing it. The Unix stream editor, sed implemented many of the scripting features of qed that were not supported by ed on Unix. In turn sed influenced the design of the programming language AWK - which inspired aspects of Perl.

These two paragraphs have a lot of information! I wish I could bold it all. Some highlights,

• Ken Thompson wrote ed in 1971. ed was actually a reimplentation of qed.
• Ken added Regular Expression in his version of qed which is actually ed.
• Inspired by ed, in 1976 William Joy (known as Bill Joy) wrote ex^wikipedia
• Bill Joy in 1976 wrote vi as the visual mode for a line editor called ex ^wikipedia
• grep was inspired by special uses of qed and later ed.
• sed was implemented as many of the scripting features of qed that were not supported by ed on Unix
• sed influenced the design of awk.

So vi Regular Expression were in ed which was written in 1971. It's long before any other regular expression flavor.