How do I correct my htaccess for proxying search engine crawl requests?
First avoid the repetetions
<IfModule mod_rewrite.c> RewriteEngine On <IfModule mod_proxy_http.c> RewriteCond %{REQUEST_FILENAME} -f [OR] RewriteCond %{REQUEST_FILENAME} -d RewriteCond %{HTTP_USER_AGENT} googlebot [NC,OR] RewriteCond %{QUERY_STRING} _escaped_fragment_ # Proxy the request ... works for inner pages only RewriteRule ^(?!.*?)$ http://example.com:3000/https://example.com/$1 [P,L] RewriteBase / RewriteRule ^index\.php$ - [L] RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d RewriteRule . /index.php [L] </IfModule></IfModule>
Then change ^(?!.*?)$
to ^.*$
or with a good pattern like [a-zA-Z0-9-.]*
. Don't forget to use 0 or more flag (*
) there.
The correct code will be
<IfModule mod_rewrite.c> RewriteEngine On <IfModule mod_proxy_http.c> RewriteCond %{REQUEST_FILENAME} -f [OR] RewriteCond %{REQUEST_FILENAME} -d RewriteCond %{HTTP_USER_AGENT} googlebot [NC,OR] RewriteCond %{QUERY_STRING} _escaped_fragment_ # Proxy the request ... works for inner pages only RewriteRule ^(.*)$ http://example.com:3000/https://example.com/$1 [P,L] RewriteBase / RewriteRule ^index\.php$ - [L] RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d RewriteRule . /index.php [L] </IfModule></IfModule>