How do I correct my htaccess for proxying search engine crawl requests? How do I correct my htaccess for proxying search engine crawl requests? apache apache

How do I correct my htaccess for proxying search engine crawl requests?


First avoid the repetetions

<IfModule mod_rewrite.c>    RewriteEngine On    <IfModule mod_proxy_http.c>    RewriteCond %{REQUEST_FILENAME} -f [OR]    RewriteCond %{REQUEST_FILENAME} -d    RewriteCond %{HTTP_USER_AGENT} googlebot [NC,OR]    RewriteCond %{QUERY_STRING} _escaped_fragment_    # Proxy the request ... works for inner pages only    RewriteRule ^(?!.*?)$ http://example.com:3000/https://example.com/$1 [P,L]    RewriteBase /    RewriteRule ^index\.php$ - [L]    RewriteCond %{REQUEST_FILENAME} !-f    RewriteCond %{REQUEST_FILENAME} !-d    RewriteRule . /index.php [L]    </IfModule></IfModule>

Then change ^(?!.*?)$ to ^.*$ or with a good pattern like [a-zA-Z0-9-.]*. Don't forget to use 0 or more flag (*) there.

The correct code will be

<IfModule mod_rewrite.c>    RewriteEngine On    <IfModule mod_proxy_http.c>    RewriteCond %{REQUEST_FILENAME} -f [OR]    RewriteCond %{REQUEST_FILENAME} -d    RewriteCond %{HTTP_USER_AGENT} googlebot [NC,OR]    RewriteCond %{QUERY_STRING} _escaped_fragment_    # Proxy the request ... works for inner pages only    RewriteRule ^(.*)$ http://example.com:3000/https://example.com/$1 [P,L]    RewriteBase /    RewriteRule ^index\.php$ - [L]    RewriteCond %{REQUEST_FILENAME} !-f    RewriteCond %{REQUEST_FILENAME} !-d    RewriteRule . /index.php [L]    </IfModule></IfModule>


Change the RewriteRule to:

RewriteRule ^(.*)/?$ http://example.com:3000/https://example.com/$1 [P,L]