Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Robots.txt disable query parameters scanning with exceptions

I am implementing some SEO changes including robots.txt changes and I am solving this problem.

I need to disable (or disallow to be precise) every URL that contains some query URL params (?xy=...) So I have added this rule

Disallow: /*?*

But there is a small problem. I need to allow specific query params for specific sub URL. So lets say I need to allow /picture?path=XY. This should be allowed but for example /picture-other?path=XY not so as /picture?other_param=XY (the /picture is okay but there is no just param named path)

Is it possible to do this in robots.txt?

EDIT: (Maybe more clear instructions added)

Disable ALL URLS containing query parameters (like /x?param=1) but allow query parameter path on URL /picture. So only one allowed URL with query param will be /picture?path=XY no other URLs containing query parameters will be allowed for robots

Examples

Allowed

/picture?path=XY
/literally/any/route/without/query/param/you/can/imagine
/home
/

Now allowed

/picture?param=1
literally/any/route/with/query/param/you/can/imagine?param=1
/some/other/url?query=5&param=1
/some/other/route/with/path/param?path=XY
like image 970
Johnczek Avatar asked Oct 29 '25 16:10

Johnczek


1 Answers

Disallowed paths you provided:

/picture?param=1
literally/any/route/with/query/param/you/can/imagine?param=1
/some/other/url?query=5&param=1
/some/other/route/with/path/param?path=XY

Robots:

Allow: /*
Disallow: /picture?param=*$
Disallow: *param?*
Disallow: *param=*

Regex supported by most (if not all) search-engines:

* - wildcard

$ - end of URL (string)

So if you want to disallow all urls containing param just wrap * around it.

Update:

Allow: /picture?*
Disallow: *?*=*
Disallow: *?*=*&*=*
Disallow: *?*=*=*

Will Block:

http://my-domain.com/test?xwadwa=1
http://my-domain.com/dwa?query=1
http://my-domain.com/test?dwadwa=1
http://my-domain.com/test?dwadwa=1&zxxxa=1
http://my-domain.com/test/dwa/dwa/dwa/dwa/dwa/dwa?xxxx=1

But will allow:

http://my-domain.com/picture?everything_after_this_point
http://my-domain.com/picture?everything_after_this_point&query=32131
http://my-domain.com/
http://my-domain.com/test/
http://my-domain.com/test/test/
http://my-domain.com/test/test/test/
http://my-domain.com/test/test/test/test
like image 181
Joel Avatar answered Oct 31 '25 09:10

Joel



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!