I understand how to use, and the use of, robots.txt. That really wasn't my question. I was interested in knowing what those particular lines had been removed from it which is now causing this annoying issue.
All well and good if you have an actual page or menu item. In the 2 examples I quoted, neither exist so, other than what I did (Disallow: /components/) the only other option is also using robots.txt and adding:
Disallow: <specific URL>
[/quote]To prevent a URL from being indexed entirely, you can use the "noindex" meta tag or HTTP header in addition to blocking it in your robots.txt file.
All well and good if you have an actual page or menu item. In the 2 examples I quoted, neither exist so, other than what I did (Disallow: /components/) the only other option is also using robots.txt and adding:
Disallow: <specific URL>
Statistics: Posted by trogladyte — Sat Oct 12, 2024 12:25 pm
