How to Prevent Filtered Pages from Being Crawled in Improved Layered Navigation?
The Improved Layered Navigation module generates filter links as standard HTML links, for example:
- <a href="...">
As a result, filtered pages may appear in Google Search Console, consume crawl budget, and create duplicate content issues. This can happen even when filtered pages contain a meta robots tag such as the following:
- <meta name="robots" content="noindex, nofollow">
Google treats nofollow as a hint rather than a strict directive, so it may still decide to crawl certain links.
To prevent search engines from crawling filtered pages, configure a filter URL key and block it in the robots.txt file.
- Go to the module settings in Amasty > Improved Layered Navigation: SEO > Configuration;
- Find the Filter Key setting and set any custom value for the filter key, for instance shopby
- Go to the design configuration in Content > Design > Configuration > Global
- Open the robots.txt custom instructions section in Search Engine Robots;
- Insert the record in the Edit custom instruction of robots.txt File using the filter key from step 3:
Disallow: */shopby/* - Save the configuration.
Related Articles
Improved Layered Navigation/Shop By Brand API
We have created public API collections for Shop By Page and Shop By Base submodules that are included into Improved Layered Navigation, with all available queries and sample data that you can both test via our demo endpoints and copy to run local ...Improved Layered Navigation | GraphQL methods
The Improved Layered Navigation extension doesn't have its own GraphQL requests. Instead, it extends the default Magento requests by introducing supplementary details. You can get additional information stored by the Improved Layered Navigation ...How to check and fix robots meta tags on filtered pages?
The Improved Layered Navigation extension creates a large number of filtered URLs, and their SEO behavior depends on several configuration layers. Because of this, the final robots meta tag is not controlled by a single setting. Instead, it is the ...How to disable Improved Layered Navigation on a specific store?
For multistore installations, this option is not covered. Although the settings for the Improved Layered Navigation module can be configured individually for each store view, the module cannot be disabled on a store view basis.How to resolve the "Please select brand attribute in Stores → Configuration → Amasty Extensions → Improved Layered Navigation: Brands." frontend error?
The message “Please select brand attribute in Stores → Configuration → Amasty Extensions → Improved Layered Navigation: Brands” appearing on your All Brands Page indicates that additional configuration is required to display the brands on the ...
Announcements
⚡ Have you enlarged the functionality of the Amasty extension with your own code? Share your improvements with us!
At Amasty, we understand that some of our clients require customized features to perfectly align the extension with their business needs. If you have developed new functionality within an Amasty module and would like to share it, please submit a ticket