Search Appliance SBE

 

Thunderstone Search Appliance Manual

Robots.txt

The Search Appliance comes with no robots.txt file. If you want to control how web walking robots visit your search pages you can upload a robots.txt file. This robots.txt file will apply to all pages returned bythe Search Appliance including admin, documentation, and searches whether accessed by HTTP or HTTPS. See http://www.robotstxt.org for the syntax of robots.txt files.

If you no longer wish to have a robots.txt check Delete.

Note: You may also control robots for searches on a profile by profile basis by using meta robots within the top HTML of the search settings or the custom XSL if you're using an XSL style sheet for search results. See http://www.robotstxt.org for the syntax of the meta robots tag.


Copyright © Thunderstone Software     Last updated: Mar 19 2020