It’s just data
Many people have been doing this for a while, but as a public service announcement for those who may not have thought to do this:
A site can set up a robots.txt file which tells robots (which includes Google) to not index a particular file or directory.
Here's mine. It is pretty easy to do.
Will it stop referrer spam? No. Many spammers are quite satisfied with fractional percent return on investment. But it allows you to opt out.
The link to your robots.txt is broken - you have it pointing to robots/txt
Posted by Luke Hutteman atLuke: thanks! Fixed.
Posted by Sam Ruby atThe real trick is to dynamically generate a robots.txt file based on who is looking. This lets google and other trusted 'bots archive you, others not to.
take a look at
[link] for an example -you can see by the final comment that it is a dynamic page.
...bla bla, conversations (asynchrones) et quelques liens :...
Excerpt from .Conforme atThe real trick is to dynamically generate a robots.txt file based on who is looking. This lets google and other trusted 'bots archive you, others not to.
Posted by avelo at