Alvin63
Guest
offline
I've never actually used a robots.txt (didn't get round to it and it hadn't been an issue generally) and have been reading around on here to find various examples and still find it a bit confusing (due to there being different examples) and also not quite understanding some of it. I thought it was just to block robots from scanning the site, but it seems people are adding forum sections to it as well.
Last time I looked into this I thought you needed to add "Allow" at the end to allow...
Read more
Continue reading...
Last time I looked into this I thought you needed to add "Allow" at the end to allow...
Read more
Continue reading...