bparodi/francescomecca.eu
1
Fork 0
Code Issues Pull requests Packages Projects Releases Wiki Activity

robots

Browse source
This commit is contained in:
Francesco Mecca 2021-12-02 14:38:20 +00:00
parent ddebc07e3b
commit bc9bafc0ce
1 changed files with 34 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

34
output/robots.txt Normal file
View file

@ -0,0 +1,34 @@
User-agent: SemrushBot
Disallow: /
User-agent: PetalBot
Disallow: /
User-agent: googlebot-image
Disallow: /
User-agent: googlebot-mobile
Disallow: /
User-agent: MSNBot
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: Robozilla
Disallow: /
User-agent: baiduspider
Disallow: /
User-agent: naverbot
Disallow: /
User-agent: yeti
Disallow: /
User-agent: yahoo-mmcrawler
Disallow: /
User-agent: psbot
Disallow: /
User-agent: yahoo-blogs/v3.9
Disallow: /
User-agent: *
Disallow:
Crawl-delay: 120
Disallow: /git/pesceWanda/*/*