Robots.txt configuration

Developer help
1

Want to modify the robots.txt, want to change currently robots.txt become:

User-agent: *
Sitemap: https://domain.com/sitemap.xml
Disallow: /ghost/
Disallow: /p/
Disallow: /member/
Disallow: /tag/
Disallow: /author/

Any body can help me?

2

As far as I’m aware, if you upload a robots.txt file as part of your theme, it should supersede the default Ghost robots file

2 Likes
3

is there a way to modify robots.txt on ghost pro’s $9/month plan?

4

Download your theme files, create a robots.txt file, save and reupload the theme.

5

You can’t upload themes in the basic plan.

6

I wish it was. Such a bummer it isn’t doable on the lowest tier :zipper_mouth_face:

7

Hey, Dan (and @Cowgirl too) depending on what your actual objective is there may be options to achieve them without even touching robots.txt (though I won’t know without chatting about that) - if you haven’t already, email support@ghost.org to discuss your Ghost(Pro) site.

8

Do you put the robots.txt files in the theme’s root directory or does it go somewhere else?

9

Answered my own question. Yes put robots.txt in the root directory of your theme.

I ended up adding and excluding some of the AI bots from my site. I pulled these examples from the New York Times’ robots.txt file. I figured they would have a good handle on what to exclude since they are suing OpenAI and hate the idea of training AI services on their journalism. I added the text below to the existing text in the default ghost robots.txt.

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ia_archiver
Disallow: /

User-Agent: omgili
Disallow: /

User-Agent: omgilibot
Disallow: /
1 Like