Custom robots.txt when in "headless" or "private" mode

Developer help
#1

  • What’s your URL? https://ghost.optezo.com

  • What version of Ghost are you using? Latest, 3.37.1
    And

  • How was Ghost installed and configured? Via CLI, ubuntu 18.04

  • What Node version, database, OS & browser are you using? Node 10.23.0, mysql local

  • What errors or information do you see in the console? None

I’m able to successfully put a custom robots.txt file in the base of my theme and have that file used, but I’m trying to run ghost headless, using the API from another site. When I make the site “private”, the robots.txt automatically turns into

User-agent: * Disallow: /

and my custom robots.txt file isn’t use. I need robots to access the /content/images folder, or things such as twitter cards and the like will not work. These agents will honor the robots.txt file and will not populate images.

How can I add a custom robots.txt file for when the site is in private mode? I can’t find anything in the docs or forums related to this. Thanks!

1 Like
#2

btw, if you look now at my robots.txt on that site, you’ll see that it’s different, but that’s just because I edited the file at

ghost/versions/3.37.1/core/frontend/apps/private-blogging/robots.txt

I know this isn’t a long term solution, as it’ll stop working on the next software upgrade. Hopefully someone has a good solution for this. Thanks!

#3

Hi, is there any way to do this currently in ghost?

1 Like
#4

Were you able to find a longer term solution to this?

#5

I was not able to find another solution. This is just part of my upgrade process to manually update this file.

#6

If your setup has ghost running behind nginx or any other reverse proxy, then you can configure it to serve /robots.txt as a static file from a custom location instead of asking ghost to serve it.

I have done a similar configuration using Caddy for my staging environment.

#7

Excellent idea, srijan. Why didn’t I think of that. Thanks for replying.

1 Like
#8

Are there any other workaround to this for if we don’t have a Ghost installation of our own? I’m currently using Ghost Pro.

#9

Hi Ghost team. Is there any solution to this problem? I added custom robots.txt to my theme but it is ignored when in private mode. Twitter bots doesn’t work with Disallow: / and it causes images to be blocked.

#10

There are some notes about this in our JAMstack docs:

In most cases the best approach is to either use an external storage adapter, or download all images locally as a part of your front-end build process so the assets are served statically with the rest of the site.

An alternative workaround to using Private <ode is to make a custom/empty Ghost theme with a no-index meta tag and a clientside redirect using window.location in default.hbs. This will prevent search engines from indexing pages, and redirect any traffic to the static front end, while leaving image and API routes in tact.