What version of Ghost are you using? Latest, 3.37.1
And
How was Ghost installed and configured? Via CLI, ubuntu 18.04
What Node version, database, OS & browser are you using? Node 10.23.0, mysql local
What errors or information do you see in the console? None
I’m able to successfully put a custom robots.txt file in the base of my theme and have that file used, but I’m trying to run ghost headless, using the API from another site. When I make the site “private”, the robots.txt automatically turns into
User-agent: * Disallow: /
and my custom robots.txt file isn’t use. I need robots to access the /content/images folder, or things such as twitter cards and the like will not work. These agents will honor the robots.txt file and will not populate images.
How can I add a custom robots.txt file for when the site is in private mode? I can’t find anything in the docs or forums related to this. Thanks!
If your setup has ghost running behind nginx or any other reverse proxy, then you can configure it to serve /robots.txt as a static file from a custom location instead of asking ghost to serve it.
I have done a similar configuration using Caddy for my staging environment.
Hi Ghost team. Is there any solution to this problem? I added custom robots.txt to my theme but it is ignored when in private mode. Twitter bots doesn’t work with Disallow: / and it causes images to be blocked.
There are some notes about this in our JAMstack docs:
In most cases the best approach is to either use an external storage adapter, or download all images locally as a part of your front-end build process so the assets are served statically with the rest of the site.
An alternative workaround to using Private <ode is to make a custom/empty Ghost theme with a no-index meta tag and a clientside redirect using window.location in default.hbs. This will prevent search engines from indexing pages, and redirect any traffic to the static front end, while leaving image and API routes in tact.