Request for maintainer feedback: towards stateless Ghost

Contributing to Ghost
1

TLDR: I have some proposed changes to Ghost that I am interested in implementing. I’d like to check if these changes make sense and are something that upstream would be interested in, before starting on them.

Background

Right now Ghost is a stateful application, persisting some of its data or state to the localdisk (such as themes). This makes running a stateless container of the application, where one only needs to pass in the relevant configuration values (connection strings, etc) not currently possible.

Some pieces of state are stored external to the application environment.

  • Database (and all of its data) can be external, with the connection information passed in as environment variables
  • Uploaded assets can be stored externally, using a storage adapter.

From looking over the application code, I can identify the following pieces of data that, however, do not have this extractability:

  • routes.yaml - stored in content/settings
  • redirects.json - stored in content/data
  • themes - stored in content/themes

I think for each of these types, it is feasible to change the application to not store this information on disk.

Proposal

I want to make upstream changes to Ghost that will allow it run in a containerized context without any need for persistent discs.

routes.yaml

Right now it is stored in content/settings. A simple approach is to store the JSON in the settings table. For the settings.value column, we only allow 64KiB, so we may need to bump that. Looking at some of the examples in the test files (site_spec) it looks like it would be difficult to normalize, but we could also do that as well.

In terms of compatibility, we could hook into app startup migration logic to run a one-time port of the routes.yaml into the database. I’m biased towards that, as putting this information into the database seems desirable even outside of my specific stateless concerns. Otherwise, we can have the database-storage functionality be a config switch.

redirects.json

This one is stored in content/data. We could move it to settings, as well. Redirects is similar to the navigation key we store in the settings database already, conceptually. A simple flat array. This one could also be normalized into its own redirects table, if that seems worthwhile.

Likewise, here, we can hook into the app startup migration logic to do a one-time conversion of the on-disk file into the database, if the file exists. Or have a config switch.

themes

When storage adapters were added, themes were not migrated to use that same storage engine, though there was some plans around doing so. It seems like we could still resolve the theme zip from the same “storage connector” that we fetch other things to, and extract it to a temporary directory on startup.

The migration here is tricky. To mirror our storage config flag, we could have a themeStorage config flag that would need to be set for this to work. The themeStorage would have the same API has the storage adapters, and would implement retrieval/saving of the zip.

Conclusion and misc contribution thoughts

In the contribution guidelines, it says that all non-issues need to be posted here, so I posted this here. I understand the need to keep feature-request issues from the backlog to avoid getting bogged down, but has there been consideration of allowing technical improvement/refactoring tickets/issues to be filed as issues? Especially if there is an interest or intent from the reporter to implement the change. Perhaps there could be an auto-expiration of such issues if no progress is made in X days. It feels strange to need to go outside my principal development platform to talk about developmental details.

1 Like
2

Hi,
I like your proposition very much. I’m not a maintainer but I do host a lot of ghost sites and I use ghost since v0.4

The idea of having all assets, configs and themes available via a storage adapter is compelling :slight_smile:. The web be a killer feature if we could go 100% stateless. Cheers!

3

Yep - I’m excited about the changes.

For those interested, I implemented two of the above three, roughly, and opened PRs for further review and discussion:

4

Hi @gnalck, thanks for taking the time to write this up.

I understand the underlying concerns you have, but the solution seems heavy handed.

Effectively, you’re removing features that Ghost users and customers rely on in order to support a very narrow hosting model. Furthermore, that hosting model isn’t the recommended model that we actively support for Ghost.

It doesn’t make a lot of sense to me to prioritise container ideology, which most people don’t understand, over features they are actively using and benefiting from.

If you wanted to propose extending the concept of storage adapters, which would expose the possibility to do what you want, without impacting user-facing features I’d be far more on board.

It’s also totally possible to do what you want with mounted file storage, without making any changes to Ghost at all.

3 Likes
5

Hey @Hannah ,

Thanks for taking the time to respond!

With regards to removing features users and customers rely on, I assumed (perhaps wrongly) that the primary interface for editing both of these things was through the Ghost admin interface, whose functionality remains unchanged. I believe for Ghost(PRO) customers this is the only interface for editing these things, as the users don’t have access to the underlying box. The docs for one of the two, redirects, mentioned that it the admin panel is the recommended method, which is partly what inspired the changes: Implementing redirects in Ghost

However, if editing the redirects and routes directly is a common use case, I agree that removing those interfaces should not be taken lightly and should be treated as a ‘breaking’ change. Is there any interest, then, in taking up similar changes in the next major (breaking) release?

Leveraging storage adapters is a great idea, and is the path I was looking into for the third point above around migrating themes. I’d love feedback on a couple questions before I start down that path though, if I also want to use that functionality for settings and redirects there too.

  • We will need new flag(s) to control where to store things other than images. How do you see those looking - should they continue to reflect the current local file structure? Something like storage:themes, storage:data (since redirects is currently stored there), storage:settings?
  • In order to continue to list all themes, we need a list API for StorageBase, and maybe others. How do you see the best path for making that change? Is adding some logic in the storage adapter loader in Ghost to throw if the storage adapter being loaded doesn’t implement all the needed APIs for the given type (e.g., themes) a reasonable approach?

Cheers,

6

Is there any chance that it would be implemented?

At least themes storage on remote storage?
Themes seems to be the largest static that is still impossible to store in bucket. External caching before the site helps, but it would be better to have it in object storage (have storage adapter support)

7

Storing theme files remotely doesn’t make any sense for the majority of users, and to be quite honest I don’t see how it would work at all.

The core team’s job is to implement the tools and features that the majority of users will benefit from.

Self hosters who want edge case setups to be supported should expect to have to implement it themselves.

As I’ve already said, if there was a proposal that extended the existing adapter system (which FYI has already been extended to support more storage cases since this was last discussed) that extended capabilities without removing any behaviour then that would be something we’d merge.

However, I really don’t see how loading theme templates from remote storage would ever be useful or usable and it seems to me mounted file systems are way more appropriate for themes.

1 Like