Importing posts to Ghost

Hello. After numerous attempts to import data from the old database, I constantly received errors due to lack of memory (I have about 100,000 entries).

https://ghost.org/faq/ghost-crashes-when-importing/ (I read this article, did not help at all )

In the end, I decided to just insert queries into the database of Ghost and stuck on columns id and uuid
9c765baa-1562-4f41-8ca2-e6c3a787de6c
Please tell me how to convert from simple id to ghost format id and uuid.
For example post id 4831.

We use https://www.npmjs.com/package/bson-objectid for the id’s in our database :relaxed:

After doing manual inserts, you’ll need to save every post (can be done via API) in order to get the content to be generated. I think you might be heading down a path of more pain than fixing the original issue.

I’d love to help you get to a resolution for the underlying the issue if you’re up for working with me.

Could you possibly share details of how much ram there is on the machine running the import and also as much detail about the errors as possible?

We can attempt to fix the out of memory issues, but also there’s a tool lying around somewhere that helps to break a Ghost JSON into multiple parts, which I’ll dig out and look at making publicly available.

16 RAM

Would you be willing to share the import file with me privately?

Yes sure

How soon do you share your answer?

A couple of important questions:

  • what version of Ghost is the export coming from?
  • what version are you trying to import into?
  • did you edit the file before getting the memory errors?
  • what version of Ghost is the export coming from?
    Ghost-CLI version: 1.11.0
    Ghost version: 2.26.0
  • what version are you trying to import into? 2.26.0
  • did you edit the file before getting the memory errors? Yes

Hmm - I feel like I’m missing something. If you exported from 2.26 and import into 2.26 there shouldn’t be any validation errors.

In your original message, you mentioned converting from simple IDs (which were pre Ghost 1.0) to Ghost’s IDs.

I feel like I’m missing part of the story? It sounds like you originally exported from Ghost 0.x?

  1. I updated today ghost and now it is 2.26.0.
  2. But that version was 2.25.7 in the original message (Importing posts to Ghost)
  3. I am importing data from custom CMS that’s why I asked about IDs to somehow import data.
    P.S. I’ve never worked on Ghost CMS before.
    AND I didn’t replied you correctly to the question
  • what version of Ghost is the export coming from?
    Sorry me :slight_smile:

100,000 entries ? this conventinal solution not will scale for you.

i am in a project like this in moment.

try migrate, in this order, (from less to more entities):

  • users (not have admin api, then you need insert in db)
  • tags (not have admin api, then you need insert in db)
  • posts (have admin api, but not bulk api)

A bulk api are much useful in this case, but are not implemented yet.

You are find some caveats in path:

  • slugs generations
  • ids generatios,
  • how convert html to mobiledoc (if yout legacy content not are compatible).
  • Careful with edit endpoint, because its not idemponent.

Ghostjs are not designed to scale with multiple instances neither multiple child process, this is another problem, because you have only one thread availible and not have all power of machine (if have multiple cores).

In have much less content (7000 posts), and migration job is extreme slow here.

In this moment i m thinkng in insert posts via db, and restart app after end, its needed because ghostjs caches urls and manage his content using a internal event system, you can try upload a routes.yml in admin to force ghost query db and regenerate your content.

I think that objectId in database was possibly a bad choice, becouse i cant see the benefits of it. i think a non auto-increment was a good choice, but couple it a objectId was bad to us case where we need export

In resume:

  • import by json not will scale to large contents
  • ghostjs not have a matured API to create a migration script
  • you need insert in db, and will enconter some caveats (test a sample of your dada first)
  • you need force ghost regenerate your content after