Update from 4.9.3 to 4.9.4 went wrong

[Lucas_Ridder]

Hi, i wonder if someone can help me with an error that popped up after updating from version 4.9.3 to 4.9.4.
Maria-db version: 10.5.11

Before doing this update i first updated my nodejs version to 14, updated npm itself and updated the ghost-cli npm module. The update itself was succesful without any errors till starting the ghost instance.
Before i was using nodejs version 10 and i had no problems at all.
Under here i’ve tried to collect alle the logs i got and replaced my domain with kb.domain.nl, i hope someone is able to tell me where the problem is and how i can fix this, thanks for reading this and taking a look.

Greetings,
Lucas Ridder

ghost start log:

kennisbank@44:/var/www/ghost$ ghost start

• sudo systemctl is-active ghost_kb-domain-nl
  Checking system Node.js version - found v14.17.2
  Ensuring user is not logged in as ghost user
  Checking if logged in user is directory owner
  Checking current folder permissions
• sudo systemctl is-active ghost_kb-domain-nl
  Validating config
  Checking folder permissions
  Checking file permissions
  Checking content folder ownership
  Checking memory availability
  Checking binary dependencies
  Checking systemd unit file
  Checking systemd node version - found v14.17.2
• sudo systemctl start ghost_kb-domain-nl
• sudo systemctl stop ghost_kb-domain-nl
  Starting Ghost: kb-domain-nl
  A GhostError occurred.

Message: Ghost was able to start, but errored during boot with: Cannot remove permission(Reset all passwords) from role(Administrator) - permission does not exist
Suggestion: journalctl -u ghost_kb-domain-nl -n 50

Debug Information:
OS: Ubuntu, v18.04.5 LTS
Node Version: v14.17.2
Ghost Version: 4.9.4
Ghost-CLI Version: 1.17.3
Environment: production
Command: ‘ghost start’

debug log file:

Debug Information:
OS: Ubuntu, v18.04.5 LTS
Node Version: v14.17.2
Ghost Version: 4.9.4
Ghost-CLI Version: 1.17.3
Environment: production
Command: ‘ghost start’
Message: Ghost was able to start, but errored during boot with: Cannot remove permission(Reset all passwords) from role(Administrator) - permission does not exist
Suggestion: journalctl -u ghost_kb-ghost-nl -n 50
Stack: Error: Ghost was able to start, but errored during boot with: Cannot remove permission(Reset all passwords) from role(Administrator) - permission does not exist
at Server. (/usr/lib/node_modules/ghost-cli/lib/utils/port-polling.js:73:28)
at Object.onceWrapper (events.js:481:28)
at Server.emit (events.js:375:28)
at emitCloseNT (net.js:1661:8)
at processTicksAndRejections (internal/process/task_queues.js:81:21)

journalctl logs:

Jul 17 01:30:28 P44 systemd[1]: Started Ghost systemd service for blog: kb-domain-nl.
Jul 17 01:30:29 P44 node[982]: - Inspecting operating system
Jul 17 01:30:29 P44 node[982]: [2021-07-16 23:30:29] INFO Ghost is running in production…
Jul 17 01:30:29 P44 node[982]: [2021-07-16 23:30:29] INFO Your site is now available on https://kb.domain.nl/
Jul 17 01:30:29 P44 node[982]: [2021-07-16 23:30:29] INFO Ctrl+C to shut down
Jul 17 01:30:29 P44 node[982]: [2021-07-16 23:30:29] INFO Ghost server started in 0.204s
Jul 17 01:30:29 P44 node[982]: [2021-07-16 23:30:29] INFO Bootstrap client was closed.
Jul 17 01:30:29 P44 node[982]: [2021-07-16 23:30:29] WARN Database state requires migration.
Jul 17 01:30:30 P44 node[982]: [2021-07-16 23:30:30] INFO Creating database backup
Jul 17 01:30:30 P44 node[982]: [2021-07-16 23:30:30] INFO Database backup written to: /var/www/ghost/content/data/redacted.ghost.2021-07-16-23-30-3
Jul 17 01:30:30 P44 node[982]: [2021-07-16 23:30:30] INFO Adding posts_meta.feature_image_caption column
Jul 17 01:30:30 P44 node[982]: [2021-07-16 23:30:30] INFO Updating all member “comped” statuses to “paid”
Jul 17 01:30:30 P44 node[982]: [2021-07-16 23:30:30] WARN Removing email_batches.member_segment column - skipping as table is correct
Jul 17 01:30:30 P44 node[982]: [2021-07-16 23:30:30] WARN Skipping dropping table: products_benefits - table does not exist
Jul 17 01:30:30 P44 node[982]: [2021-07-16 23:30:30] WARN Skipping dropping table: benefits - table does not exist
Jul 17 01:30:30 P44 node[982]: [2021-07-16 23:30:30] ERROR Cannot remove permission(Reset all passwords) from role(Administrator) - permission does not exist
Jul 17 01:30:30 P44 node[982]:
Jul 17 01:30:30 P44 node[982]: Cannot remove permission(Reset all passwords) from role(Administrator) - permission does not exist
Jul 17 01:30:30 P44 node[982]: "OuterError: alter table posts_meta add feature_image_caption text null - ER_TOO_BIG_ROWSIZE: Row size too large. The maximum row size for the used table type, not
Jul 17 01:30:30 P44 node[982]: Error ID:
Jul 17 01:30:30 P44 node[982]: 400
Jul 17 01:30:30 P44 node[982]: ----------------------------------------
Jul 17 01:30:30 P44 node[982]: InternalServerError: Cannot remove permission(Reset all passwords) from role(Administrator) - permission does not exist
Jul 17 01:30:30 P44 node[982]: at DatabaseStateManager.makeReady (/var/www/ghost/versions/4.9.4/core/server/data/db/state-manager.js:98:32)
Jul 17 01:30:30 P44 node[982]: at async initDatabase (/var/www/ghost/versions/4.9.4/core/boot.js:57:5)
Jul 17 01:30:30 P44 node[982]: at async bootGhost (/var/www/ghost/versions/4.9.4/core/boot.js:320:9)
Jul 17 01:30:30 P44 node[982]: RollbackError: Cannot remove permission(Reset all passwords) from role(Administrator) - permission does not exist
Jul 17 01:30:30 P44 node[982]: at RollbackError.KnexMigrateError (/var/www/ghost/versions/4.9.4/node_modules/knex-migrator/lib/errors.js:7:26)
Jul 17 01:30:30 P44 node[982]: at new RollbackError (/var/www/ghost/versions/4.9.4/node_modules/knex-migrator/lib/errors.js:31:26)
Jul 17 01:30:30 P44 node[982]: at /var/www/ghost/versions/4.9.4/node_modules/knex-migrator/lib/index.js:440:23
Jul 17 01:30:30 P44 node[982]: at tryCatcher (/var/www/ghost/versions/4.9.4/node_modules/bluebird/js/release/util.js:16:23)
Jul 17 01:30:30 P44 node[982]: at Promise._settlePromiseFromHandler (/var/www/ghost/versions/4.9.4/node_modules/bluebird/js/release/promise.js:547:31)
Jul 17 01:30:30 P44 node[982]: at Promise._settlePromise (/var/www/ghost/versions/4.9.4/node_modules/bluebird/js/release/promise.js:604:18)
Jul 17 01:30:30 P44 node[982]: at Promise._settlePromise0 (/var/www/ghost/versions/4.9.4/node_modules/bluebird/js/release/promise.js:649:10)
Jul 17 01:30:30 P44 node[982]: at Promise._settlePromises (/var/www/ghost/versions/4.9.4/node_modules/bluebird/js/release/promise.js:725:18)
Jul 17 01:30:30 P44 node[982]: at _drainQueueStep (/var/www/ghost/versions/4.9.4/node_modules/bluebird/js/release/async.js:93:12)
Jul 17 01:30:30 P44 node[982]: at _drainQueue (/var/www/ghost/versions/4.9.4/node_modules/bluebird/js/release/async.js:86:9)
Jul 17 01:30:30 P44 node[982]: at Async._drainQueues (/var/www/ghost/versions/4.9.4/node_modules/bluebird/js/release/async.js:102:5)
Jul 17 01:30:30 P44 node[982]: at Immediate.Async.drainQueues [as _onImmediate] (/var/www/ghost/versions/4.9.4/node_modules/bluebird/js/release/async.js:15:14)
Jul 17 01:30:30 P44 node[982]: at processImmediate (internal/timers.js:464:21)
Jul 17 01:30:30 P44 node[982]: Error: Cannot remove permission(Reset all passwords) from role(Administrator) - permission does not exist
Jul 17 01:30:30 P44 node[982]: at down (/var/www/ghost/versions/4.9.4/core/server/data/migrations/utils.js:174:23)
Jul 17 01:30:30 P44 node[982]: at processTicksAndRejections (internal/process/task_queues.js:95:5)
Jul 17 01:30:30 P44 node[982]: at async Object.down (/var/www/ghost/versions/4.9.4/core/server/data/migrations/utils.js:281:13)
Jul 17 01:30:30 P44 node[982]: at async Object.down (/var/www/ghost/versions/4.9.4/core/server/data/migrations/utils.js:305:17)
Jul 17 01:30:30 P44 node[982]:
Jul 17 01:30:30 P44 node[982]: [2021-07-16 23:30:30] WARN Ghost is shutting down
Jul 17 01:30:30 P44 node[982]: [2021-07-16 23:30:30] WARN Ghost has shut down
Jul 17 01:30:30 P44 node[982]: [2021-07-16 23:30:30] WARN Your site is now offline
Jul 17 01:30:30 P44 node[982]: [2021-07-16 23:30:30] WARN Ghost was running for a few seconds
Jul 17 01:30:30 P44 node[982]: [2021-07-16 23:30:30] INFO Bootstrap client was closed.
Jul 17 01:30:30 P44 systemd[1]: Stopping Ghost systemd service for blog: kb-domain-nl…
Jul 17 01:30:30 P44 node[982]: [2021-07-16 23:30:30] WARN Ghost is shutting down
Jul 17 01:30:30 P44 node[982]: [2021-07-16 23:30:30] WARN Ghost has shut down
Jul 17 01:30:30 P44 node[982]: [2021-07-16 23:30:30] WARN Your site is now offline
Jul 17 01:30:30 P44 node[982]: [2021-07-16 23:30:30] WARN Ghost was running for a few seconds
Jul 17 01:30:30 P44 node[982]: [2021-07-16 23:30:30] WARN Ghost is shutting down
Jul 17 01:30:30 P44 node[982]: [2021-07-16 23:30:30] WARN Ghost has shut down
Jul 17 01:30:30 P44 node[982]: [2021-07-16 23:30:30] WARN Your site is now offline
Jul 17 01:30:30 P44 node[982]: [2021-07-16 23:30:30] WARN Ghost was running for a few seconds
Jul 17 01:30:31 P44 node[982]: /usr/lib/node_modules/ghost-cli/lib/process-manager.js:46
Jul 17 01:30:31 P44 node[982]: throw error;
Jul 17 01:30:31 P44 node[982]: ^
Jul 17 01:30:31 P44 node[982]: {
Jul 17 01:30:31 P44 node[982]: message: ‘Ghost was able to start, but errored during boot with: Cannot remove permission(Reset all passwords) from role(Administrator) - permission does not exist’
Jul 17 01:30:31 P44 node[982]: }

[Hannah]

@Lucas_Ridder

Jul 17 01:30:30 P44 node[982]: "OuterError: alter table posts_meta add feature_image_caption text null - ER_TOO_BIG_ROWSIZE: Row size too large. The maximum row size for the used table type, not

Just pulling out the actual error from the stack. This needs deeper investigation (being managed internally).

[SoftwareHints]

Is there any update on the error.

During upgrade from ghost 3 to ghost 4 I’m facing the same error:

Steps:

(1) Force update

ghost update v4 --force

(2) Restart

ghost is not starting.
So I do:

ghost restart

error:

✖ Starting Ghost
A GhostError occurred.

Message: Ghost was able to start, but errored during boot with: alter table `oauth` add constraint `oauth_user_id_foreign` foreign key (`user_id`) references `users` (`id`) - UNKNOWN_CODE_PLEASE_REPORT: Referencing column 'user_id' and referenced column 'id' in foreign key constraint 'oauth_user_id_foreign' are incompatible.
Help: Error occurred while executing the following migration: 01-add-oauth-user-data.js
Suggestion: journalctl -u ghost_localhost -n 50

Debug Information:
    OS: Ubuntu, v20.04.3 LTS
    Node Version: v14.17.6
    Ghost Version: 4.16.0
    Ghost-CLI Version: 1.17.3
    Environment: production
    Command: 'ghost restart'

(3) Update Collation (despite being ok)

Then I update the table collation by:

Ghost 4.1.0 errored during boot

Hmm… it’s possibly not that. All tables in my ghost_production database have the same collation (utf8mb4_general_ci).

again restart results into

(4) Error - Cannot remove permission(Reset all passwords) with role(Administrator) - permission does not exist

Ghost instance is not running! Starting...
+ sudo systemctl start ghost_localhost
+ sudo systemctl stop ghost_localhost
✖ Starting Ghost
A GhostError occurred.

Message: Ghost was able to start, but errored during boot with: Cannot remove permission(Reset all passwords) with role(Administrator) - permission does not exist
Suggestion: journalctl -u ghost_localhost -n 50

Debug Information:
    OS: Ubuntu, v20.04.3 LTS
    Node Version: v14.17.6
    Ghost Version: 4.16.0
    Ghost-CLI Version: 1.17.3
    Environment: production
    Command: 'ghost restart'

P.S. Should I open new post?

[SoftwareHints]

I was able to solve the problem described above by generating Alter collation script for each table:

SET sql_mode='PIPES_AS_CONCAT';

SELECT DIstinct
'ALTER TABLE ' || TABLE_SCHEMA || '.' || TABLE_NAME || '  CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;'

    FROM INFORMATION_SCHEMA.COLUMNS
    where TABLE_SCHEMA='my_ghost_database'

and running the code with silent foreign key flag:

SET FOREIGN_KEY_CHECKS=0;


ALTER TABLE my_ghost_database.actions  CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;
ALTER TABLE my_ghost_database.api_keys  CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;
ALTER TABLE my_ghost_database.brute  CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;

Both errors described disappeared and ghost is running fine.

More detailed explation is posted here: Fix: UNKNOWN_CODE_PLEASE_REPORT MySQL Errors