Import fails with DB rollback

Hi everybody,

how does one debug the import script?

I’ve managed to import a 15k articles JSON, but at 30k articles it rolls back the MySQL transaction:

2020-01-06T21:49:56.548477Z        22 Query     BEGIN
...
...
2020-01-06T21:59:51.694794Z	   22 Query	select `posts`.* from `posts` where `posts`.`slug` = '**SOME-SLUG**' limit 1
2020-01-06T21:59:51.695755Z	   22 Query	insert into `posts` (`author_id`, `canonical_url`, `codeinjection_foot`, `codeinjection_head`, `comment_id`, `created_at`, `created_by`, `custom_excerpt`, `custom_template`, `feature_image`, `featured`, `html`, `id`, `locale`, `mobiledoc`, `plaintext`, `published_at`, `published_by`, `send_email_when_published`, `slug`, `status`, `title`, `type`, `updated_at`, `updated_by`, `uuid`, `visibility`) values ('5e13ab84b8bb1100014f6711', NULL, NULL, NULL, 233231, '2014-01-05 10:56:21', '5e13ab84b8bb1100014f6711', NULL, NULL, NULL, false, NULL, '5e13abe4b8bb110001503698', NULL, '{\"version\":\"0.3.1\",\"markups\":[],\"atoms\":[],\"cards\":[],\"sections\":[[1,\"p\",[[0,[],0,\"\"]]]]}', NULL, '2014-01-05 09:55:38', '1', false, '**SOME-SLUG**', 'published', '**SOME TITLE**', 'post', '2014-01-05 10:56:21', '5e13ab84b8bb1100014f6711', '8cde864e-2c78-48f2-8fd5-db212a60134e', 'public')
2020-01-06T21:59:51.701596Z	   22 Query	select `users`.* from `users` where `users`.`id` = '5e13ab84b8bb1100014f6711' limit 1
2020-01-06T21:59:51.703491Z	   22 Query	insert into `posts_authors` (`author_id`, `id`, `post_id`) values ('5e13ab84b8bb1100014f6711', '5e13add7b8bb11000150aca0', '5e13abe4b8bb110001503698')
2020-01-06T21:59:51.708306Z	   22 Query	update `posts_authors` set `sort_order` = 0 where `author_id` = '5e13ab84b8bb1100014f6711' and `post_id` = '5e13abe4b8bb110001503698'
2020-01-06T21:59:51.711180Z	   22 Query	select `posts`.* from `posts` where `posts`.`id` = '5e13abe4b8bb110001503698' limit 1
2020-01-06T21:59:51.711913Z	   22 Query	select distinct `posts_meta`.* from `posts_meta` where `posts_meta`.`post_id` in ('5e13abe4b8bb110001503698')
2020-01-06T21:59:51.714669Z	   22 Query	select `settings`.* from `settings`
2020-01-06T22:01:45.115126Z	   22 Query	ROLLBACK

My docker-config.yml is:

version: '3.1'

services:

  ghost:
    image: ghost:3.2.0
    restart: unless-stopped
    depends_on: 
      - db

    ports:
      - 8080:2368
    environment:
      database__client: mysql
      database__connection__host: db
      database__connection__user: root
      database__connection__password: example
      database__connection__database: ghost
      NODE_OPTIONS: "--max_old_space_size=5000"

  db:
    image: mysql:8.0.18
    command: --default-authentication-plugin=mysql_native_password --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci --general-log-file=/var/log/mysql/general-log.log --innodb-buffer-pool-size=1000M
    restart: unless-stopped
    environment:
      MYSQL_ROOT_PASSWORD: example
    ports:
      - 3308:3306
    volumes:
      - "./volumes/mysql/data:/var/lib/mysql"
      - "./volumes/mysql/log:/var/log/mysql"
      # /etc/mysql/conf.d path included by /etc/mysql/my.cnf
      - "./volumes/mysql/conf:/etc/mysql/conf.d"

Memory shouldn’t be an issue (“docker stats” shows memory usage at 2GB, max is configured at 9GB).

Thanks,
A.

@aand11 which import script are you referring to? Are you getting any errors?

Hi Kevin,
I was referring to the Import function under the Labs page.

The error I’m seeing is the front-end timing out, but that is to be expected.

I was getting Node out-of-memory errors before, but after setting max_old_space_size they went away.

I’ve enabled the MySQL logs to tell why the transaction rolls back but it hasn’t helped.

How else could I debug this?
Thanks

Bump

bump