5.75.1 upgrade fails on Ubuntu 20 with Node 18

This was just after a Node 16->18 upgrade that appeared successful.

It appears there maybe a problem the “re2” dep being corrupt. Some key logs are:

error /sites/pea-pod.org/versions/5.75.1/node_modules/re2: Command failed.
Exit code: 7
Command: install-from-cache --artifact build/Release/re2.node --host-var RE2_DOWNLOAD_MIRROR --skip-path-var RE2_DOWNLOAD_SKIP_PATH --skip-ver-var RE2_DOWNLOAD_SKIP_VER || npm run rebuild
Arguments: 
Directory: /sites/pea-pod.org/versions/5.75.1/node_modules/re2
Output:
Trying https://github.com/uhop/node-re2/releases/download/1.20.3/linux-x64-108.br ...
Writing to build/Release/re2.node ...
The verification has failed: building from sources ...

Maybe trying to bump the version of the re2 dep would help?

The install logs continue with a different error about New Relic “native-metrics” having a problem.

Building locally ...
/usr/lib/node_modules/npm/lib/cli.js:35
    throw err
    ^

TypeError: Class extends value undefined is not a constructor or null
    at Object.<anonymous> (/usr/lib/node_modules/npm/node_modules/fs-minipass/lib/index.js:136:4)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
    at require (node:internal/modules/cjs/helpers:121:18)
    at Object.<anonymous> (/usr/lib/node_modules/npm/node_modules/cacache/lib/content/read.js:4:13)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)

Node.js v18.17.1
node:internal/process/promises:288
            triggerUncaughtException(err, true /* fromPromise */);
            ^

[UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "7".] {
  code: 'ERR_UNHANDLED_REJECTION'
}

Node.js v18.17.1
/usr/lib/node_modules/npm/lib/cli.js:35
    throw err
    ^

TypeError: Class extends value undefined is not a constructor or null
    at Object.<anonymous> (/usr/lib/node_modules/npm/node_modules/fs-minipass/lib/index.js:136:4)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
    at require (node:internal/modules/cjs/helpers:121:18)
    at Object.<anonymous> (/usr/lib/node_modules/npm/node_modules/cacache/lib/content/read.js:4:13)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)

Node.js v18.17.1
warning Error running install script for optional dependency: "/sites/pea-pod.org/versions/5.75.1/node_modules/@newrelic/native-metrics: Command failed.
Exit code: 1
Command: node ./lib/pre-build.js install native_metrics
Arguments: 
Directory: /sites/pea-pod.org/versions/5.75.1/node_modules/@newrelic/native-metrics
Output:
============================================================================
Attempting install in native-metrics module. Please note that this is an
OPTIONAL dependency, and any resultant errors in this process will not
affect the general performance of the New Relic agent, but event loop and
garbage collection metrics will not be collected for the Node VMs page.
============================================================================

Download error: ENOENT: no such file or directory, open '/sites/pea-pod.org/versions/5.75.1/node_modules/@newrelic/native-metrics/build/Release/_newrelic_native_metrics-10_0_0-native_metrics-108-linux-x64.node', falling back to build
> /usr/bin/node /sites/pea-pod.org/versions/5.75.1/node_modules/node-gyp/bin/node-gyp.js clean configure
gyp info it worked if it ends with ok
gyp info using node-gyp@8.4.1
gyp info using node@18.17.1 | linux | x64
gyp info find Python using Python version 3.8.10 found at \"/usr/bin/python3\"
gyp info spawn /usr/bin/python3
gyp info spawn args [
gyp info spawn args   '/sites/pea-pod.org/versions/5.75.1/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/sites/pea-pod.org/versions/5.75.1/node_modules/@newrelic/native-metrics/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/sites/pea-pod.org/versions/5.75.1/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/home/mark/.cache/node-gyp/18.17.1/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/home/mark/.cache/node-gyp/18.17.1',
gyp info spawn args   '-Dnode_gyp_dir=/sites/pea-pod.org/versions/5.75.1/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=/home/mark/.cache/node-gyp/18.17.1/<(target_arch)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/sites/pea-pod.org/versions/5.75.1/node_modules/@newrelic/native-metrics',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.'
gyp info spawn args ]
gyp info ok 
> /usr/bin/node /sites/pea-pod.org/versions/5.75.1/node_modules/node-gyp/bin/node-gyp.js build -j 1 native_metrics
gyp info it worked if it ends with ok
gyp info using node-gyp@8.4.1
gyp info using node@18.17.1 | linux | x64
gyp info spawn make
gyp info spawn args [ 'native_metrics', 'BUILDTYPE=Release', '-C', 'build', '--jobs', 1 ]
make: Entering directory '/sites/pea-pod.org/versions/5.75.1/node_modules/@newrelic/native-metrics/build'
  CXX(target) Release/obj.target/native_metrics/src/native_metrics.o
make: g++: Command not found
make: *** [native_metrics.target.mk:121: Release/obj.target/native_metrics/src/native_metrics.o] Error 127
make: Leaving directory '/sites/pea-pod.org/versions/5.75.1/node_modules/@newrelic/native-metrics/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/sites/pea-pod.org/versions/5.75.1/node_modules/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (node:events:514:28)
gyp ERR! stack     at ChildProcess._handle.onexit (node:internal/child_process:291:12)
gyp ERR! System Linux 6.2.9-x86_64-linode160
gyp ERR! command \"/usr/bin/node\" \"/sites/pea-pod.org/versions/5.75.1/node_modules/node-gyp/bin/node-gyp.js\" \"build\" \"-j\" \"1\" \"native_metrics\"
gyp ERR! cwd /sites/pea-pod.org/versions/5.75.1/node_modules/@newrelic/native-metrics
gyp ERR! node -v v18.17.1
gyp ERR! node-gyp -v v8.4.1
gyp ERR! not ok 
Failed to execute native-metrics install: Command exited with non-zero code: 1

This is another recent issue about a New Relic native metrics problem, but on Windows 11:

It turns out I had multiple problems to resolve.

npm corruption

First, it seems my installation of npm was corrupt. I suspect it happened due to this sequence:

  1. Installed Node 16
  2. Ran sudo npm install -g npm at some point because npm keeps prompting me to upgrade it.
  3. Upgraded Node to version 18.

Node installs it’s own npm, but I think the old npm module directory was left over and conflicting. To resolve that:

sudo apt remove nodejs
sudo rm -rf /usr/lib/node_modules/npm
sudo apt install nodejs

After that npm was unbroken.

IPv6 problems

Next, the upgrade failed like this:

‘Ghost was able to start, but errored during boot with: connect ECONNREFUSED ::1:3306’,

Some details in that error: “::1” indicates an IPv6 connection to localhost and “:3306” is the standard port that MySQL listens on.

I believe due to an earlier upgrade to Node 18 or some related module, that connections were now being attempted over IPv6 first. So, although MySQL it was running, it wasn’t listening for IPv6 connnections, so this was failing.

That could be resolved by either forcing Ghost to connect over IPv4, or allowing MySQL to start listening on IPv6 as well. I chose the latter.

The MySQL directive for that is bind-address. In my case, the file to edit was:

/etc/mysql/mysql.conf.d/mysqld.cnf

There was already an entry for bind-address listening on port 127.0.0.1 there. I added a comma and “::1” to the list and restarted MySQL.

After that, my upgrade issues were resolved.