Failing to build better-sqlite3 for nodejs-mobile on iOS

[Jermolene]

I am writing an iOS app that uses nodejs-mobile to run TiddlyWiki. It uses the full Node.js configuration of TiddlyWiki, not the single file configuration.

In the main branch I have it working. Now I want to be able to use the npm package better-sqlite3 which includes native code.

I have a script that includes running prebuild-for-nodejs-mobile within the node_modules/better-sqlite3 directory:

npx prebuild-for-nodejs-mobile ios-arm64

The prebuild works, and I can see the better_sqlite3.node directory in the prebuilds/ios-arm64/ folder.

Now comes the tricky bit. I am trying to use node-gyp-build to modify the package so that npm rebuild will pick the prebuilds for ios-arm64, and prepare a node_modules folder that works correctly with nodejs-mobile.

After running the script below in the terminal, and then running the app itself in Xcode on an Apple Silicon Mac and an iPad Pro, I get the following error:

[Whitespace inserted for readability]

Error: dlopen(/private/var/containers/Bundle/Application/FB49F9C9-3055-48AF-B69F-2217051F055E/TiddlyWiki.app/nodejs-project/node_modules/better-sqlite3/build/Release/better_sqlite3.node, 0x0001):
 tried: '/usr/lib/system/introspection/better_sqlite3.node' (no such file, not in dyld cache),
'/private/var/containers/Bundle/Application/FB49F9C9-3055-48AF-B69F-2217051F055E/TiddlyWiki.app/nodejs-project/node_modules/better-sqlite3/build/Release/better_sqlite3.node'
(mach-o file (/private/var/containers/Bundle/Application/FB49F9C9-3055-48AF-B69F-2217051F055E/TiddlyWiki.app/nodejs-project/node_modules/better-sqlite3/build/Release/better_sqlite3.node),
but incompatible platform (have 'macOS', need 'iOS')), '/private/preboot/Cryptexes/OS/private/var/containers/Bundle/Application/FB49F9C9-3055-48AF-B69F-2217051F055E/TiddlyWiki.app/nodejs-project/node_modules/better-sqlite3/build/Release/better_sqlite3.node' (no such file),
'/private/var/containers/Bundle/Application/FB49F9C9-3055-48AF-B69F-2217051F055E/TiddlyWiki.app/nodejs-project/node_modules/better-sqlite3/build/Release/better_sqlite3.node' (mach-o file (/private/var/containers/Bundle/Application/FB49F9C9-3055-48AF-B69F-2217051F055E/TiddlyWiki.app/nodejs-project/node_modules/better-sqlite3/build/Release/better_sqlite3.node), but incompatible platform (have 'macOS', need 'iOS'))

I think the crucial part is incompatible platform (have 'macOS', need 'iOS'), which suggests that npm rebuild is ignoring the prebuilds and compiling for the native platform.

I would appreciate any pointers that anyone can give me.

Here's the full code of my script

#!/bin/bash

# This script downloads and prepares TiddlyWiki 5 for inclusion in the app by compiling native modules

# 1. Download the TiddlyWiki5 branch "multi-wiki-support" into the folder ./nodejs-project
# 2. Build better-sqlite3 for ios-arm64 in ./TiddlyWiki/nodejs-project/node_modules/better-sqlite3

# Exit immediately if a command exits with a non-zero status
set -e

# Remove existing nodejs-project
rm -rf ./TiddlyWiki/nodejs-project

# Download TW5
REPO="Jermolene/TiddlyWiki5"
BRANCH="multi-wiki-support"
ZIP_URL="https://github.com/$REPO/archive/refs/heads/$BRANCH.zip"
TEMP_DIR=$(mktemp -d)
TEMP_ZIP="$TEMP_DIR/$BRANCH.zip"
curl -L $ZIP_URL -o $TEMP_ZIP
unzip -q $TEMP_ZIP -d ./TiddlyWiki
mv ./TiddlyWiki/TiddlyWiki5-multi-wiki-support ./TiddlyWiki/nodejs-project
rm -rf $TEMP_DIR
echo "TW5 downloaded"

# Remove existing node_modules
rm -rf ./TiddlyWiki/nodejs-project/node_modules

# Navigate to project directory
pushd "./TiddlyWiki/nodejs-project"

# Install the necessary packages
npm install better-sqlite3
npm install node-gyp-build

# Navigate to the better-sqlite3 directory
pushd "node_modules/better-sqlite3"

# Run the prebuild command
npx prebuild-for-nodejs-mobile ios-arm64

# Update package.json

node -e "
const fs = require('fs');
const path = 'package.json';

// Read the package.json file
fs.readFile(path, 'utf8', (err, data) => {
  if (err) {
    console.error('Error reading package.json:', err);
    return;
  }

  // Parse the existing package.json content
  const packageJson = JSON.parse(data);

  packageJson.name = 'better-sqlite3';
  packageJson.scripts.install = 'node-gyp-build';

  // Write the updated package.json back to file
  fs.writeFile(path, JSON.stringify(packageJson, null, 2), (err) => {
    if (err) {
      console.error('Error writing package.json:', err);
      return;
    }
    console.log('package.json updated successfully.');
  });
});
"

# Return to the project directory
popd

# Rebuild
npm rebuild

[Jermolene]

I've updated the OP with some more details, please let me if there's anything else I can add.

[staltz]

Thanks for the details, this is very informative. I also appreciate TiddlyWiki as a project.

I would also assume that npm rebuild is compiling for the host platform (macOS), because this is usually done at ./build/Release/.

There are helper libraries such as node-gyp-build and node-pre-gyp which in runtime basically take care of locating the "preferred" .node file, and then defer to require() (of that selected .node file) or process.dlopen().

What you can do in the meantime is to call directly process.dlopen() on a hard-coded path (relative paths should be fine) of the prebuilt .node, and then check if that works in runtime on iOS. If it does, then the issue is not with prebuild-for-nodejs-mobile neither nodejs-mobile, but with the helper library (node-gyp-build and others) which locates the .node file in runtime.

There are other helper libraries, e.g. https://github.com/nodejs-mobile/node-gyp-build-mobile/