Minimal build

[guest271314]

To build and run Node.js as a minimal JavaScript runtime, with specific requirements that will not change: process, Buffer, Array, JSON, require('child_process')

protocol.js from https://github.com/simov/native-messaging/blob/master/protocol.js

module.exports = (handleMessage) => {

  process.stdin.on('readable', () => {
    var input = []
    var chunk
    while (chunk = process.stdin.read()) {
      input.push(chunk)
    }
    input = Buffer.concat(input)

    var msgLen = input.readUInt32LE(0)
    var dataLen = msgLen + 4

    if (input.length >= dataLen) {
      var content = input.slice(4, dataLen)
      var json = JSON.parse(content.toString())
      handleMessage(json)
    }
  })

  function sendMessage (msg) {
    var buffer = Buffer.from(JSON.stringify(msg))

    var header = Buffer.alloc(4)
    header.writeUInt32LE(buffer.length, 0)

    var data = Buffer.concat([header, buffer])
    process.stdout.write(data)
  }

  process.on('uncaughtException', (err) => {
    sendMessage({error: err.toString()})
  })

  return sendMessage

}

example.js

#!/home/user/node

// Might be good to use an explicit path to node on the shebang line
// in case it isn't in PATH when launched by Chrome

var sendMessage = require('./protocol')(handleMessage)

function handleMessage (req) {
  if (req.message === 'ping') {
    sendMessage({message: 'pong', body: `hello from nodejs ${process.version} app`})
  }

  if (req.message === 'write') {
    var {exec} = require("child_process");
    exec(req.body, (err, stdout, stderr) => {
      if (err) {
        // node couldn't execute the command
        sendMessage({message: "stderr", body: err});
        return;
      }
      sendMessage({message: "output", body: stdout});   
    });    
  }

}

I read nodejs/node#2948 and nodejs/docs#88.

This is what I have tried

$ ./configure --dest-cpu=x86_64 \
 --dest-os=linux \  
 --no-cross-compiling \
 --fully-static \ 
 --without-intl \ 
 --without-node-code-cache \
 --without-npm \
 --without-ssl \
 --without-node-snapshot \
 --without-dtrace \
 --without-etw \
 --without-inspector \
 --without-corepack

I did not locate an official, formal procedure to build minimal node.

Ideally, given existing files which contain the complete code required node executable can be build targeting only the dependencies in the existing .js files, adding nothing else, e.g., no npm, WebAssembly, SSL, etc.

Currently the nightly node executable is ~80MB. I suspect the executable can be built less expensively, without including source code that will not be used in the resulting node executable.

Kindly share the official procedure to build minimal node executable.

[mhdawson]

I think this is a better fit for the node.js repo so I'm going to move it there.

I'm also not aware of any such documentation so I'm going to tag it as a feature request.

[mhdawson]

@guest271314 I'm not sure about your use case but in terms of being able to consistently build with the minimum set possible I'm wondering if a new option that set's all of the optinos for things which are optional would make any sense. My thought is that a build option in the Makefile would be more likely to be kept up to date as options that are added versus an external doc saying to build with options x,y,z.

[guest271314]

My sole use case is using JavaScript as a Native Messaging host.

The node executable (nightly) is 82.2 MB.

Running node as a Native Messaging host with only the files at OP is 36 MB.

A C++ Native Messaging host compiled as a shared library with clang++ or g++ is 24.6 KiB.

Running a Native Messaging host is 1.5 MB.

The above data requires the question to be answered: Why fetch the entire node executable just to run the JavaScript code in protocol.js and example.js as a native executable?

Another question: How to identify only the source code files necessary to run the above (or any pre-written JavaScript), and only build the node executable for specific requirements?

@guest271314 I'm not sure about your use case but in terms of being able to consistently build with the minimum set possible I'm wondering if a new option that set's all of the optinos for things which are optional would make any sense. My thought is that a build option in the Makefile would be more likely to be kept up to date as options that are added versus an external doc saying to build with options x,y,z.

That is feasible, as long s we really can have an option to begin with the bare minimum from v8, uv, et al. required to run JavaScript as a native executable - distinct from the idea of removing options from a fully-featured download: we already have that.

Something along the lines of Debian minimal build.

That, as directly mentioned above, and for double-redundancy here, means identifying the least amount of files to run JavaScript as as native application on the users' machine. process, Array, etc. I don;t need SSL, WebAssembly, WASI, npm, npx, nor documentation, tests, history of commits. I just need to base code to run JavaScript in *nix as an executable.

[bnoordhuis]

The bulk of the node binary consists of V8 and ICU. You can disable the latter with the --without-intl flag but V8 is of course mandatory. Node's other components (libuv, openssl, etc.) are small in comparison.

If you want something more nimble and are okay with a runtime that's more limited in scope, take a look at https://github.com/saghul/txiki.js - it's based on quickjs instead of V8.

[guest271314]

but V8 is of course mandatory.

Is the entire V8 mandatory? If so, why?

Node.js is the flagship.

I am here asking the Node.js experts how to trim the 80 MB executable to the minimal code required.

I don't expect anything from any realm, yet a clear and concise answer to my inquiry is conspicuously lacking here. I am not sure why? Incapable of adaptation or scalability? Fixed in the current model?

[guest271314]

If you want something more nimble and are okay with a runtime that's more limited in scope, take a look at https://github.com/saghul/txiki.js - it's based on quickjs instead of V8.

It is never that easy.

$ sudo apt install curl

$ make 
...
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find CURL (missing: CURL_LIBRARY CURL_INCLUDE_DIR)
Call Stack (most recent call first):
  /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.16/Modules/FindCURL.cmake:143 (find_package_handle_standard_args)
  CMakeLists.txt:76 (find_package)


-- Configuring incomplete, errors occurred!

Will you help debug why make is not finding curl for that repository?

[bnoordhuis]

Is the entire V8 mandatory? If so, why?

Pretty much. V8 has build flags you can tweak but even in its most minimal configuration, it's still pretty bulky.

I don't expect anything from any realm, yet a clear and concise answer to my inquiry is conspicuously lacking here.

There's no ready-to-regurgitate recipe, if that's what you're hoping for, you're on your own here. Good luck and post your findings.

[guest271314]

Based on your reply, my findings are it is not possible. Stuck with 80MB node binary. Or, just don't use Node.js at all. At the linked issue some claimed >30MB executable.

I tried to build Node.js the other day. Hours without progress report and not completing, I had no reference point. This appear to be a severe oversight that can be corrected, given the will to do so, which I am not gathering here.

[bnoordhuis]

At the linked issue some claimed >30MB executable.

That was ~7 years ago. Node and V8 have acquired a lot of features since then.

This appear to be a severe oversight that can be corrected, given the will to do so

Your will, yes. That's pretty much what open source is about: scratch your own itch, don't expect others to do it for you (unless you pay them, perhaps, and even then.)

[guest271314]

Too big to scale.

[liuxingbaoyu]

Why don't you try to compress the executables, such as upx.
78mb->25mb

[guest271314]

Kindly explain.

[liuxingbaoyu]

https://github.com/upx/upx

[guest271314]

A. Is it worth it (just write code in C++, etc.)?
B. Test. ask. test, again. Determine if possible by testing.

1. Downloaded Node.js and extract node executable.
2. ~80 MB.
3. Ask if anyone else has even tried to build minimal node binary (while trying to build a minimal version).

I have no problem diving in to Node.js source code to achieve my requirement. I am just asking if Node.js experts, developers in the field have even tried to build a minimal node executable. There were some issues. No clear and consistent set of options and steps, in code.

I don't think it it is worth it. Just to run JavaScript as a native executable. I'll probably continue trying build a minimal Node.js, beginning at the beginning, the first commit. I ain't asking you folks to build node for me. I am just asking if you have even tried to build a minimal node. Thanks.

[guest271314]

78mb->25mb

Still uses the same amount of resources as the original node executable

$ ./upx --brute node

$ ps -eo rss,pid,euser,args:100 --sort %mem | grep -v grep | grep -i example.js | awk '{printf $1/1024 "MB"; $1=""; print }'
82.5781MB

[guest271314]

This is what I have so far using https://github.com/saghul/txiki.js.

On *nix a dependency is libcurl-dev, which prompt do download one of several packages with different names. I used the OpenSSL option.

I am not sure why array over 63 elements length does not get sent back to the Native Messaging client.

Memory used. Node.js is ~36MB. txiki.js is ~17MB.

#!/path/to/tjs
// txiki.js Native Messaging host
// guest271314, 5-5-2022
(async() => {
    async function getMessage() {
      const buffer = new Uint8Array(4);
      const input = await tjs.stdin.read(buffer);
      const [length] = new Uint32Array(buffer.buffer);
      const output = new Uint8Array(length);
      await tjs.stdin.read(output);      
      return output;
    }

    async function sendMessage(message) {
      const header = new Uint32Array(4);
      header.set([message.length], 0, true);
      const output = new Uint8Array(header.length + message.length);
      output.set(header, 0, true);
      output.set(message, 4, true);
      await tjs.stdout.write(output);
      return true;
    }

    while (true) { 
      const message = await getMessage();
      await sendMessage(message);
    }
})();

[guest271314]

I adjusted the code which reads and writes message length to fix not being able to get a message with an array having over 63 elements reading this answer https://stackoverflow.com/a/26140545:

As someone mentioned in the comments, the provided solution won't work if the length is greater than 255.

test.js

#!tjs
// txiki.js Native Messaging host
// guest271314, 5-6-2022
(async () => {

  async function getMessage() {
    const header = new Uint8Array(4);
    const input = await tjs.stdin.read(header);
    // https://stackoverflow.com/a/26140545
    const length = new Uint32Array(header.buffer)
      .reduce((a, b, index) => a = a | (b << index * 8), 0);
    const output = new Uint8Array(length);
    await tjs.stdin.read(output);
    return output;
  }

  async function sendMessage(message) {
    // https://stackoverflow.com/a/24777120
    const header = Uint32Array.from({
        length: 4
      }, (_, index) =>
      message.length >> (index * 8) & 0xFF
    );
    const output = new Uint8Array(header.length + message.length);
    output.set(header, 0, true);
    output.set(message, 4, true);
    await tjs.stdout.write(output);
  }

  while (true) {
    const message = await getMessage();
    await sendMessage(message);
  }
})();

The top entry is txiki.js, bottom Node.js Native Messaging host:

[guest271314]

I utilized QuickJS (937504 bytes executable) as a Native Messaging host with 2.8MB (which is less than Bash at 3.6MB; Python at 10MB; PHP at 15MB; C++ at 1.5MB).

Node.js is most expensive as a Native Messaging host at 36MB (82MB executable).

Given the movement in/of this feature request I don't think a minimal Node.js can be built (if at all) which requires less computing resources than txiki.js and QuickJS as JavaScript runtimes; nor the above listed languages.

#!/usr/bin/env -S /path/to/qjs -m --std
// QuickJS Native Messaging host
// guest271314, 5-6-2022
import * as std from 'std';
import * as os from 'os';

function getMessage() {
  const header = new Uint8Array(4);
  std.in.read(header.buffer, 0, header.length);
  // https://stackoverflow.com/a/26140545
  const length = new Uint32Array(header.buffer).reduce(
    (a, b, index) => a | (b << (index * 8)),
    0
  );
  const output = new Uint8Array(length);
  std.in.read(output.buffer, 0, length);
  return output;
}

function sendMessage(message) {
  // https://stackoverflow.com/a/24777120
  const header = Uint32Array.from(
    {
      length: 4,
    },
    (_, index) => (message.length >> (index * 8)) & 0xff
  );
  const output = new Uint8Array(header.length + message.length);
  output.set(header, 0, true);
  output.set(message, 4, true);
  std.out.write(output.buffer, 0, output.length);
  std.out.flush();
}

while (true) {
  const message = getMessage();
  sendMessage(message);
}

[guest271314]

Unfortunately QuickJS popen() with fread() is too slow to process real-time audio capture cf. Python and C++ versions. saghul/txiki.js#294.

[guest271314]

Unfortunately QuickJS popen() with fread() is too slow to process real-time audio capture cf. Python and C++ versions. saghul/txiki.js#294.

I am now able to stream using QuickJS https://github.com/guest271314/captureSystemAudio/blob/master/native_messaging/capture_system_audio/capture_system_audio.js with an imported .so module https://github.com/guest271314/captureSystemAudio/blob/master/native_messaging/capture_system_audio/popenAndRead.c.

Updated QuickJS Native Messaging host https://github.com/guest271314/native-messaaging-quickjs/blob/main/nm_qjs.js.