Skip to content

Latest commit

 

History

History
137 lines (94 loc) · 5.76 KB

DEVELOP.md

File metadata and controls

137 lines (94 loc) · 5.76 KB

How to develop

Getting Involved

Even if you do not plan to contribute to Apache Arrow itself or Arrow integrations in other projects, we'd be happy to have you involved:

We prefer to receive contributions in the form of GitHub pull requests. Please send pull requests against the github.com/apache/arrow-js repository.

If you are looking for some ideas on what to contribute, check out the GitHub issues for the Apache Arrow project. Comment on the issue and/or contact dev@arrow.apache.org with your questions and ideas.

If you’d like to report a bug but don’t have time to fix it, you can still post it on GitHub issues, or email the mailing list dev@arrow.apache.org

The package.json scripts

We use npm to install dependencies and run scripts.

  • npm run clean - cleans targets
  • npm run build - cleans and compiles all targets
  • npm test - executes tests against built targets

These scripts accept argument lists of targets × modules:

  • Available targets are es5, es2015, esnext, ts, and all (default: all)
  • Available modules are cjs, esm, umd, and all (default: all)

Examples:

  • npm run build -- builds all ES targets in all module formats
  • npm run build -- -t es5 -m all -- builds the ES5 target in all module formats
  • npm run build -- -t all -m cjs -- builds all ES targets in the CommonJS module format
  • npm run build -- -t es5 -t es2015 -m all -- builds the ES5 and ES2015 targets in all module formats
  • npm run build -- -t es5 -m cjs -m esm -- builds the ES5 target in CommonJS and ESModules module formats

This argument configuration also applies to clean and test scripts.

To run tests on the bundles, you need to build them first. To run tests directly on the sources without bundling, use the src target (e.g. npm run test -- -t src).

  • npm run doc

Compiles the documentation with Typedoc. Use npm run doc -- --watch to automatically rebuild when the docs change.

Running the Performance Benchmarks

You can run the benchmarks with npm run perf. To print the results to stderr as JSON, add the --json flag (e.g. npm run perf -- --json 2> perf.json).

You can change the target you want to test by changing the imports in perf/index.ts. Note that you need to compile the bundles with npm run build before you can import them.

Testing Bundling

The bundles use apache-arrow so make sure to build it with npm run build -- -t apache-arrow. To bundle with a variety of bundlers, run npm run test:bundle or npx gulp bundle.

Run npx gulp bundle:webpack:analyze to open Webpack Bundle Analyzer.

Updating the Arrow format flatbuffers generated code

  1. Once generated, the flatbuffers format code needs to be adjusted for our build scripts (assumes gnu-sed):

    cd $ARROW_HOME

# Create a tmpdir to store modified flatbuffers schemas
tmp_format_dir=$(mktemp -d)
cp ./format/*.fbs $tmp_format_dir

# Remove namespaces from the flatbuffers schemas
sed -i '+s+namespace org.apache.arrow.flatbuf;++ig' $tmp_format_dir/*.fbs
sed -i '+s+org.apache.arrow.flatbuf.++ig' $tmp_format_dir/*.fbs

# Generate TS source from the modified Arrow flatbuffers schemas
flatc --ts -o ./js/src/fb $tmp_format_dir/{File,Schema,Message,Tensor,SparseTensor}.fbs

# Remove the tmpdir
rm -rf $tmp_format_dir

  2. Manually fix the unused imports and add // @ts-ignore for other errors

  3. Add .js to the imports. In VSCode, you can search for ^(import [^';]* from '(\./|(\.\./)+)[^';.]*)'; and replace with $1.js';.

  4. Execute npm run lint from the js directory to fix the linting errors

How to preview documentation on your fork repository

Our GitHub Actions workflows will create the gh-pages branch on your fork repository automatically. It's for previewing documentation with your changes in a branch.

You need to enable GitHub Pages on your fork repository at https://github.com/${YOUR_GITHUB_ID}/arrow-js/settings/pages manually. Choose the gh-pages branch in the "Build and deploy" -> "Branch" configuration item and press the "Save" button. We can preview documentation by GitHub Pages for your fork repository: https://${YOUR_GITHUB_ID}.github.io/arrow-js/

Example: https://kou.github.io/arrow-js/

The gh-pages branch keeps previews of all branches. We recommend that you delete needless branches and merged branches from your fork.