[SPARK-56438][SQL][CORE] Optimize `VectorizedPlainValuesReader.readBinary` for direct ByteBuffer by eliminating intermediate byte[] copy by LuciferYang · Pull Request #55296 · apache/spark

LuciferYang · 2026-04-10T16:04:43Z

What changes were proposed in this pull request?

This PR optimizes VectorizedPlainValuesReader.readBinary for the direct (non-heap) ByteBuffer path by eliminating an intermediate byte[] copy.

Previously, when reading binary/string values from a direct ByteBuffer, each value required:

Allocating a new byte[len]
Copying from the ByteBuffer into the temp array
Copying from the temp array into the column vector's backing storage

This PR adds ByteBuffer-aware putBytes/putByteArray overloads to WritableColumnVector, enabling a single-copy path:

OnHeapColumnVector: uses ByteBuffer.get(index, byte[], offset, length) (absolute bulk get) to copy directly into the backing byte[] — one native-to-heap copy.
OffHeapColumnVector: uses Platform.copyMemory with the direct buffer's native address — one native-to-native copy.

A Platform.getDirectBufferAddress(ByteBuffer) helper is added to read a DirectByteBuffer's native address via the Buffer.address field offset, consistent with Platform's existing Unsafe-based accessor pattern.

Why are the changes needed?

In Spark's vectorized Parquet reader, binary/string columns from memory-mapped (direct) ByteBuffer sources incur two full memcpy operations per value. The intermediate byte[] allocation also adds GC pressure.

Eliminating one copy per value yields a 10–22% improvement on the default DIRECT/ON_HEAP path across JDK 17, 21, and 25.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass Github Actions
A micro-benchmark that simulates changes with JMH to verify the performance improvement.

Benchmark Code (click to expand)

package org.apache.spark.sql.execution.datasources.parquet;

import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.util.Random;
import java.util.concurrent.TimeUnit;

import org.apache.parquet.bytes.ByteBufferInputStream;
import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.options.CommandLineOptions;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
import org.apache.spark.sql.types.DataTypes;

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Warmup(iterations = 20, time = 1)
@Measurement(iterations = 50, time = 1)
@Fork(1)
@State(Scope.Benchmark)
public class VectorizedPlainValuesReaderBenchmark {

  @Param({"1024", "4096", "8192"})
  private int totalValues;

  @Param({"16", "64", "256"})
  private int valueLength;

  @Param({"HEAP", "DIRECT"})
  private String bufferType;

  @Param({"ON_HEAP", "OFF_HEAP"})
  private String columnVectorType;

  /** Plain-encoded binary payload: repeated [int length][bytes]. */
  private byte[] rawData;

  private VectorizedPlainValuesReader newReader;
  private OldVectorizedPlainValuesReader oldReader;
  private WritableColumnVector columnVector;

  @Setup(Level.Trial)
  public void setupTrial() {
    Random rng = new Random(42);
    rawData = new byte[totalValues * (4 + valueLength)];
    ByteBuffer tmp = ByteBuffer.wrap(rawData).order(ByteOrder.LITTLE_ENDIAN);
    byte[] payload = new byte[valueLength];
    for (int i = 0; i < totalValues; i++) {
      tmp.putInt(valueLength);
      rng.nextBytes(payload);
      tmp.put(payload);
    }

    newReader = new VectorizedPlainValuesReader();
    oldReader = new OldVectorizedPlainValuesReader();
  }

  @Setup(Level.Invocation)
  public void setupInvocation() throws IOException {
    // Recreate the column vector and reinitialize readers before each invocation
    // so that readBinary always starts from the beginning of the buffer.
    if (columnVector != null) {
      columnVector.close();
    }

    if ("ON_HEAP".equals(columnVectorType)) {
      columnVector = new OnHeapColumnVector(totalValues, DataTypes.BinaryType);
    } else {
      columnVector = new OffHeapColumnVector(totalValues, DataTypes.BinaryType);
    }

    newReader.initFromPage(totalValues, ByteBufferInputStream.wrap(makeBuffer()));
    oldReader.initFromPage(totalValues, ByteBufferInputStream.wrap(makeBuffer()));
  }

  private ByteBuffer makeBuffer() {
    if ("HEAP".equals(bufferType)) {
      // Heap buffer: hasArray() returns true
      return ByteBuffer.wrap(rawData.clone());
    }
    // Direct buffer: hasArray() returns false
    ByteBuffer buffer = ByteBuffer.allocateDirect(rawData.length);
    buffer.put(rawData);
    buffer.flip();
    return buffer;
  }

  @TearDown(Level.Trial)
  public void tearDown() {
    if (columnVector != null) {
      columnVector.close();
      columnVector = null;
    }
  }

  @Benchmark
  public void newReadBinary() {
    newReader.readBinary(totalValues, columnVector, 0);
  }

  @Benchmark
  public void oldReadBinary() {
    oldReader.readBinary(totalValues, columnVector, 0);
  }

  public static void main(String[] args) throws Exception {
    // Forward CLI args (e.g. -p, -wi, -i, -f) so callers can override the
    // defaults specified by class-level annotations.
    Options opt = new OptionsBuilder()
        .parent(new CommandLineOptions(args))
        .include(VectorizedPlainValuesReaderBenchmark.class.getSimpleName())
        .build();
    new Runner(opt).run();
  }
}

Perform build/sbt "sql/Test/runMain org.apache.spark.sql.execution.datasources.parquet.VectorizedPlainValuesReaderBenchmark" to conduct the test

Benchmark results(click to expand)

Java 17

Benchmark                                           (bufferType)  (columnVectorType)  (totalValues)  (valueLength)  Mode  Cnt        Score        Error  Units
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           1024             16  avgt   50    21133.686 ±    103.769  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           1024             64  avgt   50    40714.762 ±    327.320  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           1024            256  avgt   50   127898.922 ±   1814.317  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           4096             16  avgt   50    83794.427 ±    423.578  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           4096             64  avgt   50   148011.789 ±   1657.605  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           4096            256  avgt   50   466875.025 ±   6739.475  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           8192             16  avgt   50   166724.800 ±    966.615  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           8192             64  avgt   50   310514.587 ±   3240.505  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           8192            256  avgt   50   982629.498 ±  11190.783  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           1024             16  avgt   50    18746.843 ±     51.526  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           1024             64  avgt   50    24426.161 ±     85.028  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           1024            256  avgt   50    58474.238 ±    237.509  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           4096             16  avgt   50    91843.587 ±    410.836  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           4096             64  avgt   50   121805.218 ±    122.957  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           4096            256  avgt   50  1413572.197 ±   4883.098  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           8192             16  avgt   50   154660.449 ±    895.345  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           8192             64  avgt   50   250413.950 ±    984.141  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           8192            256  avgt   50  1553024.626 ±   9369.443  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           1024             16  avgt   50    22171.027 ±     63.213  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           1024             64  avgt   50    39802.235 ±    300.151  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           1024            256  avgt   50   118034.848 ±    933.948  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           4096             16  avgt   50    87004.832 ±    330.969  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           4096             64  avgt   50   154505.438 ±   1108.452  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           4096            256  avgt   50   481165.804 ±   4692.772  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           8192             16  avgt   50   175463.325 ±    620.053  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           8192             64  avgt   50   317021.089 ±   1836.755  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           8192            256  avgt   50   962231.599 ±   9226.917  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           1024             16  avgt   50    22266.132 ±    181.957  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           1024             64  avgt   50    42590.654 ±    204.662  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           1024            256  avgt   50    76168.938 ±   1020.339  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           4096             16  avgt   50    91004.256 ±   2420.553  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           4096             64  avgt   50   363483.527 ±   9405.034  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           4096            256  avgt   50   530389.201 ± 358392.769  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           8192             16  avgt   50   192375.671 ±   9134.541  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           8192             64  avgt   50   775415.608 ±   7968.378  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           8192            256  avgt   50   948844.125 ± 184802.997  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           1024             16  avgt   50    20143.714 ±    127.970  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           1024             64  avgt   50    37213.528 ±    294.623  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           1024            256  avgt   50   115370.388 ±   1277.143  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           4096             16  avgt   50    81896.442 ±    319.206  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           4096             64  avgt   50   149584.303 ±   1966.657  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           4096            256  avgt   50   478158.897 ±   4064.925  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           8192             16  avgt   50   164793.858 ±   1039.258  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           8192             64  avgt   50   299629.657 ±   1684.947  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           8192            256  avgt   50   960621.319 ±   9025.326  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           1024             16  avgt   50    19042.590 ±     33.934  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           1024             64  avgt   50    24329.773 ±     86.621  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           1024            256  avgt   50    58411.323 ±    217.244  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           4096             16  avgt   50    92017.040 ±    376.556  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           4096             64  avgt   50   126487.647 ±    604.634  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           4096            256  avgt   50   230289.460 ±   2777.458  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           8192             16  avgt   50   149320.253 ±    595.932  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           8192             64  avgt   50   248600.054 ±   1645.170  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           8192            256  avgt   50   768789.899 ±   7270.904  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           1024             16  avgt   50    28069.933 ±     94.204  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           1024             64  avgt   50    48188.101 ±    289.352  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           1024            256  avgt   50   128637.878 ±   2340.360  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           4096             16  avgt   50   112014.092 ±    777.772  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           4096             64  avgt   50   192862.134 ±    847.013  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           4096            256  avgt   50   566865.312 ±   7235.458  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           8192             16  avgt   50   223146.102 ±   1132.153  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           8192             64  avgt   50   393649.960 ±   3152.221  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           8192            256  avgt   50  1091348.875 ±   9892.825  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           1024             16  avgt   50    27603.142 ±     81.746  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           1024             64  avgt   50    41045.293 ±    243.418  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           1024            256  avgt   50    96053.138 ±   2003.381  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           4096             16  avgt   50   105388.181 ±    599.746  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           4096             64  avgt   50   167514.849 ±    645.858  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           4096            256  avgt   50   362335.778 ±   2938.215  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           8192             16  avgt   50   223378.419 ±   2566.009  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           8192             64  avgt   50   379184.204 ±   6134.777  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           8192            256  avgt   50  2243461.086 ± 322551.647  ns/op

Java 21

Benchmark                                           (bufferType)  (columnVectorType)  (totalValues)  (valueLength)  Mode  Cnt        Score        Error  Units
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           1024             16  avgt   50    20792.936 ±     11.228  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           1024             64  avgt   50    32154.816 ±    328.300  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           1024            256  avgt   50    78592.323 ±    299.584  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           4096             16  avgt   50    82743.923 ±     79.815  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           4096             64  avgt   50   123702.471 ±    344.559  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           4096            256  avgt   50   326172.070 ±   2779.330  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           8192             16  avgt   50   165555.669 ±    318.138  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           8192             64  avgt   50   246627.341 ±    881.138  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           8192            256  avgt   50   646678.788 ±   2935.725  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           1024             16  avgt   50    21927.422 ±     24.128  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           1024             64  avgt   50    26163.304 ±     30.701  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           1024            256  avgt   50    50298.061 ±    123.971  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           4096             16  avgt   50    77656.063 ±    175.033  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           4096             64  avgt   50   103280.982 ±    221.227  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           4096            256  avgt   50   189700.428 ±    551.328  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           8192             16  avgt   50   154309.639 ±    291.313  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           8192             64  avgt   50   205462.097 ±    400.891  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           8192            256  avgt   50  1552902.350 ±  14219.023  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           1024             16  avgt   50    22033.101 ±    117.469  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           1024             64  avgt   50    32175.461 ±    177.490  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           1024            256  avgt   50    81801.706 ±    875.499  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           4096             16  avgt   50    87343.132 ±    105.192  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           4096             64  avgt   50   126698.538 ±    342.324  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           4096            256  avgt   50   318603.784 ±   1587.226  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           8192             16  avgt   50   173906.502 ±    173.984  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           8192             64  avgt   50   251866.225 ±    277.314  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           8192            256  avgt   50   641804.331 ±   2665.650  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           1024             16  avgt   50    24193.047 ±     77.459  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           1024             64  avgt   50    31668.278 ±    844.566  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           1024            256  avgt   50   335385.137 ±   4701.693  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           4096             16  avgt   50    95884.649 ±    663.374  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           4096             64  avgt   50   147133.621 ±  19982.708  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           4096            256  avgt   50  1518057.520 ±  16671.100  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           8192             16  avgt   50   211044.238 ±  12543.959  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           8192             64  avgt   50   349151.424 ±  37775.452  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           8192            256  avgt   50   998617.960 ± 268723.495  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           1024             16  avgt   50    23289.402 ±     61.648  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           1024             64  avgt   50    31459.997 ±    146.714  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           1024            256  avgt   50    77385.110 ±    288.525  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           4096             16  avgt   50    82537.027 ±     54.329  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           4096             64  avgt   50   122728.156 ±    428.668  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           4096            256  avgt   50   314828.587 ±    430.744  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           8192             16  avgt   50   164735.844 ±   1007.932  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           8192             64  avgt   50   245329.614 ±   1094.253  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           8192            256  avgt   50   621594.576 ±    890.237  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           1024             16  avgt   50    21882.702 ±     18.040  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           1024             64  avgt   50    26182.730 ±      9.474  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           1024            256  avgt   50    49825.929 ±     70.440  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           4096             16  avgt   50    86606.432 ±     92.960  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           4096             64  avgt   50   103397.329 ±    247.826  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           4096            256  avgt   50   188128.946 ±    216.008  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           8192             16  avgt   50   154325.782 ±     92.420  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           8192             64  avgt   50   205757.658 ±    427.901  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           8192            256  avgt   50  1571651.603 ±   5295.372  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           1024             16  avgt   50    27107.719 ±     27.489  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           1024             64  avgt   50    41349.661 ±     45.008  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           1024            256  avgt   50    94134.921 ±    659.365  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           4096             16  avgt   50   107704.612 ±     93.806  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           4096             64  avgt   50   159254.420 ±    598.247  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           4096            256  avgt   50   363235.908 ±    787.751  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           8192             16  avgt   50   220019.108 ±    252.893  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           8192             64  avgt   50   323097.051 ±    288.586  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           8192            256  avgt   50   745273.951 ±   1250.609  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           1024             16  avgt   50    26419.773 ±     76.761  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           1024             64  avgt   50    39714.193 ±    480.726  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           1024            256  avgt   50    88174.807 ±   3618.109  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           4096             16  avgt   50   102708.357 ±    173.331  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           4096             64  avgt   50   308566.639 ±   3679.216  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           4096            256  avgt   50   278667.492 ±   1505.321  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           8192             16  avgt   50   231747.397 ±   3885.522  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           8192             64  avgt   50   382719.924 ±  12336.323  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           8192            256  avgt   50   847664.028 ±  23769.709  ns/op

Java 25

Benchmark                                           (bufferType)  (columnVectorType)  (totalValues)  (valueLength)  Mode  Cnt        Score        Error  Units
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           1024             16  avgt   50    19454.336 ±     51.571  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           1024             64  avgt   50    35224.107 ±     93.873  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           1024            256  avgt   50   106094.049 ±    607.124  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           4096             16  avgt   50    76377.758 ±    170.304  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           4096             64  avgt   50   136363.493 ±    420.498  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           4096            256  avgt   50   401909.415 ±   1978.047  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           8192             16  avgt   50   153405.787 ±    329.935  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           8192             64  avgt   50   277290.682 ±    638.883  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP             ON_HEAP           8192            256  avgt   50   830389.462 ±   4434.923  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           1024             16  avgt   50    19447.941 ±     42.082  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           1024             64  avgt   50    24238.319 ±     84.138  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           1024            256  avgt   50    46731.494 ±    190.227  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           4096             16  avgt   50    74605.605 ±     50.031  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           4096             64  avgt   50    99683.575 ±    131.143  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           4096            256  avgt   50   184584.818 ±   2159.308  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           8192             16  avgt   50   150929.836 ±    237.595  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           8192             64  avgt   50   200577.256 ±    812.859  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary          HEAP            OFF_HEAP           8192            256  avgt   50   369071.136 ±   3405.637  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           1024             16  avgt   50    21421.781 ±     60.139  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           1024             64  avgt   50    34617.991 ±     65.177  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           1024            256  avgt   50    98082.743 ±    333.873  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           4096             16  avgt   50    83154.102 ±    138.081  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           4096             64  avgt   50   137378.799 ±    475.677  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           4096            256  avgt   50   397669.586 ±   1724.217  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           8192             16  avgt   50   165375.588 ±    308.414  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           8192             64  avgt   50   281011.423 ±    515.650  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT             ON_HEAP           8192            256  avgt   50   799352.042 ±   3406.062  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           1024             16  avgt   50    23396.644 ±    203.316  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           1024             64  avgt   50    30945.248 ±    172.520  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           1024            256  avgt   50    67093.924 ±    305.839  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           4096             16  avgt   50    95412.589 ±   1732.252  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           4096             64  avgt   50   321344.149 ±   9031.271  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           4096            256  avgt   50  1329057.617 ±  28490.214  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           8192             16  avgt   50   196217.213 ±   8448.670  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           8192             64  avgt   50   817464.101 ±  16282.918  ns/op
VectorizedPlainValuesReaderBenchmark.newReadBinary        DIRECT            OFF_HEAP           8192            256  avgt   50   796312.740 ± 147155.047  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           1024             16  avgt   50    19174.985 ±     91.202  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           1024             64  avgt   50    33204.299 ±     76.652  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           1024            256  avgt   50    94013.579 ±    327.183  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           4096             16  avgt   50    76527.485 ±    344.190  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           4096             64  avgt   50   131084.445 ±    303.759  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           4096            256  avgt   50   388912.079 ±   1287.693  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           8192             16  avgt   50   153907.585 ±    272.593  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           8192             64  avgt   50   269241.711 ±   1199.887  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP             ON_HEAP           8192            256  avgt   50   793134.485 ±   3236.322  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           1024             16  avgt   50    19431.150 ±     21.515  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           1024             64  avgt   50    24472.057 ±     25.534  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           1024            256  avgt   50    47889.963 ±    213.726  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           4096             16  avgt   50    75002.105 ±     70.072  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           4096             64  avgt   50    99832.226 ±    153.149  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           4096            256  avgt   50   188772.060 ±   1725.794  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           8192             16  avgt   50   150155.171 ±    104.622  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           8192             64  avgt   50   200136.556 ±    684.288  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary          HEAP            OFF_HEAP           8192            256  avgt   50   367258.001 ±   4519.780  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           1024             16  avgt   50    25810.554 ±     59.352  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           1024             64  avgt   50    43051.872 ±    112.457  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           1024            256  avgt   50   109254.476 ±    477.191  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           4096             16  avgt   50   104029.132 ±    298.893  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           4096             64  avgt   50   166544.608 ±    371.249  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           4096            256  avgt   50   449783.642 ±   2843.943  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           8192             16  avgt   50   207660.121 ±    246.678  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           8192             64  avgt   50   342410.957 ±    896.064  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT             ON_HEAP           8192            256  avgt   50   886503.174 ±   5756.155  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           1024             16  avgt   50    27903.709 ±    116.360  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           1024             64  avgt   50    38895.079 ±    112.427  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           1024            256  avgt   50    86762.539 ±   3300.275  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           4096             16  avgt   50   104535.716 ±    299.104  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           4096             64  avgt   50   248648.137 ±   2828.450  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           4096            256  avgt   50  1087469.272 ±   7033.347  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           8192             16  avgt   50   215132.226 ±   2714.048  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           8192             64  avgt   50   346034.950 ±  11971.660  ns/op
VectorizedPlainValuesReaderBenchmark.oldReadBinary        DIRECT            OFF_HEAP           8192            256  avgt   50   744647.868 ±  11141.025  ns/op

Across JDK 17, 21 and 25, the default DIRECT/ON_HEAP path achieves a performance improvement of 10%–22% and HEAP path results are unchanged as expected (code path not affected).

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code

LuciferYang · 2026-04-10T16:09:08Z

common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java

+   * Returns the native memory address of a direct {@link ByteBuffer}.
+   * The buffer must be direct; passing a heap buffer produces an undefined result.
+   */
+  public static long getDirectBufferAddress(ByteBuffer buffer) {


If this pr is accepted, a corresponding benchmark can be added to PlatformBenchmark at a later time.

+1. It'd be great to have some benchmark results!

sunchao

Looks reasonable to me. Left some comments.

sunchao · 2026-04-11T16:52:11Z

sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OffHeapColumnVector.java

+      Platform.copyMemory(src.array(), Platform.BYTE_ARRAY_OFFSET + src.arrayOffset() + srcIndex,
+        null, data + rowId, count);
+    } else {
+      long srcAddr = Platform.getDirectBufferAddress(src) + srcIndex;


I think hasArray() does not necessarily mean the buffer is a direct buffer. We can perhaps strengthen this by also check src.isDirect?

d2e8bb3 address this

sunchao · 2026-04-11T16:54:41Z

sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java

+   * Copies {@code count} bytes from a {@link ByteBuffer} starting at absolute position
+   * {@code srcIndex} into this column at {@code rowId}. Does not modify the buffer's position.
+   */
+  public abstract void putBytes(int rowId, int count, ByteBuffer src, int srcIndex);


Anyway we can add some test cases for this new API?

LuciferYang · 2026-04-13T03:39:55Z

sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnVectorSuite.scala

+    verifyPutByteArray(testVector)
+  }
+
+  testVectors("putBytes from ByteBuffer", 16, ByteType) { testVector =>


@sunchao Added new tests for both new APIs.

init

901fd0f

LuciferYang commented Apr 10, 2026

View reviewed changes

format

75012bb

LuciferYang marked this pull request as draft April 10, 2026 17:51

LuciferYang marked this pull request as ready for review April 11, 2026 01:59

sunchao reviewed Apr 11, 2026

View reviewed changes

LuciferYang added 2 commits April 13, 2026 11:32

address comments

d2e8bb3

add tests for putBytes

76f2f1f

LuciferYang commented Apr 13, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-56438][SQL][CORE] Optimize `VectorizedPlainValuesReader.readBinary` for direct ByteBuffer by eliminating intermediate byte[] copy#55296

[SPARK-56438][SQL][CORE] Optimize `VectorizedPlainValuesReader.readBinary` for direct ByteBuffer by eliminating intermediate byte[] copy#55296
LuciferYang wants to merge 4 commits intoapache:masterfrom
LuciferYang:SPARK-56438

LuciferYang commented Apr 10, 2026 •

edited

Loading

Uh oh!

LuciferYang Apr 10, 2026

Uh oh!

sunchao Apr 11, 2026

Uh oh!

sunchao left a comment

Uh oh!

sunchao Apr 11, 2026

Uh oh!

LuciferYang Apr 13, 2026

Uh oh!

sunchao Apr 11, 2026

Uh oh!

LuciferYang Apr 13, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

LuciferYang commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

LuciferYang Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

sunchao Apr 11, 2026

Choose a reason for hiding this comment

Uh oh!

sunchao left a comment

Choose a reason for hiding this comment

Uh oh!

sunchao Apr 11, 2026

Choose a reason for hiding this comment

Uh oh!

LuciferYang Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

sunchao Apr 11, 2026

Choose a reason for hiding this comment

Uh oh!

LuciferYang Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

LuciferYang commented Apr 10, 2026 •

edited

Loading

LuciferYang Apr 13, 2026 •

edited

Loading