Commit c3473e4
committed
[SPARK-56411][SQL] Register Decimal in KryoSerializer so cached-batch spill works
### What changes were proposed in this pull request?
Register `Decimal`, `Decimal[]`, `java.math.BigDecimal`, and `java.math.BigInteger` in `KryoSerializer.loadableSparkClasses` so that Kryo strict registration mode(`spark.kryo.registrationRequired=true`) can serialize these types without throwing `Class is not registered`.
### Why are the changes needed?
`DefaultCachedBatchSerializer` writes cached batch stats via `kryo.writeClassAndObject(output, batch.stats)`. The stats row (`GenericInternalRow`) can contain `Decimal` values (e.g. min/max stats for a `DecimalType` column). When Kryo walks the row it encounters:
- `org.apache.spark.sql.types.Decimal` -- the value class itself.
- `java.math.BigDecimal` / `java.math.BigInteger` -- used internally when the value overflows `Long` precision (> 18 digits).
None of these were registered. Under strict mode, any cache spill or eviction of a batch with Decimal stats crashes:
```
com.esotericsoftware.kryo.KryoException: Class is not registered: org.apache.spark.sql.types.Decimal
```
This can be reproduced by caching a table with `DecimalType` columns(e.g. `CACHE TABLE store_sales`) and triggering a memory-pressure spill under Kryo strict mode.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Added new tests `DefaultCachedBatchKryoSerializerSuite`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
Closes #55287 from LuciferYang/SPARK-56411.
Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: yangjie01 <yangjie01@baidu.com>1 parent 2ad52c2 commit c3473e4
File tree
2 files changed
+120
-0
lines changed- core/src/main/scala/org/apache/spark/serializer
- sql/core/src/test/scala/org/apache/spark/sql/execution/columnar
2 files changed
+120
-0
lines changedLines changed: 4 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
598 | 598 | | |
599 | 599 | | |
600 | 600 | | |
| 601 | + | |
| 602 | + | |
601 | 603 | | |
602 | 604 | | |
603 | 605 | | |
| 606 | + | |
| 607 | + | |
604 | 608 | | |
605 | 609 | | |
606 | 610 | | |
| |||
Lines changed: 116 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
0 commit comments