[CH] simdjson doesn't support strings which contain invalid unicodes

[lgbo-ustc]

Backend

CH (ClickHouse)

Bug description

[Expected behavior] and [actual behavior].

simdjson fails to parse following json

{"a":"431924697b1\ufffdSC\u2995\u06a2\r8\u0010\u06e3\udee4\udff0L\u0001Y"}

\udee4 and \udff0 are invalid unicodes

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

No response

[PHILO-HE]

@lgbo-ustc, in velox, we made simdjson skip utf8 validation by setting SIMDJSON_SKIPUTF8VALIDATION=ON. See

incubator-gluten/ep/build-velox/src/modify_velox.patch

Line 70 in aac1b87

+set(SIMDJSON_SKIPUTF8VALIDATION ON)

Is this setting applicable to fixing your issue?

[lgbo-ustc]

@lgbo-ustc, in velox, we made simdjson skip utf8 validation by setting SIMDJSON_SKIPUTF8VALIDATION=ON. See

incubator-gluten/ep/build-velox/src/modify_velox.patch

Line 70 in aac1b87

+set(SIMDJSON_SKIPUTF8VALIDATION ON)

Is this setting applicable to fixing your issue?

Thanks, @PHILO-HE . We try this option, but it doesn't work. We print the calling stack as following

* thread #1, name = 'clickhouse', stop reason = signal SIGSEGV
  * frame #0: 0x00000000140efe9f clickhouse`simdjson::haswell::dom_parser_implementation::stage2(simdjson::dom::document&) [inlined] simdjson::haswell::(anonymous namespace)::stage2::tape_builder::on_end_string(this=<unavailable>, dst=0x0000000000000000) at tape_builder.h:290:8
    frame #1: 0x00000000140efe92 clickhouse`simdjson::haswell::dom_parser_implementation::stage2(simdjson::dom::document&) [inlined] simdjson::haswell::(anonymous namespace)::stage2::tape_builder::visit_string(this=<unavailable>, iter=<unavailable>, value=<unavailable>, key=false) at tape_builder.h:167:3
    frame #2: 0x00000000140efde3 clickhouse`simdjson::haswell::dom_parser_implementation::stage2(simdjson::dom::document&) [inlined] simdjson::error_code simdjson::haswell::(anonymous namespace)::stage2::json_iterator::visit_primitive<simdjson::haswell::(anonymous namespace)::stage2::tape_builder>(this=<unavailable>, visitor=<unavailable>, value=<unavailable>) at json_iterator.h:308:20
    frame #3: 0x00000000140efde3 clickhouse`simdjson::haswell::dom_parser_implementation::stage2(simdjson::dom::document&) [inlined] simdjson::haswell::(anonymous namespace)::stage2::tape_builder::visit_primitive(this=<unavailable>, iter=<unavailable>, value=<unavailable>) at tape_builder.h:112:15
    frame #4: 0x00000000140efde3 clickhouse`simdjson::haswell::dom_parser_implementation::stage2(simdjson::dom::document&) [inlined] simdjson::error_code simdjson::haswell::(anonymous namespace)::stage2::json_iterator::walk_document<false, simdjson::haswell::(anonymous namespace)::stage2::tape_builder>(this=<unavailable>, visitor=<unavailable>) at json_iterator.h:176:16
    frame #5: 0x00000000140efd04 clickhouse`simdjson::haswell::dom_parser_implementation::stage2(simdjson::dom::document&) [inlined] simdjson::error_code simdjson::haswell::(anonymous namespace)::stage2::tape_builder::parse_document<false>(dom_parser=<unavailable>, doc=<unavailable>) at tape_builder.h:105:15
    frame #6: 0x00000000140ef859 clickhouse`simdjson::haswell::dom_parser_implementation::stage2(this=<unavailable>, _doc=<unavailable>) at haswell.cpp:143:10
    frame #7: 0x0000000006c156c0 clickhouse`simdjson::dom::parser::parse_into_document(simdjson::dom::document&, unsigned char const*, unsigned long, bool) & + 480
    frame #8: 0x0000000006c13b74 clickhouse`DB::SimdJSONParser::parse(std::__1::basic_string_view<char, std::__1::char_traits<char>>, DB::SimdJSONParser::Element&) + 52

And found that we must let allow_replacement be true in parse_string, but there is no control option here.

simdjson_warn_unused simdjson_inline error_code tape_builder::visit_string(json_iterator &iter, const uint8_t *value, bo
ol key) noexcept {
  iter.log_value(key ? "key" : "string");
  uint8_t *dst = on_start_string(iter);
  dst = stringparsing::parse_string(value+1, dst, false); // We do not allow replacement when the escape characters are
invalid.
  std::cout << "xxx dist is null " << (dst == nullptr) << "\n";
  if (dst == nullptr) {
    iter.log_error("Invalid escape in string");
    return STRING_ERROR;
  }
  on_end_string(dst);
  return SUCCESS;
}

Which api of simdjson do you use, ondemand or dom ?

[PHILO-HE]

@lgbo-ustc, we are using ondemand api which may only validate part of JSON until given JSON path is found, not whole JSON string.

[PHILO-HE]

What we encountered is a case like the below:
https://github.com/apache/incubator-gluten/blob/main/backends-velox/src/test/scala/org/apache/gluten/execution/ScalarFunctionsValidateSuite.scala#L270