Skip to content

Commit

Permalink
chore: fix compatibility guide (#978)
Browse files Browse the repository at this point in the history
## Rationale for this change

This is a followup PR of #953 

## What changes are included in this PR?

Updated the template, otherwise `make release` will wipe it out
  • Loading branch information
kazuyukitanimura authored Sep 27, 2024
1 parent 0ee7df8 commit f64553b
Showing 1 changed file with 8 additions and 0 deletions.
8 changes: 8 additions & 0 deletions docs/templates/compatibility-template.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,14 @@ Comet uses the Rust regexp crate for evaluating regular expressions, and this ha
regular expression engine. Comet will fall back to Spark for patterns that are known to produce different results, but
this can be overridden by setting `spark.comet.regexp.allowIncompatible=true`.

## Floating number comparison

Spark normalizes NaN and zero for floating point numbers for several cases. See `NormalizeFloatingNumbers` optimization rule in Spark.
However, one exception is comparison. Spark does not normalize NaN and zero when comparing values
because they are handled well in Spark (e.g., `SQLOrderingUtil.compareFloats`). But the comparison
functions of arrow-rs used by DataFusion do not normalize NaN and zero (e.g., [arrow::compute::kernels::cmp::eq](https://docs.rs/arrow/latest/arrow/compute/kernels/cmp/fn.eq.html#)).
So Comet will add additional normalization expression of NaN and zero for comparison.

## Cast

Cast operations in Comet fall into three levels of support:
Expand Down

0 comments on commit f64553b

Please sign in to comment.