Skip to content

Commit

Permalink
[SPARK-46262][PS] Enable test for np.left_shift for Pandas-on-Spark…
Browse files Browse the repository at this point in the history
… object

### What changes were proposed in this pull request?

This PR proposes to enable test for `np.left_shift` for Pandas-on-Spark object.

### Why are the changes needed?

Because now we completely support the PyArrow>=4.0.0, so the behavior difference within PyArrow<0.15 will not appear.

So we can enable the test since the behavior always same within the supported version as below:

```python
>>> pdf = pd.DataFrame({"A": [1, 2, 3, 4]})
>>> psdf = ps.DataFrame({"A": [1, 2, 3, 4]})
>>> np.left_shift(pdf, 1)
   A
0  2
1  4
2  6
3  8
>>> np.left_shift(psdf, 1)
   A
0  2
1  4
2  6
3  8
```

### Does this PR introduce _any_ user-facing change?

No, this just enables the test case.

### How was this patch tested?

The existing CI especially `NumPyCompatTests` should pass.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#44178 from itholic/enable_np_left_shift.

Authored-by: Haejoon Lee <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
  • Loading branch information
itholic authored and zhengruifeng committed Dec 5, 2023
1 parent 1281703 commit 60bf681
Showing 1 changed file with 0 additions and 4 deletions.
4 changes: 0 additions & 4 deletions python/pyspark/pandas/tests/test_numpy_compat.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,10 +41,6 @@ class NumPyCompatTestsMixin:
"log1p", # flaky
"modf",
"floor_divide", # flaky
# Results seem inconsistent in a different version of, I (Hyukjin) suspect, PyArrow.
# From PyArrow 0.15, seems it returns the correct results via PySpark. Probably we
# can enable it later when Koalas switches to PyArrow 0.15 completely.
"left_shift",
]

@property
Expand Down

0 comments on commit 60bf681

Please sign in to comment.