Skip to content

Commit

Permalink
[SPARK-42268][CONNECT][PYTHON][TESTS][FOLLOWUP] Add test_simple_udt
Browse files Browse the repository at this point in the history
… for `UserDefinedType`

### What changes were proposed in this pull request?
add more tests for UDT

### Why are the changes needed?
apache#39835 added tests for UDT for this conversion:
UDT in Server -> proto -> UDT in Python Client

As per xinrong-meng 's suggestion, this PR adds more tests to cover this:
UDT in Python Client -> proto -> UDT in Server -> proto -> UDT in Python Client

### Does this PR introduce _any_ user-facing change?
No, test-only

### How was this patch tested?
added UT

Closes apache#39913 from zhengruifeng/connect_more_test_udt.

Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit b99fc5e)
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
zhengruifeng authored and dongjoon-hyun committed Feb 7, 2023
1 parent 3f922fa commit ea8ff26
Showing 1 changed file with 16 additions and 0 deletions.
16 changes: 16 additions & 0 deletions python/pyspark/sql/tests/connect/test_connect_basic.py
Original file line number Diff line number Diff line change
Expand Up @@ -2577,6 +2577,22 @@ def test_collect_nested_type(self):
).collect(),
)

def test_simple_udt(self):
from pyspark.ml.linalg import MatrixUDT, VectorUDT

for schema in [
StructType().add("key", LongType()).add("val", PythonOnlyUDT()),
StructType().add("key", LongType()).add("val", ArrayType(PythonOnlyUDT())),
StructType().add("key", LongType()).add("val", MapType(LongType(), PythonOnlyUDT())),
StructType().add("key", LongType()).add("val", PythonOnlyUDT()),
StructType().add("key", LongType()).add("vec", VectorUDT()),
StructType().add("key", LongType()).add("mat", MatrixUDT()),
]:
cdf = self.connect.createDataFrame(data=[], schema=schema)
sdf = self.spark.createDataFrame(data=[], schema=schema)

self.assertEqual(cdf.schema, sdf.schema)

def test_simple_udt_from_read(self):
from pyspark.ml.linalg import Matrices, Vectors

Expand Down

0 comments on commit ea8ff26

Please sign in to comment.