Skip to content

Commit ea8ff26

Browse files
zhengruifengdongjoon-hyun
authored andcommitted
[SPARK-42268][CONNECT][PYTHON][TESTS][FOLLOWUP] Add test_simple_udt for UserDefinedType
### What changes were proposed in this pull request? add more tests for UDT ### Why are the changes needed? apache#39835 added tests for UDT for this conversion: UDT in Server -> proto -> UDT in Python Client As per xinrong-meng 's suggestion, this PR adds more tests to cover this: UDT in Python Client -> proto -> UDT in Server -> proto -> UDT in Python Client ### Does this PR introduce _any_ user-facing change? No, test-only ### How was this patch tested? added UT Closes apache#39913 from zhengruifeng/connect_more_test_udt. Authored-by: Ruifeng Zheng <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit b99fc5e) Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 3f922fa commit ea8ff26

File tree

1 file changed

+16
-0
lines changed

1 file changed

+16
-0
lines changed

python/pyspark/sql/tests/connect/test_connect_basic.py

+16
Original file line numberDiff line numberDiff line change
@@ -2577,6 +2577,22 @@ def test_collect_nested_type(self):
25772577
).collect(),
25782578
)
25792579

2580+
def test_simple_udt(self):
2581+
from pyspark.ml.linalg import MatrixUDT, VectorUDT
2582+
2583+
for schema in [
2584+
StructType().add("key", LongType()).add("val", PythonOnlyUDT()),
2585+
StructType().add("key", LongType()).add("val", ArrayType(PythonOnlyUDT())),
2586+
StructType().add("key", LongType()).add("val", MapType(LongType(), PythonOnlyUDT())),
2587+
StructType().add("key", LongType()).add("val", PythonOnlyUDT()),
2588+
StructType().add("key", LongType()).add("vec", VectorUDT()),
2589+
StructType().add("key", LongType()).add("mat", MatrixUDT()),
2590+
]:
2591+
cdf = self.connect.createDataFrame(data=[], schema=schema)
2592+
sdf = self.spark.createDataFrame(data=[], schema=schema)
2593+
2594+
self.assertEqual(cdf.schema, sdf.schema)
2595+
25802596
def test_simple_udt_from_read(self):
25812597
from pyspark.ml.linalg import Matrices, Vectors
25822598

0 commit comments

Comments
 (0)