[SPARK-50133][PYTHON] Support df.argument() for conversion to table argument in Spark Classic

[xinrong-meng]

What changes were proposed in this pull request?

We want to implement a function df.argument(), which returns a Column object as a table arument. This Column can then be passed as input to call a User-Defined Table Function (UDTF) for example.

The PR targets at Spark Classic only.

Why are the changes needed?

To reach parity with SQL functionality, specifically, #41750.

Does this PR introduce any user-facing change?

Yes.

For example:

>>> @udtf(returnType="a: int")
... class TestUDTF:
...     def eval(self, row: Row):
...         if row[0] > 5:
...             yield row[0],
... 
>>> df.argument()
Column<'functiontablesubqueryargumentexpression()'>
>>> TestUDTF(df.argument()).show()
+---+                                                                           
|  a|
+---+
|  6|
|  7|
+---+

How was this patch tested?

Unit tests.

Was this patch authored or co-authored using generative AI tooling?

No.

[ueshin]

Shall we compare with the result from SQL, which clarifies the equivalent SQL and also checks the schema.

[xinrong-meng]

Sounds good. I will combine test_udtf_with_table_argument_query and test_df_argument once Spark Connect support is ready.