[KYUUBI #6912][LINEAGE] Properly handle empty attribute set on mergeRelationColumnLineage

[xglv1985]

Why are the changes needed?

Issue reference:

#6912

How to reproduce the issue?

The changes in this PR will avoid a wrong result when generating the instance of org.apache.kyuubi.plugin.lineage.Lineage, in the certain case as follows:
step 1: create a temporary view from a file
step 2: insert into a table by selecting from the temporary view in step 1
step 3: generate the lineage when executing the insert statement in step 2
In detail, please see the UT code submission in this patch.

The issue analysis

Let's see the current code when getting the Lineage object by resolving a LogicalPlan object:

According to the above logic, a None org.apache.kyuubi.plugin.lineage.Lineage object will be generated due to "try-catch" self-protection, in this certain case. This None object will lead to problems in the following 2 scenes:

Unit Test Environment

In Unit Test, when the code runs here a "None.get" exception will be raised:

Here's the runtime exception stack:

None.get
java.util.NoSuchElementException: None.get
	at scala.None$.get(Option.scala:529)
	at scala.None$.get(Option.scala:527)
	at org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.extractLineageWithoutExecuting(SparkSQLLineageParserHelperSuite.scala:1485)
	at org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.$anonfun$new$83(SparkSQLLineageParserHelperSuite.scala:1465)

Production Environment

This Lineage object cannot be used in the production environment because it has a None value which lacks some necessary lineage information. The right content of the Lineage instance in the above case should be:

inputTables(List())
outputTables(List(spark_catalog.test_db.test_table_from_dir))
columnLineage(List(ColumnLineage(spark_catalog.test_db.test_table_from_dir.a0,Set()), ColumnLineage(spark_catalog.test_db.test_table_from_dir.b0,Set())))

a newly added test case(test directory to table) passed after this issue is fixed.

How to fix the issue?

Add a "Empty judgment" logic. In detail, please see the code submission in this patch.

How was this patch tested?

1. by adding a new test case in UT code and make sure it passes
2. by submitting a Spark application including the SQL of this case in the production environment, and make sure a right Lineage instance is generated, instead of a None object

Was this patch authored or co-authored using generative AI tooling?

No

[pan3793]

maybe can be written in

relationOutput.foldLeft((ListMap[Attribute, AttributeSet](), relationColumnLineage)) {
        case ((acc, x), attr) if x.isEmpty =>
          (acc + (attr -> AttributeSet.empty), ListMap.empty)
        case ((acc, x), attr) =>
          (acc + (attr -> x.head._2), x.tail)
      }._1

TBH, the code is not much readable ...

[pan3793]

nit: use UPPER CASE for SQL keywords, and format the SQL in a short readable way

Suggested change

	s"""
	|insert overwrite table test_db.test_table_from_dir
	|SELECT
	| `a`,
	| `b`
	|FROM temp_view
	|""".stripMargin)
	s"""
	|INSERT OVERWRITE TABLE test_db.test_table_from_dir
	|SELECT `a`, `b` FROM temp_view
	|""".stripMargin)

[pan3793]

Suggested change

	s"""
	| CREATE OR REPLACE TEMPORARY VIEW temp_view
	|(
	| `a` STRING COMMENT '',
	| `b` STRING COMMENT ''
	|)
	|USING csv OPTIONS(
	| sep='\t',
	| path='${sourceFile.path}'
	|);
	|""".stripMargin).collect()
	s"""
	|CREATE OR REPLACE TEMPORARY VIEW temp_view (
	| `a` STRING COMMENT '',
	| `b` STRING COMMENT ''
	|) USING csv OPTIONS(
	| sep='\t',
	| path='${sourceFile.path}'
	|)
	|""".stripMargin).collect()

[pan3793]

thanks for fixing the issue, leave some nits, BTW, you can run dev/reformat to fix the code style.

[xglv1985]

thanks for fixing the issue, leave some nits, BTW, you can run dev/reformat to fix the code style.

thanks for reviewing. I will optimize the code by your advice.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.

Project coverage is 0.00%. Comparing base (dd9cc0e) to head (13a7107).
Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
...in/lineage/helper/SparkSQLLineageParseHelper.scala	0.00%	2 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff           @@
##           master   #6911   +/-   ##
======================================
  Coverage    0.00%   0.00%           
======================================
  Files         688     688           
  Lines       42590   42592    +2     
  Branches     5805    5806    +1     
======================================
- Misses      42590   42592    +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pan3793]

@xglv1985 Thanks for the contribution, CI failure is irrelevant, I have sent a PR(#6915) to fix that, and will merge this PR once CI is recovered.

[pan3793]

Thanks, merged to master/1.10.2/1.9.4