[SPARK-52401][SQL] Fix DataFrame.collect() cache invalidation after saveAsTable append; add regression test

[ikshitiz]

[SPARK-52401][SQL] Fix DataFrame.collect() cache invalidation after saveAsTable append; add regression test

What changes were proposed in this pull request?

This PR fixes a cache invalidation bug in Spark SQL where DataFrame.collect() could return stale results after appending data to a table using saveAsTable with mode="append". The fix ensures that all DataFrame operations (collect(), count(), etc.) consistently reflect the latest table contents after append, overwrite, or similar operations.

Key changes:

• In DataSourceV2Strategy.scala, the refreshCache function now uses uncacheQuery instead of recacheByPlan for append/overwrite operations. This properly invalidates the cache so that subsequent DataFrame actions read fresh data from the table.
• Added a regression test suite (DataFrameCacheSuite.scala) to verify correct cache invalidation and DataFrame behavior after table updates.
• Included a summary document (SPARK-52401_FIX_SUMMARY.md) and Python scripts for easy verification and review.

Why are the changes needed?

Previously, after appending data to a table, .count() would return the correct row count, but .collect() could return outdated results (e.g., an empty list). This was due to the cache manager re-executing the same logical plan, which did not reflect the updated table data. This PR ensures that the cache is invalidated, so all DataFrame operations see the latest data.

Does this PR introduce any user-facing change?

Yes. After this fix:

• .collect(), .count(), and all DataFrame actions will always reflect the current state of the table after append, overwrite, or similar operations.
• This resolves a long-standing source of confusion and inconsistency for Spark SQL users.

How was this patch tested?

• Added a new regression test suite: sql/core/src/test/scala/org/apache/spark/sql/DataFrameCacheSuite.scala
• Included Python scripts (test_spark_52401.py, test_spark_52401_comprehensive.py) for manual and cross-language verification.
• All tests verify that DataFrame operations return up-to-date results after table modifications.

Additional context

• See the included SPARK-52401_FIX_SUMMARY.md for a detailed explanation of the issue, root cause, and solution.
• This change only affects cache invalidation for operations that modify table data (append, overwrite, delete, etc.) and is backward compatible.