Fix NOTICE and LICENSE in the spark-runtime jar

[jbonofre]

This PR does:

• Update Avro copyright
• Remove paranamer (not found in the jar)
• Update Parquet copyright
• Update Thrift copyright
• Update ORC copyright
• Update Hive copyright
• Remove airlift slice (not found in the jar)
• Remove presto (not found in the jar)
• Remove findbugs (jsr305) (not found in the jar)
• Remove j2objc (not found in the jar)
• Remove animal sniffer annotations (not found in the jar)
• Remove carrot (not found in the jar)
• Remove lucene (not found in the jar)
• Remove yetus (not found in the jar)
• Update Nessie copyright
• Remove delta (not found in the jar)
• Remove Apache projects from NOTICE

[jbonofre]

Update Nessie NOTICE from the latest version.

[rdblue]

As in #12145, I think we need Nessie to update license documentation before we can conclude that this is accurate. The current NOTICE for binary artifacts (that we use) still includes a line about GPL.

[jbonofre]

I've updated Nessie NOTICE corresponding to 0.102.5 release.

[amogh-jahagirdar]

I think it mostly looks great, just one comment on keeping the mention of Presto in the LICENSE. Thank you @jbonofre !

[rdblue]

The description for this issue has:

Remove delta (not found in the jar)

I don't see the change anymore, but in case I'm missing it: this was included because source code is partially based on a Delta class, so to avoid any doubt we wanted to include it:

AssignmentAlignmentSupport is an independent development but UpdateExpressionsSupport in Delta was used as a reference.

[jbonofre]

The description for this issue has:

Remove delta (not found in the jar)

I don't see the change anymore, but in case I'm missing it: this was included because source code is partially based on a Delta class, so to avoid any doubt we wanted to include it:

AssignmentAlignmentSupport is an independent development but UpdateExpressionsSupport in Delta was used as a reference.

Yes, it was my bad. I re-add delta mention when I saw the code in the root LICENSE. So, we are clean now.

[rdblue]

This can be trimmed because it is from our own NOTICE. We are responsible for all the content about what Iceberg ships, so there is no need to duplicate.

[jbonofre]

I trimmed the Nessie NOTICE by removing Iceberg and Netty (as it's already mentioned in the Iceberg Spark runtime NOTICE). I keep Polaris mention as it's not duplicate (ok, it's about build/gradle resources in Polaris, but for consistency, better to keep).

[rdblue]

It's too bad that this was based on a source inclusion in Nessie. Nessie could probably check that none of this applies to ResolveConf that was copied to avoid more work for downstream bundlers.

[jbonofre]

I think Nessie is using a similar approach as in Iceberg: as discussed, Iceberg is using LICENSE and NOTICE from source in jar files by default (only bundle and runtime jars have specific LICENSE and NOTICE.
For instance, Iceberg mentions ScriptRunner which is used for iceberg-hive-metastore test, but mention in all Iceberg jars. Or AssignmentAlignmentSupport is used only in iceberg-spark-extensions but mentioned in all iceberg jars.
So, I'm not very confortable to ask Nessie to change that (even if it's what I asked but the Nessie team says they will plan this as it's more work) when Iceberg is doing the same.

[rdblue]

The difference is that we're more selective with NOTICE content when code is copied. That's because we can more easily verify that sections of the original project's NOTICE do or do not apply to the specific section that was copied or used.

[Fokko]

I'm seeing

This binary artifact contains code from Daniel Lemire's JavaFastPFOR project.

Copyright: 2013 Daniel Lemire
Home page: https://github.com/lemire/JavaFastPFOR
License: Apache License Version 2.0 http://www.apache.org/licenses/LICENSE-2.0

In the license, but I'm not able to find this in the dependency tree

[amogh-jahagirdar]

I'm seeing

This binary artifact contains code from Daniel Lemire's JavaFastPFOR project.

Copyright: 2013 Daniel Lemire
Home page: https://github.com/lemire/JavaFastPFOR
License: Apache License Version 2.0 http://www.apache.org/licenses/LICENSE-2.0

In the license, but I'm not able to find this in the dependency tree

The Parquet License still references JavaFastPFOR https://github.com/apache/parquet-java/blob/parquet-1.15.x/LICENSE#L189 and I still see some references to the source in Parquet https://github.com/apache/parquet-java/blob/fb6f0be0323f5f52715b54b8c6602763d8d0128d/parquet-generator/src/main/java/org/apache/parquet/encoding/bitpacking/IntBasedBitPackingGenerator.java#L29 . I think we'd need to keep this, until we update Parquet (based on the code comment it looks like that code path I referenced is no longer really used upstream)?

[Fokko]

Thanks for the clarification @amogh-jahagirdar, I need to work on my grep skills :D