Spark 2.3 Merge#97

Open

ymahajan wants to merge 142 commits into

snappy/branch-2.3from

spark_2.3_merge

Commits on Feb 21, 2018

[SPARK-13904][SCHEDULER] Add support for pluggable cluster manager

Hemant Bhanawat
authored and
ymahajan
committed
[SPARK-14729][SCHEDULER] Refactored YARN scheduler creation code to use newly added ExternalClusterManager

Hemant Bhanawat
authored and
ymahajan
committed
[SNAPPYDATA] increasing visibility of SparkContext.activeContext

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] add SnappyData's modification headers in updated files

Neeraj Kumar
authored and
ymahajan
committed
[SNAP-404] Address #comment about increasing decimal precision

Soubhik Chakraborty
authored and
ymahajan
committed
[SNAP-860] Removed hardcoding of size of Array used for storing DecimalType (#38)

SachinJanani
authored and
ymahajan
committed
[SNAPPYDATA] Try hard to not schedule on others if ExecutorCacheTaskLocation is alive

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] Use SnappyContext as default SQLContext on shell (#35)

nthanvi
authored and
ymahajan
committed
[SNAP-643] Increase visibility of some methods in GenerateUnsafeProjection

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] Fixing sequence of expression in an option

ahshahid
authored and
ymahajan
committed
[SNAP-931] Use non-secure randomUUID where appropriate (#40)

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] Adding SnappyData modification headers for missing files

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] Updated README.md with information on SnappyData's changes

Hemant Bhanawat
authored and
ymahajan
committed
[SNAPPYDATA] Optimizations for bootstrap

ahshahid
authored and
ymahajan
committed
[SNAPPYDATA] Gradle build scripts and build fixes

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] Dynamic CQ changes in spark streaming
ymahajan
committed
[SNAPPYDATA] Fix cluster startup due to executionId format

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] Accept Spark properties with "snappydata." prefix

nthanvi
authored and
ymahajan
committed
[SNAPPYDATA] More fixes for SnappyData for Spark 2.0

Sumedh Wale
authored and
ymahajan
committed
Snap 293 (#1)

nthanvi
authored and
ymahajan
committed
[SNAPPYDATA] fix a scalaStyle issue
ymahajan
committed
[SNAP-966] Prefer conversions to date/timestamp and not strings (#7)

Sumedh Wale
authored and
ymahajan
committed

Commits on Feb 22, 2018

[SNAP-1034] Optimizations at Spark layer as seen in profiling (#10)

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] Updated Benchmark code from Spark PR#13899

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] Spark version 2.0.1-2

Amogh Shetkar
authored and
ymahajan
committed
[SNAPPYDATA] fixing antlr generated code for IDEA

Sumedh Wale
authored and
ymahajan
committed
[SNAP-1083] fix numBuckets handling (#15)

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] Spark version 2.0.1-3

Amogh Shetkar
authored and
ymahajan
committed
[SNAPPYDATA] updating snappy-spark version after the merge

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] Bumping version to 2.0.3-1

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] Made two methods in Executor as protected to make them customizable for SnappyExecutors. (#26)

rishitesh
authored and
ymahajan
committed
[SNAPPYDATA]: Honoring JAVA_HOME variable while compiling java files

Soubhik Chakraborty
authored and
ymahajan
committed
[SNAP-1198] Use ConcurrentHashMap instead of queue for ContextCleaner.referenceBuffer (#32)

Sumedh Wale
authored and
ymahajan
committed

Commits on Feb 23, 2018

[SNAP-1194] explicit addLong/longValue methods in SQLMetrics (#33)

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] More optimizations to UTF8String

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] Adding fixed stats to common filter expressions

Hemant Bhanawat
authored and
ymahajan
committed
[SNAPPYDATA] adding kryo serialization missing in LongHashedRelation

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] Correcting HashPartitioning interface to match apache spark

Sumedh Wale
authored and
ymahajan
committed
[SNAP-1233] clear InMemorySorter before calling its reset (#35)

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] Adding more filter conditions for plan sizing as followup

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] reduced factors in filters a bit to be more conservative

Sumedh Wale
authored and
ymahajan
committed
[SNAP-1240] Snappy monitoring dashboard (#36)

snappy-sachin
authored and
ymahajan
committed

Commits on Mar 1, 2018

[SNAP-1251] Avoid exchange when number of shuffle partitions > child partitions (#37)

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] reverting lazy val to def for defaultNumPreShufflePartitions

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] Code changes for displaying product version details. (#38)

snappy-sachin
authored and
ymahajan
committed
[SNAPPYDATA] Removing duplicate RDD already in snappy-core

Sumedh Wale
authored and
ymahajan
committed
SNAP-1257 (#40)

snappy-sachin
authored and
ymahajan
committed
[SNAPPYDATA] Spark Version 2.0.3-2

Amogh Shetkar
authored and
ymahajan
committed
SNAP-1281: UI does not show up if spark shell is run without snappydata (#42)

snappy-sachin
authored and
ymahajan
committed

Commits on Mar 2, 2018

[SNAP-1185] Guard logging and time measurements (#28)

Sumedh Wale
authored and
ymahajan
committed
Snap 982 (#43)

rishitesh
authored and
ymahajan
committed
Snap 1890 : Snappy Pulse UI suggestions for 1.0 (#69)

snappy-sachin
authored and
ymahajan
committed
[SNAP-1067] Optimizations seen in perf analysis related to SnappyData PR#381 (#11)

Sumedh Wale
authored and
ymahajan
committed
[SNAP-1067] Optimizations seen in perf analysis related to SnappyData PR#381 (#11)

Sumedh Wale
authored and
ymahajan
committed

Commits on Mar 5, 2018

[SNAP-1136] Kryo closure serialtization support and optimizations (#27)

Sumedh Wale
authored and
ymahajan
committed
[SNAP-1190] Reduce partition message overhead from driver to executor (#31)

Sumedh Wale
authored and
ymahajan
committed
[SNAP-1202] Reduce serialization overheads of biggest contributors in queries (#34)

Sumedh Wale
authored and
ymahajan
committed

Commits on Mar 6, 2018

[SNAPPYDATA] Code changes for displaying product version details. (#38)

snappy-sachin
authored and
ymahajan
committed
[SNAPPYDATA] Update to gradle-scalatest version 0.13.1

Amogh Shetkar
authored and
ymahajan
committed
[SNAPPYDATA] Skip cast if non-nullable type is being inserted in nullable target

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] optimized versions for a couple of string functions

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] Increasing the code generation cache eviction size to 300 from 100

Sumedh Wale
authored and
ymahajan
committed
[SNAP-1398] Update janino version to latest 3.0.x

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] made some methods protected to be used by SnappyUnifiedManager (#47)

rishitesh
authored and
ymahajan
committed
[SNAPPYDATA] Reducing file read/write buffer sizes

Sumedh Wale
authored and
ymahajan
committed
[SNAP-1486] make QueryPlan.cleanArgs a transient lazy val (#51)

Sumedh Wale
authored and
ymahajan
committed
[SPARK-19500] [SQL] Fix off-by-one bug in BytesToBytesMap (#53)

rishitesh
authored and
ymahajan
committed
SNAP-1545: Snappy Dashboard UI Revamping (#52)

snappy-sachin
authored and
ymahajan
committed
[SNAPPYDATA] fixing scalastyle errors introduced in previous commits

Sumedh Wale
authored and
ymahajan
committed
SNAP-1698: Snappy Dashboard UI Enhancements (#55)

snappy-sachin
authored and
ymahajan
committed
[SNAPPYDATA] reduce a byte copy reading from ColumnVector

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] moved UTF8String.fromBuffer to Utils.stringFromBuffer

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] handle "prepare" in answer comparison inside Map types too

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] reverting changes to increase DECIMAL precision to 127

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA][MERGE-2.1] Some fixes after the merge
ymahajan
committed
[SNAPPYDATA][MERGE-2.1]
ymahajan
committed
[SNAPPYDATA][MERGE-2.1]
ymahajan
committed
[SNAPPYDATA][MERGE-2.1]
ymahajan
committed
[SNAPPYDATA][MERGE-2.1]
ymahajan
committed
[SNAP-1790] Fix one case of incorrect offset in ByteArrayMethods.arrayEquals

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA][MERGE-2.1] Missing patches and version changes

Sumedh Wale
authored and
ymahajan
committed
[SNAP-1389] Optimized UTF8String.compareTo (#62)

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA][PERF] optimized pattern matching for byte/time strings

Sumedh Wale
authored and
ymahajan
committed
SNAP-1792: Display snappy members logs on Snappy Pulse UI (#58)

snappy-sachin
authored and
ymahajan
committed
SNAP-1744: UI itself needs to consistently refer to itself as "SnappyData Pulse" (#64)

snappy-sachin
authored and
ymahajan
committed
[SNAP-1377,SNAP-902] Proper handling of exception in case of Lead and Server HA (#65)

SachinJanani
authored and
ymahajan
committed
Snap 1833 (#67)

rishitesh
authored and
ymahajan
committed
Refactored the executor exception handling for cache (#71)

rishitesh
authored and
ymahajan
committed
[SNAP-1930] Rectified a code in WholeStageCodeGenRdd. (#73)

rishitesh
authored and
ymahajan
committed
Snap 1813 : Security - Add Server (Jetty web server) level user authentication for Web UI in SnappyData. (#72)

snappy-sachin
authored and
ymahajan
committed
[SNAPPYDATA] fixing scalastyle failure introduced by last commit

Sumedh Wale
authored and
ymahajan
committed
Resized company logo (#74)

snappy-sachin
authored and
ymahajan
committed
[SNAPPYDATA] update janino to latest release 3.0.7

Sumedh Wale
authored and
ymahajan
committed

Commits on Mar 7, 2018

[SNAP-1951] move authentication handler bind to be inside connect (#75)

Sumedh Wale
authored and
ymahajan
committed
Bump version spark 2.1.1.1-rc1, store 1.5.6-rc1 and sparkJobserver 0.6.2.6-rc1

Amogh Shetkar
authored and
ymahajan
committed
Updated the year in the Snappydata copyright header. (#76)

kneeraj
authored and
ymahajan
committed
[SNAPPYDATA] upgrade netty versions (SPARK-18971, SPARK-18586)

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] more efficient passing of non-primitive literals

Sumedh Wale
authored and
ymahajan
committed
[SNAP-1993] Optimize UTF8String.contains (#78)

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA][AQP-293] Native JNI callback changes for UTF8String (#80)

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] update jetty version

Sumedh Wale
authored and
ymahajan
committed
[SNAP-2033] pass the original number of buckets in table via OrderlessHashPartitioning (#82)

Sumedh Wale
authored and
ymahajan
committed
Update versions for snappydata 1.0.0, store 1.6.0, spark 2.1.1.1 and spark-jobserver 0.6.2.6

Amogh Shetkar
authored and
ymahajan
committed
[SNAPPYDATA] use common "vendorName" in build scripts

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] relax access-level of Executor thread pools to protected

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] version upgrades as per previous cherry-picks

Sumedh Wale
authored and
ymahajan
committed
Snap 2044 (#85)

rishitesh
authored and
ymahajan
committed
Snap 2061 (#83)

ahshahid
authored and
ymahajan
committed
[SNAPPYDATA] build changes/fixes (#81)

Sumedh Wale
authored and
ymahajan
committed
[SNAP-2061] fix scalastyle errors, add test

Sumedh Wale
authored and
ymahajan
committed

Commits on Mar 8, 2018

[SNAPPYDATA] add missing jersey-hk2 dependency

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA][SNAP-2120] make codegen cache size configurable (#87)

Sumedh Wale
authored and
ymahajan
committed
Snap 2084 (#86)

rishitesh
authored and
ymahajan
committed
[SNAPPYDATA] some optimizations to ExecutionMemoryPool

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] fixing all failures in snappy-spark test suite

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] fixing one remaining failure in gradle runs

Sumedh Wale
authored and
ymahajan
committed
Preserve the preferred location in MapPartitionRDD. (#92)

rishitesh
authored and
ymahajan
committed
* SnappyData Spark Version 2.1.1.2

Amogh Shetkar
authored and
ymahajan
committed
[SNAP-2218] honour timeout in netty RPC transfers (#93)

Sumedh Wale
authored and
ymahajan
committed

Commits on Mar 12, 2018

Adding commons crypto libraries
ymahajan
committed
compilation issues
ymahajan
committed

Commits on Mar 13, 2018

fixing compilation errors
ymahajan
committed

Commits on Mar 16, 2018

compilation errors
ymahajan
committed

Commits on Mar 19, 2018

compilation issues
ymahajan
committed
compilation issues
ymahajan
committed

Commits on Mar 21, 2018

Added new build.gradle for spark-kvstore subproject
ymahajan
committed

Commits on Mar 29, 2018

Addressing precheckin failures
ymahajan
committed

Commits on Apr 2, 2018

Addressing precheckin issues
ymahajan
committed

Commits on Apr 6, 2018

Addressing precheckin failures
ymahajan
committed

Commits on Apr 9, 2018

Addressing precheckin failures
ymahajan
committed

Commits on Apr 23, 2018

Addressing prechekin failures
ymahajan
committed

Commits on Apr 30, 2018

Addressing precheckin failures
ymahajan
committed

Commits on May 2, 2018

Addressing precheckin failures
ymahajan
committed

Commits on May 4, 2018

Addressing precheckin failures
ymahajan
committed

Commits on May 8, 2018

Addressing precheckin failures
ymahajan
committed

Commits on May 10, 2018

[SNAPPYDATA] revert changes in Logging to upstream

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] Changed TestSparkSession in test class APIs to base SparkSession

hemanthmeka
authored and
ymahajan
committed
[SNAPPYDATA] increased default codegen cache size to 2K

Sumedh Wale
authored and
ymahajan
committed
[SNAPPYDATA] make Dataset.boundEnc as lazy val

Sumedh Wale
authored and
ymahajan
committed
[SNAP-2242] Unique application names & kill app by names (#98)

rishitesh
authored and
ymahajan
committed
[SNAP-2225] Removed OrderlessHashPartitioning. (#95)

rishitesh
authored and
ymahajan
committed

Commits on May 12, 2018

Fixing issues after master down merge
ymahajan
committed