Release

1.18.0 (2024-05-28)

Snowpark pandas API Updates

New Features

• Added DataFrame.cache_result and Series.cache_result methods for users to persist DataFrames and Series to a temporary table lasting the duration of the session to improve latency of subsequent operations.

Improvements

• Added partial support for DataFrame.pivot_table with no index parameter, as well as for margins parameter.
• Updated the signature of DataFrame.shift/Series.shift/DataFrameGroupBy.shift/SeriesGroupBy.shift to match pandas 2.2.1. Snowpark pandas does not yet support the newly-added suffix argument, or sequence values of periods.
• Re-added support for Series.str.split.

Bug Fixes

• Fixed how we support mixed columns for string methods (Series.str.*).

Snowpark Local Testing Updates

New Features

• Added support for the following DataFrameReader read options to file formats csv and json:
  • PURGE
  • PATTERN
  • INFER_SCHEMA with value being False
  • ENCODING with value being UTF8
• Added support for DataFrame.analytics.moving_agg and DataFrame.analytics.cumulative_agg_agg.
• Added support for if_not_exists parameter during UDF and stored procedure registration.

Bug Fixes

• Fixed a bug that when processing time format, fractional second part is not handled properly.
• Fixed a bug that caused function calls on * to fail.
• Fixed a bug that prevented creation of map and struct type objects.
• Fixed a bug that function date_add was unable to handle some numeric types.
• Fixed a bug that TimestampType casting resulted in incorrect data.
• Fixed a bug that caused DecimalType data to have incorrect precision in some cases.
• Fixed a bug where referencing missing table or view raises confusing IndexError.
• Fixed a bug that mocked function to_timestamp_ntz can not handle None data.
• Fixed a bug that mocked UDFs handles output data of None improperly.
• Fixed a bug where DataFrame.with_column_renamed ignores attributes from parent DataFrames after join operations.
• Fixed a bug that integer precision of large value gets lost when converted to pandas DataFrame.
• Fixed a bug that the schema of datetime object is wrong when create DataFrame from a pandas DataFrame.
• Fixed a bug in the implementation of Column.equal_nan where null data is handled incorrectly.
• Fixed a bug where DataFrame.drop ignore attributes from parent DataFrames after join operations.
• Fixed a bug in mocked function date_part where Column type is set wrong.
• Fixed a bug where DataFrameWriter.save_as_table does not raise exceptions when inserting null data into non-nullable columns.
• Fixed a bug in the implementation of DataFrameWriter.save_as_table where
  • Append or Truncate fails when incoming data has different schema than existing table.
  • Truncate fails when incoming data does not specify columns that are nullable.

Improvements

• Removed dependency check for pyarrow as it is not used.
• Improved target type coverage of Column.cast, adding support for casting to boolean and all integral types.
• Aligned error experience when calling UDFs and stored procedures.
• Added appropriate error messages for is_permanent and anonymous options in UDFs and stored procedures registration to make it more clear that those features are not yet supported.
• File read operation with unsupported options and values now raises NotImplementedError instead of warnings and unclear error information.

Release

1.17.0 (2024-05-21)

Snowpark Python API Updates

New Features

• Added support to add a comment on tables and views using the functions listed below:
  • DataFrameWriter.save_as_table
  • DataFrame.create_or_replace_view
  • DataFrame.create_or_replace_temp_view
  • DataFrame.create_or_replace_dynamic_table

Improvements

• Improved error message to remind users to set {"infer_schema": True} when reading CSV file without specifying its schema.

Snowpark pandas API Updates

New Features

• Start of Public Preview of Snowpark pandas API. Refer to the Snowpark pandas API Docs for more details.

Snowpark Local Testing Updates

New Features

• Added support for NumericType and VariantType data conversion in the mocked function to_timestamp_ltz, to_timestamp_ntz, to_timestamp_tz and to_timestamp.
• Added support for DecimalType, BinaryType, ArrayType, MapType, TimestampType, DateType and TimeType data conversion in the mocked function to_char.
• Added support for the following APIs:
  • snowflake.snowpark.functions:
    • to_varchar
  • snowflake.snowpark.DataFrame:
    • pivot
  • snowflake.snowpark.Session:
    • cancel_all
• Introduced a new exception class snowflake.snowpark.mock.exceptions.SnowparkLocalTestingException.
• Added support for casting to FloatType

Bug Fixes

• Fixed a bug that stored procedure and UDF should not remove imports already in the sys.path during the clean-up step.
• Fixed a bug that when processing datetime format, the fractional second part is not handled properly.
• Fixed a bug that on Windows platform that file operations was unable to properly handle file separator in directory name.
• Fixed a bug that on Windows platform that when reading a pandas dataframe, IntervalType column with integer data can not be processed.
• Fixed a bug that prevented users from being able to select multiple columns with the same alias.
• Fixed a bug that Session.get_current_[schema|database|role|user|account|warehouse] returns upper-cased identifiers when identifiers are quoted.
• Fixed a bug that function substr and substring can not handle 0-based start_expr.

Improvements

• Standardized the error experience by raising SnowparkLocalTestingException in error cases which is on par with SnowparkSQLException raised in non-local execution.
• Improved error experience of Session.write_pandas method that NotImplementError will be raised when called.
• Aligned error experience with reusing a closed session in non-local execution.

Release

1.16.0 (2024-05-07)

New Features

• Added snowflake.snowpark.Session.lineage.trace to explore data lineage of Snowflake objects.
• Support stored procedure registration with packages given as Python modules.
• Added support for structured type schema parsing.

Bug Fixes

• Fixed a bug that when inferring a schema, single quotes are added to stage files that already have single quotes.

Local Testing Updates

New Features

• Added support for StringType, TimestampType and VariantType data conversion in the mocked function to_date.
• Added support for the following APIs:
  • snowflake.snowpark.functions
    • get
    • concat
    • concat_ws

Bug Fixes

• Fixed a bug that caused NaT and NaN values to not be recognized.
• Fixed a bug when inferring schema, single quotes are added to stage files already have single quotes.
• Fixed a bug where DataFrameReader.csv was unable to handle quoted values containing a delimiter.
• Fixed a bug that when there is None value in an arithmetic calculation, the output should remain None instead of math.nan.
• Fixed a bug in function sum and covar_pop that when there is math.nan in the data, the output should also be math.nan.
• Fixed a bug that stage operation can not handle directories.
• Fixed a bug that DataFrame.to_pandas should take Snowflake numeric types with precision 38 as int64.

Release

1.15.0 (2024-04-24)

New Features

• Added truncate save mode in DataFrameWrite to overwrite existing tables by truncating the underlying table instead of dropping it.
• Added telemetry to calculate query plan height and number of duplicate nodes during collect operations.
• Added the functions below to unload data from a DataFrame into one or more files in a stage:
  • DataFrame.write.json
  • DataFrame.write.csv
  • DataFrame.write.parquet
• Added distributed tracing using open telemetry APIs for action functions in DataFrame and DataFrameWriter:
  • snowflake.snowpark.DataFrame:
    • collect
    • collect_nowait
    • to_pandas
    • count
    • show
  • snowflake.snowpark.DataFrameWriter:
    • save_as_table
• Added support for snow:// URLs to snowflake.snowpark.Session.file.get and snowflake.snowpark.Session.file.get_stream
• Added support to register stored procedures and UDxFs with a comment.
• UDAF client support is ready for public preview. Please stay tuned for the Snowflake announcement of UDAF public preview.
• Added support for dynamic pivot. This feature is currently in private preview.

Improvements

• Improved the generated query performance for both compilation and execution by converting duplicate subqueries to Common Table Expressions (CTEs). It is still an experimental feature not enabled by default, and can be enabled by setting session.cte_optimization_enabled to True.

Bug Fixes

• Fixed a bug where statement_params was not passed to query executions that register stored procedures and user defined functions.
• Fixed a bug causing snowflake.snowpark.Session.file.get_stream to fail for quoted stage locations.
• Fixed a bug that an internal type hint in utils.py might raise AttributeError in case the underlying module can not be found.

Local Testing Updates

New Features

• Added support for registering UDFs and stored procedures.
• Added support for the following APIs:
  • snowflake.snowpark.Session:
    • file.put
    • file.put_stream
    • file.get
    • file.get_stream
    • read.json
    • add_import
    • remove_import
    • get_imports
    • clear_imports
    • add_packages
    • add_requirements
    • clear_packages
    • remove_package
    • udf.register
    • udf.register_from_file
    • sproc.register
    • sproc.register_from_file
  • snowflake.snowpark.functions
    • current_database
    • current_session
    • date_trunc
    • object_construct
    • object_construct_keep_null
    • pow
    • sqrt
    • udf
    • sproc
• Added support for StringType, TimestampType and VariantType data conversion in the mocked function to_time.

Bug Fixes

• Fixed a bug that null filled columns for constant functions.
• Fixed a bug that implementation of to_object, to_array and to_binary to better handle null inputs.
• Fixed a bug that timestamp data comparison can not handle year beyond 2262.
• Fixed a bug that Session.builder.getOrCreate should return the created mock session.

Release

1.14.0 (2024-03-20)

New Features

• Added support for creating vectorized UDTFs with process method.
• Added support for dataframe functions:
  • to_timestamp_ltz
  • to_timestamp_ntz
  • to_timestamp_tz
  • locate
• Added support for ASOF JOIN type.
• Added support for the following local testing APIs:
  • snowflake.snowpark.functions:
    • to_double
    • to_timestamp
    • to_timestamp_ltz
    • to_timestamp_ntz
    • to_timestamp_tz
    • greatest
    • least
    • convert_timezone
    • dateadd
    • date_part
  • snowflake.snowpark.Session:
    • get_current_account
    • get_current_warehouse
    • get_current_role
    • use_schema
    • use_warehouse
    • use_database
    • use_role

Bug Fixes

• Fixed a bug in SnowflakePlanBuilder that save_as_table does not filter column that name start with '$' and follow by number correctly.
• Fixed a bug that statement parameters may have no effect when resolving imports and packages.
• Fixed bugs in local testing:
  • LEFT ANTI and LEFT SEMI joins drop rows with null values.
  • DataFrameReader.csv incorrectly parses data when the optional parameter field_optionally_enclosed_by is specified.
  • Column.regexp only considers the first entry when pattern is a Column.
  • Table.update raises KeyError when updating null values in the rows.
  • VARIANT columns raise errors at DataFrame.collect.
  • count_distinct does not work correctly when counting.
  • Null values in integer columns raise TypeError.

Improvements

• Added telemetry to local testing.
• Improved the error message of DataFrameReader to raise FileNotFound error when reading a path that does not exist or when there are no files under the path.

Release

1.13.0 (2024-02-26)

New Features

• Added support for an optional date_part argument in function last_day
• SessionBuilder.app_name will set the query_tag after the session is created.
• Added support for the following local testing functions:
  • current_timestamp
  • current_date
  • current_time
  • strip_null_value
  • upper
  • lower
  • length
  • initcap

Improvements

• Added cleanup logic at interpreter shutdown to close all active sessions.

Bug Fixes

• Fixed a bug in DataFrame.to_local_iterator where the iterator could yield wrong results if another query is executed before the iterator finishes due to wrong isolation level. For details, please see #945.
• Fixed a bug that truncated table names in error messages while running a plan with local testing enabled.
• Fixed a bug that Session.range returns empty result when the range is large.

Release

1.12.1 (2024-02-08)

Improvements

• Use split_blocks=True by default during to_pandas conversion, for optimal memory allocation. This parameter is passed to pyarrow.Table.to_pandas, which enables PyArrow to split the memory allocation into smaller, more manageable blocks instead of allocating a single contiguous block. This results in better memory management when dealing with larger datasets.

Bug Fixes

• Fixed a bug in DataFrame.to_pandas that caused an error when evaluating on a Dataframe with an IntergerType column with null values.

v1.12.0

1.12.0 (2024-01-30)

New Features

• Exposed statement_params in StoredProcedure.__call__.
• Added two optional arguments to Session.add_import.
  • chunk_size: The number of bytes to hash per chunk of the uploaded files.
  • whole_file_hash: By default only the first chunk of the uploaded import is hashed to save time. When this is set to True each uploaded file is fully hashed instead.
• Added parameters external_access_integrations and secrets when creating a UDAF from Snowpark Python to allow integration with external access.
• Added a new method Session.append_query_tag. Allows an additional tag to be added to the current query tag by appending it as a comma separated value.
• Added a new method Session.update_query_tag. Allows updates to a JSON encoded dictionary query tag.
• SessionBuilder.getOrCreate will now attempt to replace the singleton it returns when token expiration has been detected.
• Added support for new functions in snowflake.snowpark.functions:
  • array_except
  • create_map
  • sign/signum
• Added the following functions to DataFrame.analytics:
  • Added the moving_agg function in DataFrame.analytics to enable moving aggregations like sums and averages with multiple window sizes.
  • Added the cummulative_agg function in DataFrame.analytics to enable moving aggregations like sums and averages with multiple window sizes.

Bug Fixes

• Fixed a bug in DataFrame.na.fill that caused Boolean values to erroneously override integer values.

• Fixed a bug in Session.create_dataframe where the Snowpark DataFrames created using pandas DataFrames were not inferring the type for timestamp columns correctly. The behavior is as follows:

  • Earlier timestamp columns without a timezone would be converted to nanosecond epochs and inferred as LongType(), but will now be correctly maintained as timestamp values and be inferred as TimestampType(TimestampTimeZone.NTZ).
  • Earlier timestamp columns with a timezone would be inferred as TimestampType(TimestampTimeZone.NTZ) and loose timezone information but will now be correctly inferred as TimestampType(TimestampTimeZone.LTZ) and timezone information is retained correctly.
  • Set session parameter PYTHON_SNOWPARK_USE_LOGICAL_TYPE_FOR_CREATE_DATAFRAME to revert back to old behavior. It is recommended that you update your code to align with correct behavior because the parameter will be removed in the future.

• Fixed a bug that DataFrame.to_pandas gets decimal type when scale is not 0, and creates an object dtype in pandas. Instead, we cast the value to a float64 type.

• Fixed bugs that wrongly flattened the generated SQL when one of the following happens:

  • DataFrame.filter() is called after DataFrame.sort().limit().
  • DataFrame.sort() or filter() is called on a DataFrame that already has a window function or sequence-dependent data generator column.
    For instance, df.select("a", seq1().alias("b")).select("a", "b").sort("a") won't flatten the sort clause anymore.
  • a window or sequence-dependent data generator column is used after DataFrame.limit(). For instance, df.limit(10).select(row_number().over()) won't flatten the limit and select in the generated SQL.

• Fixed a bug where aliasing a DataFrame column raised an error when the DataFame was copied from another DataFrame with an aliased column. For instance,

  df = df.select(col("a").alias("b"))
  df = copy(df)
  df.select(col("b").alias("c"))  # threw an error. Now it's fixed.

• Fixed a bug in Session.create_dataframe that the non-nullable field in a schema is not respected for boolean type. Note that this fix is only effective when the user has the privilege to create a temp table.

• Fixed a bug in SQL simplifier where non-select statements in session.sql dropped a SQL query when used with limit().

• Fixed a bug that raised an exception when session parameter ERROR_ON_NONDETERMINISTIC_UPDATE is true.

Behavior Changes (API Compatible)

• When parsing data types during a to_pandas operation, we rely on GS precision value to fix precision issues for large integer values. This may affect users where a column that was earlier returned as int8 gets returned as int64. Users can fix this by explicitly specifying precision values for their return column.
• Aligned behavior for Session.call in case of table stored procedures where running Session.call would not trigger stored procedure unless a collect() operation was performed.
• StoredProcedureRegistration will now automatically add snowflake-snowpark-python as a package dependency. The added dependency will be on the client's local version of the library and an error is thrown if the server cannot support that version.

Release

1.11.1 (2023-12-07)

Bug Fixes

• Fixed a bug that numpy should not be imported at the top level of mock module.

Release

1.11.0 (2023-12-05)

New Features

• Add the conn_error attribute to SnowflakeSQLException that stores the whole underlying exception from snowflake-connector-python.

• Added support for RelationalGroupedDataframe.pivot() to access pivot in the following pattern Dataframe.group_by(...).pivot(...).

• Added experimental feature: Local Testing Mode, which allows you to create and operate on Snowpark Python DataFrames locally without connecting to a Snowflake account. You can use the local testing framework to test your DataFrame operations locally, on your development machine or in a CI (continuous integration) pipeline, before deploying code changes to your account.

• Added support for arrays_to_object new functions in snowflake.snowpark.functions.

• Added support for the vector data type.

Dependency Updates

• Bumped cloudpickle dependency to work with cloudpickle==2.2.1
• Updated snowflake-connector-python to 3.4.0.

Bug Fixes

• DataFrame column names quoting check now supports newline characters.
• Fix a bug where a DataFrame generated by session.read.with_metadata creates inconsistent table when doing df.write.save_as_table.