SEA: Fetch Phase #650
Open
SEA: Fetch Phase #650
+3,074
−481
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* decouple session class from existing Connection ensure maintenance of current APIs of Connection while delegating responsibility Signed-off-by: varun-edachali-dbx <[email protected]> * add open property to Connection to ensure maintenance of existing API Signed-off-by: varun-edachali-dbx <[email protected]> * update unit tests to address ThriftBackend through session instead of through Connection Signed-off-by: varun-edachali-dbx <[email protected]> * chore: move session specific tests from test_client to test_session Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) as in CONTRIBUTING.md Signed-off-by: varun-edachali-dbx <[email protected]> * use connection open property instead of long chain through session Signed-off-by: varun-edachali-dbx <[email protected]> * trigger integration workflow Signed-off-by: varun-edachali-dbx <[email protected]> * fix: ensure open attribute of Connection never fails in case the openSession takes long, the initialisation of the session will not complete immediately. This could make the session attribute inaccessible. If the Connection is deleted in this time, the open() check will throw because the session attribute does not exist. Thus, we default to the Connection being closed in this case. This was not an issue before because open was a direct attribute of the Connection class. Caught in the integration tests. Signed-off-by: varun-edachali-dbx <[email protected]> * fix: de-complicate earlier connection open logic earlier, one of the integration tests was failing because 'session was not an attribute of Connection'. This is likely tied to a local configuration issue related to unittest that was causing an error in the test suite itself. The tests are now passing without checking for the session attribute. c676f9b Signed-off-by: varun-edachali-dbx <[email protected]> * Revert "fix: de-complicate earlier connection open logic" This reverts commit d6b1b19. Signed-off-by: varun-edachali-dbx <[email protected]> * [empty commit] attempt to trigger ci e2e workflow Signed-off-by: varun-edachali-dbx <[email protected]> * Update CODEOWNERS (#562) new codeowners Signed-off-by: varun-edachali-dbx <[email protected]> * Enhance Cursor close handling and context manager exception management to prevent server side resource leaks (#554) * Enhance Cursor close handling and context manager exception management * tests * fmt * Fix Cursor.close() to properly handle CursorAlreadyClosedError * Remove specific test message from Cursor.close() error handling * Improve error handling in connection and cursor context managers to ensure proper closure during exceptions, including KeyboardInterrupt. Add tests for nested cursor management and verify operation closure on server-side errors. * add * add Signed-off-by: varun-edachali-dbx <[email protected]> * PECOBLR-86 improve logging on python driver (#556) * PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <[email protected]> * PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <[email protected]> * fixed format Signed-off-by: Sai Shree Pradhan <[email protected]> * used lazy logging Signed-off-by: Sai Shree Pradhan <[email protected]> * changed debug to error logs Signed-off-by: Sai Shree Pradhan <[email protected]> * used lazy logging Signed-off-by: Sai Shree Pradhan <[email protected]> --------- Signed-off-by: Sai Shree Pradhan <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]> * Revert "Merge remote-tracking branch 'upstream/sea-migration' into decouple-session" This reverts commit dbb2ec5, reversing changes made to 7192f11. Signed-off-by: varun-edachali-dbx <[email protected]> * Reapply "Merge remote-tracking branch 'upstream/sea-migration' into decouple-session" This reverts commit bdb8381. Signed-off-by: varun-edachali-dbx <[email protected]> * fix: separate session opening logic from instantiation ensures correctness of self.session.open call in Connection Signed-off-by: varun-edachali-dbx <[email protected]> * fix: use is_open attribute to denote session availability Signed-off-by: varun-edachali-dbx <[email protected]> * fix: access thrift backend through session Signed-off-by: varun-edachali-dbx <[email protected]> * chore: use get_handle() instead of private session attribute in client Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * fix: remove accidentally removed assertions Signed-off-by: varun-edachali-dbx <[email protected]> --------- Signed-off-by: varun-edachali-dbx <[email protected]> Signed-off-by: Sai Shree Pradhan <[email protected]> Co-authored-by: Jothi Prakash <[email protected]> Co-authored-by: Madhav Sainanee <[email protected]> Co-authored-by: Sai Shree Pradhan <[email protected]>
NOTE: the `test_complex_types` e2e test was not working at the time of this merge. The test must be triggered when the test is back up and running as intended. * remove excess logs, assertions, instantiations large merge artifacts Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) + remove excess log (merge artifact) Signed-off-by: varun-edachali-dbx <[email protected]> * fix typing Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary check Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary replace call Signed-off-by: varun-edachali-dbx <[email protected]> * introduce __str__ methods for CommandId and SessionId Signed-off-by: varun-edachali-dbx <[email protected]> * docstrings for DatabricksClient interface Signed-off-by: varun-edachali-dbx <[email protected]> * stronger typing of Cursor and ExecuteResponse Signed-off-by: varun-edachali-dbx <[email protected]> * remove utility functions from backend interface, fix circular import Signed-off-by: varun-edachali-dbx <[email protected]> * rename info to properties Signed-off-by: varun-edachali-dbx <[email protected]> * newline for cleanliness Signed-off-by: varun-edachali-dbx <[email protected]> * fix circular import Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * to_hex_id -> get_hex_id Signed-off-by: varun-edachali-dbx <[email protected]> * better comment on protocol version getter Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * move guid to hex id to new utils module Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * move staging allowed local path to connection props Signed-off-by: varun-edachali-dbx <[email protected]> * add strong return type for execute_command Signed-off-by: varun-edachali-dbx <[email protected]> * skip auth, error handling in databricksclient interface Signed-off-by: varun-edachali-dbx <[email protected]> * chore: docstring + line width Signed-off-by: varun-edachali-dbx <[email protected]> * get_id -> get_guid Signed-off-by: varun-edachali-dbx <[email protected]> * chore: docstring Signed-off-by: varun-edachali-dbx <[email protected]> * fix: to_hex_id -> to_hex_guid Signed-off-by: varun-edachali-dbx <[email protected]> --------- Signed-off-by: varun-edachali-dbx <[email protected]>
…574) * ensure backend client returns a ResultSet type in backend tests Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * newline for cleanliness Signed-off-by: varun-edachali-dbx <[email protected]> * fix circular import Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * to_hex_id -> get_hex_id Signed-off-by: varun-edachali-dbx <[email protected]> * better comment on protocol version getter Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * stricter typing for cursor Signed-off-by: varun-edachali-dbx <[email protected]> * correct typing Signed-off-by: varun-edachali-dbx <[email protected]> * correct tests and merge artifacts Signed-off-by: varun-edachali-dbx <[email protected]> * remove accidentally modified workflow files remnants of old merge Signed-off-by: varun-edachali-dbx <[email protected]> * chore: remove accidentally modified workflow files Signed-off-by: varun-edachali-dbx <[email protected]> * add back accidentally removed docstrings Signed-off-by: varun-edachali-dbx <[email protected]> * clean up docstrings Signed-off-by: varun-edachali-dbx <[email protected]> * log hex Signed-off-by: varun-edachali-dbx <[email protected]> * remove unnecessary _replace call Signed-off-by: varun-edachali-dbx <[email protected]> * add __str__ for CommandId Signed-off-by: varun-edachali-dbx <[email protected]> * take TOpenSessionResp in get_protocol_version to maintain existing interface Signed-off-by: varun-edachali-dbx <[email protected]> * active_op_handle -> active_mmand_id Signed-off-by: varun-edachali-dbx <[email protected]> * ensure None returned for close_command Signed-off-by: varun-edachali-dbx <[email protected]> * account for ResultSet return in new pydocs Signed-off-by: varun-edachali-dbx <[email protected]> * pydoc for types Signed-off-by: varun-edachali-dbx <[email protected]> * move common state to ResultSet aprent Signed-off-by: varun-edachali-dbx <[email protected]> * stronger typing in resultSet behaviour Signed-off-by: varun-edachali-dbx <[email protected]> * remove redundant patch in test Signed-off-by: varun-edachali-dbx <[email protected]> * add has_been_closed_server_side assertion Signed-off-by: varun-edachali-dbx <[email protected]> * remove redundancies in tests Signed-off-by: varun-edachali-dbx <[email protected]> * more robust close check Signed-off-by: varun-edachali-dbx <[email protected]> * use normalised state in e2e test Signed-off-by: varun-edachali-dbx <[email protected]> * simplify corrected test Signed-off-by: varun-edachali-dbx <[email protected]> * add line gaps after multi-line pydocs for consistency Signed-off-by: varun-edachali-dbx <[email protected]> * use normalised CommandState type in ExecuteResponse Signed-off-by: varun-edachali-dbx <[email protected]> --------- Signed-off-by: varun-edachali-dbx <[email protected]>
* introduce http client (temp) and sea test file Signed-off-by: varun-edachali-dbx <[email protected]> * reduce verbosity Signed-off-by: varun-edachali-dbx <[email protected]> * redundant comment Signed-off-by: varun-edachali-dbx <[email protected]> * reduce redundancy, params and data separate Signed-off-by: varun-edachali-dbx <[email protected]> * rename client Signed-off-by: varun-edachali-dbx <[email protected]> * fix type issues Signed-off-by: varun-edachali-dbx <[email protected]> * reduce repetition in request calls Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary elifs Signed-off-by: varun-edachali-dbx <[email protected]> * add newline at EOF Signed-off-by: varun-edachali-dbx <[email protected]> --------- Signed-off-by: varun-edachali-dbx <[email protected]>
* [squashed from prev branch] introduce sea client with session open and close functionality Signed-off-by: varun-edachali-dbx <[email protected]> * remove accidental changes to workflows (merge artifacts) Signed-off-by: varun-edachali-dbx <[email protected]> * pass test_input to get_protocol_version instead of session_id to maintain previous API Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black + line gaps after multi-line pydocs) Signed-off-by: varun-edachali-dbx <[email protected]> * use factory for backend instantiation Signed-off-by: varun-edachali-dbx <[email protected]> * fix type issues Signed-off-by: varun-edachali-dbx <[email protected]> * remove redundant comments Signed-off-by: varun-edachali-dbx <[email protected]> * introduce models for requests and responses Signed-off-by: varun-edachali-dbx <[email protected]> * remove http client and test script to prevent diff from showing up post http-client merge Signed-off-by: varun-edachali-dbx <[email protected]> * Introduce Sea HTTP Client and test script (#583) * introduce http client (temp) and sea test file Signed-off-by: varun-edachali-dbx <[email protected]> * reduce verbosity Signed-off-by: varun-edachali-dbx <[email protected]> * redundant comment Signed-off-by: varun-edachali-dbx <[email protected]> * reduce redundancy, params and data separate Signed-off-by: varun-edachali-dbx <[email protected]> * rename client Signed-off-by: varun-edachali-dbx <[email protected]> * fix type issues Signed-off-by: varun-edachali-dbx <[email protected]> * reduce repetition in request calls Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary elifs Signed-off-by: varun-edachali-dbx <[email protected]> * add newline at EOF Signed-off-by: varun-edachali-dbx <[email protected]> --------- Signed-off-by: varun-edachali-dbx <[email protected]> * CustomHttpClient -> SeaHttpClient Signed-off-by: varun-edachali-dbx <[email protected]> * redundant comment in backend client Signed-off-by: varun-edachali-dbx <[email protected]> * regex for warehouse_id instead of .split, remove excess imports and behaviour Signed-off-by: varun-edachali-dbx <[email protected]> * remove redundant attributes Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * [nit] reduce nested code Signed-off-by: varun-edachali-dbx <[email protected]> * line gap after multi-line pydoc Signed-off-by: varun-edachali-dbx <[email protected]> * redundant imports Signed-off-by: varun-edachali-dbx <[email protected]> * move sea backend and models into separate sea/ dir Signed-off-by: varun-edachali-dbx <[email protected]> * move http client into separate sea/ dir Signed-off-by: varun-edachali-dbx <[email protected]> * change commands to include ones in docs Signed-off-by: varun-edachali-dbx <[email protected]> * add link to sql-ref-parameters for session-confs Signed-off-by: varun-edachali-dbx <[email protected]> * add client side filtering for session confs, add note on warehouses over endoints Signed-off-by: varun-edachali-dbx <[email protected]> * test unimplemented methods and max_download_threads prop Signed-off-by: varun-edachali-dbx <[email protected]> --------- Signed-off-by: varun-edachali-dbx <[email protected]>
* [squash from exec-sea] bring over execution phase changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess test Signed-off-by: varun-edachali-dbx <[email protected]> * add docstring Signed-off-by: varun-edachali-dbx <[email protected]> * remvoe exec func in sea backend Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess files Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess models Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess sea backend tests Signed-off-by: varun-edachali-dbx <[email protected]> * cleanup Signed-off-by: varun-edachali-dbx <[email protected]> * re-introduce get_schema_desc Signed-off-by: varun-edachali-dbx <[email protected]> * remove SeaResultSet Signed-off-by: varun-edachali-dbx <[email protected]> * clean imports and attributes Signed-off-by: varun-edachali-dbx <[email protected]> * pass CommandId to ExecResp Signed-off-by: varun-edachali-dbx <[email protected]> * remove changes in types Signed-off-by: varun-edachali-dbx <[email protected]> * add back essential types (ExecResponse, from_sea_state) Signed-off-by: varun-edachali-dbx <[email protected]> * fix fetch types Signed-off-by: varun-edachali-dbx <[email protected]> * excess imports Signed-off-by: varun-edachali-dbx <[email protected]> * reduce diff by maintaining logs Signed-off-by: varun-edachali-dbx <[email protected]> * fix int test types Signed-off-by: varun-edachali-dbx <[email protected]> * move guid_to_hex_id import to utils Signed-off-by: varun-edachali-dbx <[email protected]> * reduce diff in guid utils import Signed-off-by: varun-edachali-dbx <[email protected]> * move arrow_schema_bytes back into ExecuteResult Signed-off-by: varun-edachali-dbx <[email protected]> * maintain log Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary assignment Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary tuple response Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-ncessary verbose mocking Signed-off-by: varun-edachali-dbx <[email protected]> * move Queue construction to ResultSert Signed-off-by: varun-edachali-dbx <[email protected]> * move description to List[Tuple] Signed-off-by: varun-edachali-dbx <[email protected]> * frmatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * reduce diff (remove explicit tuple conversion) Signed-off-by: varun-edachali-dbx <[email protected]> * remove has_more_rows from ExecuteResponse Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary has_more_rows aclc Signed-off-by: varun-edachali-dbx <[email protected]> * default has_more_rows to True Signed-off-by: varun-edachali-dbx <[email protected]> * return has_more_rows from ExecResponse conversion during GetRespMetadata Signed-off-by: varun-edachali-dbx <[email protected]> * remove unnecessary replacement Signed-off-by: varun-edachali-dbx <[email protected]> * better mocked backend naming Signed-off-by: varun-edachali-dbx <[email protected]> * remove has_more_rows test in ExecuteResponse Signed-off-by: varun-edachali-dbx <[email protected]> * introduce replacement of original has_more_rows read test Signed-off-by: varun-edachali-dbx <[email protected]> * call correct method in test_use_arrow_schema Signed-off-by: varun-edachali-dbx <[email protected]> * call correct method in test_fall_back_to_hive_schema Signed-off-by: varun-edachali-dbx <[email protected]> * re-introduce result response read test Signed-off-by: varun-edachali-dbx <[email protected]> * simplify test Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess fetch_results mocks Signed-off-by: varun-edachali-dbx <[email protected]> * more minimal changes to thrift_backend tests Signed-off-by: varun-edachali-dbx <[email protected]> * move back to old table types Signed-off-by: varun-edachali-dbx <[email protected]> * remove outdated arrow_schema_bytes return Signed-off-by: varun-edachali-dbx <[email protected]> * remove duplicate import Signed-off-by: varun-edachali-dbx <[email protected]> * rephrase model docstrings to explicitly denote that they are representations and not used over the wire Signed-off-by: varun-edachali-dbx <[email protected]> * has_more_rows -> is_direct_results Signed-off-by: varun-edachali-dbx <[email protected]> * switch docstring format to align with Connection class Signed-off-by: varun-edachali-dbx <[email protected]> --------- Signed-off-by: varun-edachali-dbx <[email protected]>
* [squash from exec-sea] bring over execution phase changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess test Signed-off-by: varun-edachali-dbx <[email protected]> * add docstring Signed-off-by: varun-edachali-dbx <[email protected]> * remvoe exec func in sea backend Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess files Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess models Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess sea backend tests Signed-off-by: varun-edachali-dbx <[email protected]> * cleanup Signed-off-by: varun-edachali-dbx <[email protected]> * re-introduce get_schema_desc Signed-off-by: varun-edachali-dbx <[email protected]> * remove SeaResultSet Signed-off-by: varun-edachali-dbx <[email protected]> * clean imports and attributes Signed-off-by: varun-edachali-dbx <[email protected]> * pass CommandId to ExecResp Signed-off-by: varun-edachali-dbx <[email protected]> * remove changes in types Signed-off-by: varun-edachali-dbx <[email protected]> * add back essential types (ExecResponse, from_sea_state) Signed-off-by: varun-edachali-dbx <[email protected]> * fix fetch types Signed-off-by: varun-edachali-dbx <[email protected]> * excess imports Signed-off-by: varun-edachali-dbx <[email protected]> * reduce diff by maintaining logs Signed-off-by: varun-edachali-dbx <[email protected]> * fix int test types Signed-off-by: varun-edachali-dbx <[email protected]> * [squashed from exec-sea] init execution func Signed-off-by: varun-edachali-dbx <[email protected]> * remove irrelevant changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove ResultSetFilter functionality Signed-off-by: varun-edachali-dbx <[email protected]> * remove more irrelevant changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove more irrelevant changes Signed-off-by: varun-edachali-dbx <[email protected]> * even more irrelevant changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove sea response as init option Signed-off-by: varun-edachali-dbx <[email protected]> * exec test example scripts Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * [squashed from sea-exec] merge sea stuffs Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess removed docstring Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess changes in backend Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess imports Signed-off-by: varun-edachali-dbx <[email protected]> * remove accidentally removed _get_schema_desc Signed-off-by: varun-edachali-dbx <[email protected]> * remove unnecessary init with sea_response tests Signed-off-by: varun-edachali-dbx <[email protected]> * rmeove unnecessary changes Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * improved models and filters from cloudfetch-sea branch Signed-off-by: varun-edachali-dbx <[email protected]> * filters stuff (align with JDBC) Signed-off-by: varun-edachali-dbx <[email protected]> * backend from cloudfetch-sea Signed-off-by: varun-edachali-dbx <[email protected]> * remove filtering, metadata ops Signed-off-by: varun-edachali-dbx <[email protected]> * raise NotImplementedErrror for metadata ops Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary backend cahnges Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-needed GetChunksResponse only relevant in Fetch phase Signed-off-by: varun-edachali-dbx <[email protected]> * reduce code duplication Signed-off-by: varun-edachali-dbx <[email protected]> * more clear docstrings Signed-off-by: varun-edachali-dbx <[email protected]> * introduce strongly typed ChunkInfo Signed-off-by: varun-edachali-dbx <[email protected]> * remove is_volume_operation from response Signed-off-by: varun-edachali-dbx <[email protected]> * add is_volume_op and more ResultData fields Signed-off-by: varun-edachali-dbx <[email protected]> --------- Signed-off-by: varun-edachali-dbx <[email protected]>
* [squash from exec-sea] bring over execution phase changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess test Signed-off-by: varun-edachali-dbx <[email protected]> * add docstring Signed-off-by: varun-edachali-dbx <[email protected]> * remvoe exec func in sea backend Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess files Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess models Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess sea backend tests Signed-off-by: varun-edachali-dbx <[email protected]> * cleanup Signed-off-by: varun-edachali-dbx <[email protected]> * re-introduce get_schema_desc Signed-off-by: varun-edachali-dbx <[email protected]> * remove SeaResultSet Signed-off-by: varun-edachali-dbx <[email protected]> * clean imports and attributes Signed-off-by: varun-edachali-dbx <[email protected]> * pass CommandId to ExecResp Signed-off-by: varun-edachali-dbx <[email protected]> * remove changes in types Signed-off-by: varun-edachali-dbx <[email protected]> * add back essential types (ExecResponse, from_sea_state) Signed-off-by: varun-edachali-dbx <[email protected]> * fix fetch types Signed-off-by: varun-edachali-dbx <[email protected]> * excess imports Signed-off-by: varun-edachali-dbx <[email protected]> * reduce diff by maintaining logs Signed-off-by: varun-edachali-dbx <[email protected]> * fix int test types Signed-off-by: varun-edachali-dbx <[email protected]> * [squashed from exec-sea] init execution func Signed-off-by: varun-edachali-dbx <[email protected]> * remove irrelevant changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove ResultSetFilter functionality Signed-off-by: varun-edachali-dbx <[email protected]> * remove more irrelevant changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove more irrelevant changes Signed-off-by: varun-edachali-dbx <[email protected]> * even more irrelevant changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove sea response as init option Signed-off-by: varun-edachali-dbx <[email protected]> * move guid_to_hex_id import to utils Signed-off-by: varun-edachali-dbx <[email protected]> * reduce diff in guid utils import Signed-off-by: varun-edachali-dbx <[email protected]> * move arrow_schema_bytes back into ExecuteResult Signed-off-by: varun-edachali-dbx <[email protected]> * maintain log Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary assignment Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary tuple response Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-ncessary verbose mocking Signed-off-by: varun-edachali-dbx <[email protected]> * move Queue construction to ResultSert Signed-off-by: varun-edachali-dbx <[email protected]> * move description to List[Tuple] Signed-off-by: varun-edachali-dbx <[email protected]> * frmatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * reduce diff (remove explicit tuple conversion) Signed-off-by: varun-edachali-dbx <[email protected]> * remove has_more_rows from ExecuteResponse Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary has_more_rows aclc Signed-off-by: varun-edachali-dbx <[email protected]> * default has_more_rows to True Signed-off-by: varun-edachali-dbx <[email protected]> * return has_more_rows from ExecResponse conversion during GetRespMetadata Signed-off-by: varun-edachali-dbx <[email protected]> * remove unnecessary replacement Signed-off-by: varun-edachali-dbx <[email protected]> * better mocked backend naming Signed-off-by: varun-edachali-dbx <[email protected]> * remove has_more_rows test in ExecuteResponse Signed-off-by: varun-edachali-dbx <[email protected]> * introduce replacement of original has_more_rows read test Signed-off-by: varun-edachali-dbx <[email protected]> * call correct method in test_use_arrow_schema Signed-off-by: varun-edachali-dbx <[email protected]> * call correct method in test_fall_back_to_hive_schema Signed-off-by: varun-edachali-dbx <[email protected]> * re-introduce result response read test Signed-off-by: varun-edachali-dbx <[email protected]> * simplify test Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess fetch_results mocks Signed-off-by: varun-edachali-dbx <[email protected]> * more minimal changes to thrift_backend tests Signed-off-by: varun-edachali-dbx <[email protected]> * move back to old table types Signed-off-by: varun-edachali-dbx <[email protected]> * remove outdated arrow_schema_bytes return Signed-off-by: varun-edachali-dbx <[email protected]> * align SeaResultSet with new structure Signed-off-by: varun-edachali-dbx <[email protected]> * correct sea res set tests Signed-off-by: varun-edachali-dbx <[email protected]> * remove duplicate import Signed-off-by: varun-edachali-dbx <[email protected]> * rephrase model docstrings to explicitly denote that they are representations and not used over the wire Signed-off-by: varun-edachali-dbx <[email protected]> * has_more_rows -> is_direct_results Signed-off-by: varun-edachali-dbx <[email protected]> * switch docstring format to align with Connection class Signed-off-by: varun-edachali-dbx <[email protected]> * has_more_rows -> is_direct_results Signed-off-by: varun-edachali-dbx <[email protected]> * fix type errors with arrow_schema_bytes Signed-off-by: varun-edachali-dbx <[email protected]> * spaces after multi line pydocs Signed-off-by: varun-edachali-dbx <[email protected]> * remove duplicate queue init (merge artifact) Signed-off-by: varun-edachali-dbx <[email protected]> * reduce diff (remove newlines) Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary changes covered by #588 anyway Signed-off-by: varun-edachali-dbx <[email protected]> * Revert "remove un-necessary changes" This reverts commit a70a6ce. Signed-off-by: varun-edachali-dbx <[email protected]> * b"" -> None Signed-off-by: varun-edachali-dbx <[email protected]> --------- Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
* decouple session class from existing Connection ensure maintenance of current APIs of Connection while delegating responsibility Signed-off-by: varun-edachali-dbx <[email protected]> * add open property to Connection to ensure maintenance of existing API Signed-off-by: varun-edachali-dbx <[email protected]> * update unit tests to address ThriftBackend through session instead of through Connection Signed-off-by: varun-edachali-dbx <[email protected]> * chore: move session specific tests from test_client to test_session Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) as in CONTRIBUTING.md Signed-off-by: varun-edachali-dbx <[email protected]> * use connection open property instead of long chain through session Signed-off-by: varun-edachali-dbx <[email protected]> * trigger integration workflow Signed-off-by: varun-edachali-dbx <[email protected]> * fix: ensure open attribute of Connection never fails in case the openSession takes long, the initialisation of the session will not complete immediately. This could make the session attribute inaccessible. If the Connection is deleted in this time, the open() check will throw because the session attribute does not exist. Thus, we default to the Connection being closed in this case. This was not an issue before because open was a direct attribute of the Connection class. Caught in the integration tests. Signed-off-by: varun-edachali-dbx <[email protected]> * fix: de-complicate earlier connection open logic earlier, one of the integration tests was failing because 'session was not an attribute of Connection'. This is likely tied to a local configuration issue related to unittest that was causing an error in the test suite itself. The tests are now passing without checking for the session attribute. c676f9b Signed-off-by: varun-edachali-dbx <[email protected]> * Revert "fix: de-complicate earlier connection open logic" This reverts commit d6b1b19. Signed-off-by: varun-edachali-dbx <[email protected]> * [empty commit] attempt to trigger ci e2e workflow Signed-off-by: varun-edachali-dbx <[email protected]> * Update CODEOWNERS (#562) new codeowners Signed-off-by: varun-edachali-dbx <[email protected]> * Enhance Cursor close handling and context manager exception management to prevent server side resource leaks (#554) * Enhance Cursor close handling and context manager exception management * tests * fmt * Fix Cursor.close() to properly handle CursorAlreadyClosedError * Remove specific test message from Cursor.close() error handling * Improve error handling in connection and cursor context managers to ensure proper closure during exceptions, including KeyboardInterrupt. Add tests for nested cursor management and verify operation closure on server-side errors. * add * add Signed-off-by: varun-edachali-dbx <[email protected]> * PECOBLR-86 improve logging on python driver (#556) * PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <[email protected]> * PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <[email protected]> * fixed format Signed-off-by: Sai Shree Pradhan <[email protected]> * used lazy logging Signed-off-by: Sai Shree Pradhan <[email protected]> * changed debug to error logs Signed-off-by: Sai Shree Pradhan <[email protected]> * used lazy logging Signed-off-by: Sai Shree Pradhan <[email protected]> --------- Signed-off-by: Sai Shree Pradhan <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]> * Revert "Merge remote-tracking branch 'upstream/sea-migration' into decouple-session" This reverts commit dbb2ec5, reversing changes made to 7192f11. Signed-off-by: varun-edachali-dbx <[email protected]> * Reapply "Merge remote-tracking branch 'upstream/sea-migration' into decouple-session" This reverts commit bdb8381. Signed-off-by: varun-edachali-dbx <[email protected]> * fix: separate session opening logic from instantiation ensures correctness of self.session.open call in Connection Signed-off-by: varun-edachali-dbx <[email protected]> * fix: use is_open attribute to denote session availability Signed-off-by: varun-edachali-dbx <[email protected]> * fix: access thrift backend through session Signed-off-by: varun-edachali-dbx <[email protected]> * chore: use get_handle() instead of private session attribute in client Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * fix: remove accidentally removed assertions Signed-off-by: varun-edachali-dbx <[email protected]> --------- Signed-off-by: varun-edachali-dbx <[email protected]> Signed-off-by: Sai Shree Pradhan <[email protected]> Co-authored-by: Jothi Prakash <[email protected]> Co-authored-by: Madhav Sainanee <[email protected]> Co-authored-by: Sai Shree Pradhan <[email protected]>
NOTE: the `test_complex_types` e2e test was not working at the time of this merge. The test must be triggered when the test is back up and running as intended. * remove excess logs, assertions, instantiations large merge artifacts Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) + remove excess log (merge artifact) Signed-off-by: varun-edachali-dbx <[email protected]> * fix typing Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary check Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary replace call Signed-off-by: varun-edachali-dbx <[email protected]> * introduce __str__ methods for CommandId and SessionId Signed-off-by: varun-edachali-dbx <[email protected]> * docstrings for DatabricksClient interface Signed-off-by: varun-edachali-dbx <[email protected]> * stronger typing of Cursor and ExecuteResponse Signed-off-by: varun-edachali-dbx <[email protected]> * remove utility functions from backend interface, fix circular import Signed-off-by: varun-edachali-dbx <[email protected]> * rename info to properties Signed-off-by: varun-edachali-dbx <[email protected]> * newline for cleanliness Signed-off-by: varun-edachali-dbx <[email protected]> * fix circular import Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * to_hex_id -> get_hex_id Signed-off-by: varun-edachali-dbx <[email protected]> * better comment on protocol version getter Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * move guid to hex id to new utils module Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * move staging allowed local path to connection props Signed-off-by: varun-edachali-dbx <[email protected]> * add strong return type for execute_command Signed-off-by: varun-edachali-dbx <[email protected]> * skip auth, error handling in databricksclient interface Signed-off-by: varun-edachali-dbx <[email protected]> * chore: docstring + line width Signed-off-by: varun-edachali-dbx <[email protected]> * get_id -> get_guid Signed-off-by: varun-edachali-dbx <[email protected]> * chore: docstring Signed-off-by: varun-edachali-dbx <[email protected]> * fix: to_hex_id -> to_hex_guid Signed-off-by: varun-edachali-dbx <[email protected]> --------- Signed-off-by: varun-edachali-dbx <[email protected]>
…574) * ensure backend client returns a ResultSet type in backend tests Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * newline for cleanliness Signed-off-by: varun-edachali-dbx <[email protected]> * fix circular import Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * to_hex_id -> get_hex_id Signed-off-by: varun-edachali-dbx <[email protected]> * better comment on protocol version getter Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * stricter typing for cursor Signed-off-by: varun-edachali-dbx <[email protected]> * correct typing Signed-off-by: varun-edachali-dbx <[email protected]> * correct tests and merge artifacts Signed-off-by: varun-edachali-dbx <[email protected]> * remove accidentally modified workflow files remnants of old merge Signed-off-by: varun-edachali-dbx <[email protected]> * chore: remove accidentally modified workflow files Signed-off-by: varun-edachali-dbx <[email protected]> * add back accidentally removed docstrings Signed-off-by: varun-edachali-dbx <[email protected]> * clean up docstrings Signed-off-by: varun-edachali-dbx <[email protected]> * log hex Signed-off-by: varun-edachali-dbx <[email protected]> * remove unnecessary _replace call Signed-off-by: varun-edachali-dbx <[email protected]> * add __str__ for CommandId Signed-off-by: varun-edachali-dbx <[email protected]> * take TOpenSessionResp in get_protocol_version to maintain existing interface Signed-off-by: varun-edachali-dbx <[email protected]> * active_op_handle -> active_mmand_id Signed-off-by: varun-edachali-dbx <[email protected]> * ensure None returned for close_command Signed-off-by: varun-edachali-dbx <[email protected]> * account for ResultSet return in new pydocs Signed-off-by: varun-edachali-dbx <[email protected]> * pydoc for types Signed-off-by: varun-edachali-dbx <[email protected]> * move common state to ResultSet aprent Signed-off-by: varun-edachali-dbx <[email protected]> * stronger typing in resultSet behaviour Signed-off-by: varun-edachali-dbx <[email protected]> * remove redundant patch in test Signed-off-by: varun-edachali-dbx <[email protected]> * add has_been_closed_server_side assertion Signed-off-by: varun-edachali-dbx <[email protected]> * remove redundancies in tests Signed-off-by: varun-edachali-dbx <[email protected]> * more robust close check Signed-off-by: varun-edachali-dbx <[email protected]> * use normalised state in e2e test Signed-off-by: varun-edachali-dbx <[email protected]> * simplify corrected test Signed-off-by: varun-edachali-dbx <[email protected]> * add line gaps after multi-line pydocs for consistency Signed-off-by: varun-edachali-dbx <[email protected]> * use normalised CommandState type in ExecuteResponse Signed-off-by: varun-edachali-dbx <[email protected]> --------- Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
* [squash from exec-sea] bring over execution phase changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess test Signed-off-by: varun-edachali-dbx <[email protected]> * add docstring Signed-off-by: varun-edachali-dbx <[email protected]> * remvoe exec func in sea backend Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess files Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess models Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess sea backend tests Signed-off-by: varun-edachali-dbx <[email protected]> * cleanup Signed-off-by: varun-edachali-dbx <[email protected]> * re-introduce get_schema_desc Signed-off-by: varun-edachali-dbx <[email protected]> * remove SeaResultSet Signed-off-by: varun-edachali-dbx <[email protected]> * clean imports and attributes Signed-off-by: varun-edachali-dbx <[email protected]> * pass CommandId to ExecResp Signed-off-by: varun-edachali-dbx <[email protected]> * remove changes in types Signed-off-by: varun-edachali-dbx <[email protected]> * add back essential types (ExecResponse, from_sea_state) Signed-off-by: varun-edachali-dbx <[email protected]> * fix fetch types Signed-off-by: varun-edachali-dbx <[email protected]> * excess imports Signed-off-by: varun-edachali-dbx <[email protected]> * reduce diff by maintaining logs Signed-off-by: varun-edachali-dbx <[email protected]> * fix int test types Signed-off-by: varun-edachali-dbx <[email protected]> * [squashed from exec-sea] init execution func Signed-off-by: varun-edachali-dbx <[email protected]> * remove irrelevant changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove ResultSetFilter functionality Signed-off-by: varun-edachali-dbx <[email protected]> * remove more irrelevant changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove more irrelevant changes Signed-off-by: varun-edachali-dbx <[email protected]> * even more irrelevant changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove sea response as init option Signed-off-by: varun-edachali-dbx <[email protected]> * exec test example scripts Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * [squashed from sea-exec] merge sea stuffs Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess removed docstring Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess changes in backend Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess imports Signed-off-by: varun-edachali-dbx <[email protected]> * remove accidentally removed _get_schema_desc Signed-off-by: varun-edachali-dbx <[email protected]> * remove unnecessary init with sea_response tests Signed-off-by: varun-edachali-dbx <[email protected]> * rmeove unnecessary changes Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * improved models and filters from cloudfetch-sea branch Signed-off-by: varun-edachali-dbx <[email protected]> * filters stuff (align with JDBC) Signed-off-by: varun-edachali-dbx <[email protected]> * backend from cloudfetch-sea Signed-off-by: varun-edachali-dbx <[email protected]> * remove filtering, metadata ops Signed-off-by: varun-edachali-dbx <[email protected]> * raise NotImplementedErrror for metadata ops Signed-off-by: varun-edachali-dbx <[email protected]> * change to valid table name Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary changes covered by #588 Signed-off-by: varun-edachali-dbx <[email protected]> * simplify test module Signed-off-by: varun-edachali-dbx <[email protected]> * logging -> debug level Signed-off-by: varun-edachali-dbx <[email protected]> * change table name in log Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary backend cahnges Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-needed GetChunksResponse Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-needed GetChunksResponse only relevant in Fetch phase Signed-off-by: varun-edachali-dbx <[email protected]> * reduce code duplication in response parsing Signed-off-by: varun-edachali-dbx <[email protected]> * reduce code duplication Signed-off-by: varun-edachali-dbx <[email protected]> * more clear docstrings Signed-off-by: varun-edachali-dbx <[email protected]> * introduce strongly typed ChunkInfo Signed-off-by: varun-edachali-dbx <[email protected]> * remove is_volume_operation from response Signed-off-by: varun-edachali-dbx <[email protected]> * add is_volume_op and more ResultData fields Signed-off-by: varun-edachali-dbx <[email protected]> * add test scripts Signed-off-by: varun-edachali-dbx <[email protected]> * Revert "Merge branch 'exec-models-sea' into exec-phase-sea" This reverts commit be1997e, reversing changes made to 37813ba. * change logging level Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove _get_schema_bytes (for now) Signed-off-by: varun-edachali-dbx <[email protected]> * redundant comments Signed-off-by: varun-edachali-dbx <[email protected]> * remove fetch phase methods Signed-off-by: varun-edachali-dbx <[email protected]> * reduce code repetititon + introduce gaps after multi line pydocs Signed-off-by: varun-edachali-dbx <[email protected]> * remove unused imports Signed-off-by: varun-edachali-dbx <[email protected]> * move description extraction to helper func Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * add more unit tests Signed-off-by: varun-edachali-dbx <[email protected]> * streamline unit tests Signed-off-by: varun-edachali-dbx <[email protected]> * test getting the list of allowed configurations Signed-off-by: varun-edachali-dbx <[email protected]> * reduce diff Signed-off-by: varun-edachali-dbx <[email protected]> * reduce diff Signed-off-by: varun-edachali-dbx <[email protected]> * house constants in enums for readability and immutability Signed-off-by: varun-edachali-dbx <[email protected]> * add note on hybrid disposition Signed-off-by: varun-edachali-dbx <[email protected]> * remove redundant note on arrow_schema_bytes Signed-off-by: varun-edachali-dbx <[email protected]> * remove invalid import Signed-off-by: varun-edachali-dbx <[email protected]> * add strong typing for manifest in _extract_description Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary column skipping Signed-off-by: varun-edachali-dbx <[email protected]> * remove parsing in backend Signed-off-by: varun-edachali-dbx <[email protected]> * fix: convert sea statement id to CommandId type Signed-off-by: varun-edachali-dbx <[email protected]> * make polling interval a separate constant Signed-off-by: varun-edachali-dbx <[email protected]> * align state checking with Thrift implementation Signed-off-by: varun-edachali-dbx <[email protected]> * update unit tests according to changes Signed-off-by: varun-edachali-dbx <[email protected]> * add unit tests for added methods Signed-off-by: varun-edachali-dbx <[email protected]> * add spec to description extraction docstring, add strong typing to params Signed-off-by: varun-edachali-dbx <[email protected]> * add strong typing for backend parameters arg Signed-off-by: varun-edachali-dbx <[email protected]> --------- Signed-off-by: varun-edachali-dbx <[email protected]>
…nd forward refs, remove some unused imports Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
…nd Cursor Signed-off-by: varun-edachali-dbx <[email protected]>
* test getting the list of allowed configurations Signed-off-by: varun-edachali-dbx <[email protected]> * reduce diff Signed-off-by: varun-edachali-dbx <[email protected]> * reduce diff Signed-off-by: varun-edachali-dbx <[email protected]> * house constants in enums for readability and immutability Signed-off-by: varun-edachali-dbx <[email protected]> * add note on hybrid disposition Signed-off-by: varun-edachali-dbx <[email protected]> * [squashed from cloudfetch-sea] introduce external links + arrow functionality Signed-off-by: varun-edachali-dbx <[email protected]> * reduce responsibility of Queue Signed-off-by: varun-edachali-dbx <[email protected]> * reduce repetition in arrow tablee creation Signed-off-by: varun-edachali-dbx <[email protected]> * reduce redundant code in CloudFetchQueue Signed-off-by: varun-edachali-dbx <[email protected]> * move chunk link progression to separate func Signed-off-by: varun-edachali-dbx <[email protected]> * remove redundant log Signed-off-by: varun-edachali-dbx <[email protected]> * improve logging Signed-off-by: varun-edachali-dbx <[email protected]> * remove reliance on schema_bytes in SEA Signed-off-by: varun-edachali-dbx <[email protected]> * remove redundant note on arrow_schema_bytes Signed-off-by: varun-edachali-dbx <[email protected]> * use more fetch methods Signed-off-by: varun-edachali-dbx <[email protected]> * remove redundant schema_bytes from parent constructor Signed-off-by: varun-edachali-dbx <[email protected]> * only call get_chunk_link with non null chunk index Signed-off-by: varun-edachali-dbx <[email protected]> * align SeaResultSet structure with ThriftResultSet Signed-off-by: varun-edachali-dbx <[email protected]> * remvoe _fill_result_buffer from SeaResultSet Signed-off-by: varun-edachali-dbx <[email protected]> * reduce code repetition Signed-off-by: varun-edachali-dbx <[email protected]> * align SeaResultSet with ext-links-sea Signed-off-by: varun-edachali-dbx <[email protected]> * remove redundant methods Signed-off-by: varun-edachali-dbx <[email protected]> * update unit tests Signed-off-by: varun-edachali-dbx <[email protected]> * remove accidental venv changes Signed-off-by: varun-edachali-dbx <[email protected]> * pre-fetch next chunk link on processing current Signed-off-by: varun-edachali-dbx <[email protected]> * reduce nesting Signed-off-by: varun-edachali-dbx <[email protected]> * line break after multi line pydoc Signed-off-by: varun-edachali-dbx <[email protected]> * re-introduce schema_bytes for better abstraction (likely temporary) Signed-off-by: varun-edachali-dbx <[email protected]> * add fetchmany_arrow and fetchall_arrow Signed-off-by: varun-edachali-dbx <[email protected]> * remove accidental changes in sea backend tests Signed-off-by: varun-edachali-dbx <[email protected]> * remove irrelevant changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary test changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary changes in thrift backend tests Signed-off-by: varun-edachali-dbx <[email protected]> * remove unimplemented methods test Signed-off-by: varun-edachali-dbx <[email protected]> * remove unimplemented method tests Signed-off-by: varun-edachali-dbx <[email protected]> * modify example scripts to include fetch calls Signed-off-by: varun-edachali-dbx <[email protected]> * add GetChunksResponse Signed-off-by: varun-edachali-dbx <[email protected]> * remove changes to sea test Signed-off-by: varun-edachali-dbx <[email protected]> * re-introduce accidentally removed description extraction method Signed-off-by: varun-edachali-dbx <[email protected]> * fix type errors (ssl_options, CHUNK_PATH_WITH_ID..., etc.) Signed-off-by: varun-edachali-dbx <[email protected]> * access ssl_options through connection Signed-off-by: varun-edachali-dbx <[email protected]> * DEBUG level Signed-off-by: varun-edachali-dbx <[email protected]> * remove explicit multi chunk test Signed-off-by: varun-edachali-dbx <[email protected]> * move cloud fetch queues back into utils.py Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess docstrings Signed-off-by: varun-edachali-dbx <[email protected]> * move ThriftCloudFetchQueue above SeaCloudFetchQueue Signed-off-by: varun-edachali-dbx <[email protected]> * fix sea connector tests Signed-off-by: varun-edachali-dbx <[email protected]> * correct patch module path in cloud fetch queue tests Signed-off-by: varun-edachali-dbx <[email protected]> * remove unimplemented methods test Signed-off-by: varun-edachali-dbx <[email protected]> * correct add_link docstring Signed-off-by: varun-edachali-dbx <[email protected]> * remove invalid import Signed-off-by: varun-edachali-dbx <[email protected]> * better align queries with JDBC impl Signed-off-by: varun-edachali-dbx <[email protected]> * line breaks after multi-line PRs Signed-off-by: varun-edachali-dbx <[email protected]> * remove unused imports Signed-off-by: varun-edachali-dbx <[email protected]> * fix: introduce ExecuteResponse import Signed-off-by: varun-edachali-dbx <[email protected]> * remove unimplemented metadata methods test, un-necessary imports Signed-off-by: varun-edachali-dbx <[email protected]> * introduce unit tests for metadata methods Signed-off-by: varun-edachali-dbx <[email protected]> * remove verbosity in ResultSetFilter docstring Co-authored-by: jayant <[email protected]> * remove un-necessary info in ResultSetFilter docstring Signed-off-by: varun-edachali-dbx <[email protected]> * remove explicit type checking, string literals around forward annotations Signed-off-by: varun-edachali-dbx <[email protected]> * house SQL commands in constants Signed-off-by: varun-edachali-dbx <[email protected]> * convert complex types to string if not _use_arrow_native_complex_types Signed-off-by: varun-edachali-dbx <[email protected]> * introduce unit tests for altered functionality Signed-off-by: varun-edachali-dbx <[email protected]> * Revert "Merge branch 'fetch-json-inline' into ext-links-sea" This reverts commit dabba55, reversing changes made to dd7dc6a. Signed-off-by: varun-edachali-dbx <[email protected]> * reduce verbosity of ResultSetFilter docstring Signed-off-by: varun-edachali-dbx <[email protected]> * remove unused imports Signed-off-by: varun-edachali-dbx <[email protected]> * Revert "Merge branch 'fetch-json-inline' into ext-links-sea" This reverts commit 3a999c0, reversing changes made to a1f9b9c. * Revert "reduce verbosity of ResultSetFilter docstring" This reverts commit a1f9b9c. * Reapply "Merge branch 'fetch-json-inline' into ext-links-sea" This reverts commit 48ad7b3. * Revert "Merge branch 'fetch-json-inline' into ext-links-sea" This reverts commit dabba55, reversing changes made to dd7dc6a. * remove un-necessary filters changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary backend changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove constants changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove changes in filters tests Signed-off-by: varun-edachali-dbx <[email protected]> * remove unit test backend and JSON queue changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove changes in sea result set testing Signed-off-by: varun-edachali-dbx <[email protected]> * Revert "remove changes in sea result set testing" This reverts commit d210ccd. * Revert "remove unit test backend and JSON queue changes" This reverts commit f6c5950. * Revert "remove changes in filters tests" This reverts commit f3f795a. * Revert "remove constants changes" This reverts commit 802d045. * Revert "remove un-necessary backend changes" This reverts commit 20822e4. * Revert "remove un-necessary filters changes" This reverts commit 5e75fb5. * remove unused imports Signed-off-by: varun-edachali-dbx <[email protected]> * working version Signed-off-by: varun-edachali-dbx <[email protected]> * adopy _wait_until_command_done Signed-off-by: varun-edachali-dbx <[email protected]> * introduce metadata commands Signed-off-by: varun-edachali-dbx <[email protected]> * use new backend structure Signed-off-by: varun-edachali-dbx <[email protected]> * constrain backend diff Signed-off-by: varun-edachali-dbx <[email protected]> * remove changes to filters Signed-off-by: varun-edachali-dbx <[email protected]> * make _parse methods in models internal Signed-off-by: varun-edachali-dbx <[email protected]> * reduce changes in unit tests Signed-off-by: varun-edachali-dbx <[email protected]> * run small queries with SEA during integration tests Signed-off-by: varun-edachali-dbx <[email protected]> * run some tests for sea Signed-off-by: varun-edachali-dbx <[email protected]> * allow empty schema bytes for alignment with SEA Signed-off-by: varun-edachali-dbx <[email protected]> * pass is_vl_op to Sea backend ExecuteResponse Signed-off-by: varun-edachali-dbx <[email protected]> * remove catalog requirement in get_tables Signed-off-by: varun-edachali-dbx <[email protected]> * move filters.py to SEA utils Signed-off-by: varun-edachali-dbx <[email protected]> * ensure SeaResultSet Signed-off-by: varun-edachali-dbx <[email protected]> * prevent circular imports Signed-off-by: varun-edachali-dbx <[email protected]> * remove unused imports Signed-off-by: varun-edachali-dbx <[email protected]> * remove cast, throw error if not SeaResultSet Signed-off-by: varun-edachali-dbx <[email protected]> * pass param as TSparkParameterValue Signed-off-by: varun-edachali-dbx <[email protected]> * remove failing test (temp) Signed-off-by: varun-edachali-dbx <[email protected]> * remove SeaResultSet type assertion Signed-off-by: varun-edachali-dbx <[email protected]> * change errors to align with spec, instead of arbitrary ValueError Signed-off-by: varun-edachali-dbx <[email protected]> * make SEA backend methods return SeaResultSet Signed-off-by: varun-edachali-dbx <[email protected]> * use spec-aligned Exceptions in SEA backend Signed-off-by: varun-edachali-dbx <[email protected]> * remove defensive row type check Signed-off-by: varun-edachali-dbx <[email protected]> * raise ProgrammingError for invalid id Signed-off-by: varun-edachali-dbx <[email protected]> * make is_volume_operation strict bool Signed-off-by: varun-edachali-dbx <[email protected]> * remove complex types code Signed-off-by: varun-edachali-dbx <[email protected]> * Revert "remove complex types code" This reverts commit 138359d. * introduce type conversion for primitive types for JSON + INLINE Signed-off-by: varun-edachali-dbx <[email protected]> * remove SEA running on metadata queries (known failures Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary docstrings Signed-off-by: varun-edachali-dbx <[email protected]> * align expected types with databricks sdk Signed-off-by: varun-edachali-dbx <[email protected]> * link rest api reference to validate types Signed-off-by: varun-edachali-dbx <[email protected]> * remove test_catalogs_returns_arrow_table test metadata commands not expected to pass Signed-off-by: varun-edachali-dbx <[email protected]> * fix fetchall_arrow and fetchmany_arrow Signed-off-by: varun-edachali-dbx <[email protected]> * remove thrift aligned test_cancel_during_execute from SEA tests Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary changes in example scripts Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary chagnes in example scripts Signed-off-by: varun-edachali-dbx <[email protected]> * _convert_json_table -> _create_json_table Signed-off-by: varun-edachali-dbx <[email protected]> * remove accidentally removed test Signed-off-by: varun-edachali-dbx <[email protected]> * remove new unit tests (to be re-added based on new arch) Signed-off-by: varun-edachali-dbx <[email protected]> * remove changes in sea_result_set functionality (to be re-added) Signed-off-by: varun-edachali-dbx <[email protected]> * introduce more integration tests Signed-off-by: varun-edachali-dbx <[email protected]> * remove SEA tests in parameterized queries Signed-off-by: varun-edachali-dbx <[email protected]> * remove partial parameter fix changes Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary timestamp tests (pass with minor disparity) Signed-off-by: varun-edachali-dbx <[email protected]> * slightly stronger typing of _convert_json_types Signed-off-by: varun-edachali-dbx <[email protected]> * stronger typing of json utility func s Signed-off-by: varun-edachali-dbx <[email protected]> * stronger typing of fetch*_json Signed-off-by: varun-edachali-dbx <[email protected]> * remove unused helper methods in SqlType Signed-off-by: varun-edachali-dbx <[email protected]> * line breaks after multi line pydocs, remove excess logs Signed-off-by: varun-edachali-dbx <[email protected]> * line breaks after multi line pydocs, reduce diff of redundant changes Signed-off-by: varun-edachali-dbx <[email protected]> * reduce diff of redundant changes Signed-off-by: varun-edachali-dbx <[email protected]> * mandate ResultData in SeaResultSet constructor Signed-off-by: varun-edachali-dbx <[email protected]> * remove complex type conversion Signed-off-by: varun-edachali-dbx <[email protected]> * correct fetch*_arrow Signed-off-by: varun-edachali-dbx <[email protected]> * recover old sea tests Signed-off-by: varun-edachali-dbx <[email protected]> * move queue and result set into SEA specific dir Signed-off-by: varun-edachali-dbx <[email protected]> * pass ssl_options into CloudFetchQueue Signed-off-by: varun-edachali-dbx <[email protected]> * reduce diff Signed-off-by: varun-edachali-dbx <[email protected]> * remove redundant conversion.py Signed-off-by: varun-edachali-dbx <[email protected]> * fix type issues Signed-off-by: varun-edachali-dbx <[email protected]> * ValueError not ProgrammingError Signed-off-by: varun-edachali-dbx <[email protected]> * reduce diff Signed-off-by: varun-edachali-dbx <[email protected]> * introduce SEA cloudfetch e2e tests Signed-off-by: varun-edachali-dbx <[email protected]> * allow empty cloudfetch result Signed-off-by: varun-edachali-dbx <[email protected]> * add unit tests for CloudFetchQueue and SeaResultSet Signed-off-by: varun-edachali-dbx <[email protected]> * skip pyarrow dependent tests Signed-off-by: varun-edachali-dbx <[email protected]> * simplify download process: no pre-fetching Signed-off-by: varun-edachali-dbx <[email protected]> * correct class name in logs Signed-off-by: varun-edachali-dbx <[email protected]> * align with old impl Signed-off-by: varun-edachali-dbx <[email protected]> * align next_n_rows with prev imple Signed-off-by: varun-edachali-dbx <[email protected]> * align remaining_rows with prev impl Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary Optional params Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary changes in thrift field if tests Signed-off-by: varun-edachali-dbx <[email protected]> * remove unused imports Signed-off-by: varun-edachali-dbx <[email protected]> * init hybrid * run large queries Signed-off-by: varun-edachali-dbx <[email protected]> * hybrid disposition Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-ncessary log Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * remove redundant tests Signed-off-by: varun-edachali-dbx <[email protected]> * multi frame decompression of lz4 Signed-off-by: varun-edachali-dbx <[email protected]> * ensure no compression (temp) Signed-off-by: varun-edachali-dbx <[email protected]> * introduce separate link fetcher Signed-off-by: varun-edachali-dbx <[email protected]> * log time to create table Signed-off-by: varun-edachali-dbx <[email protected]> * add chunk index to table creation time log Signed-off-by: varun-edachali-dbx <[email protected]> * remove custom multi-frame decompressor for lz4 Signed-off-by: varun-edachali-dbx <[email protected]> * remove excess logs * remove redundant tests (temp) Signed-off-by: varun-edachali-dbx <[email protected]> * add link to download manager before notifying consumer Signed-off-by: varun-edachali-dbx <[email protected]> * move link fetching immediately before table creation so link expiry is not an issue Signed-off-by: varun-edachali-dbx <[email protected]> * resolve merge artifacts Signed-off-by: varun-edachali-dbx <[email protected]> * remove redundant methods Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * introduce callback to handle link expiry Signed-off-by: varun-edachali-dbx <[email protected]> * fix types Signed-off-by: varun-edachali-dbx <[email protected]> * fix param type in unit tests Signed-off-by: varun-edachali-dbx <[email protected]> * formatting + minor type fixes Signed-off-by: varun-edachali-dbx <[email protected]> * Revert "introduce callback to handle link expiry" This reverts commit bd51b1c. * remove unused callback (to be introduced later) Signed-off-by: varun-edachali-dbx <[email protected]> * correct param extraction Signed-off-by: varun-edachali-dbx <[email protected]> * remove common constructor for databricks client abc Signed-off-by: varun-edachali-dbx <[email protected]> * make SEA Http Client instance a private member Signed-off-by: varun-edachali-dbx <[email protected]> * make GetChunksResponse model more robust Signed-off-by: varun-edachali-dbx <[email protected]> * add link to doc of GetChunk response model Signed-off-by: varun-edachali-dbx <[email protected]> * pass result_data instead of "initial links" into SeaCloudFetchQueue Signed-off-by: varun-edachali-dbx <[email protected]> * move download_manager init into parent CloudFetchQueue Signed-off-by: varun-edachali-dbx <[email protected]> * raise ServerOperationError for no 0th chunk Signed-off-by: varun-edachali-dbx <[email protected]> * unused iports Signed-off-by: varun-edachali-dbx <[email protected]> * return None in case of empty respose Signed-off-by: varun-edachali-dbx <[email protected]> * ensure table is empty on no initial link s Signed-off-by: varun-edachali-dbx <[email protected]> * account for total chunk count Signed-off-by: varun-edachali-dbx <[email protected]> * iterate by chunk index instead of link Signed-off-by: varun-edachali-dbx <[email protected]> * make LinkFetcher convert link static Signed-off-by: varun-edachali-dbx <[email protected]> * add helper for link addition, check for edge case to prevent inf wait Signed-off-by: varun-edachali-dbx <[email protected]> * add unit tests for LinkFetcher Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary download manager check Signed-off-by: varun-edachali-dbx <[email protected]> * remove un-necessary string literals around param type Signed-off-by: varun-edachali-dbx <[email protected]> * remove duplicate download_manager init Signed-off-by: varun-edachali-dbx <[email protected]> * account for empty response in LinkFetcher init Signed-off-by: varun-edachali-dbx <[email protected]> * make get_chunk_link return mandatory ExternalLink Signed-off-by: varun-edachali-dbx <[email protected]> * set shutdown_event instead of breaking on completion so get_chunk_link is informed Signed-off-by: varun-edachali-dbx <[email protected]> * docstrings, logging, pydoc Signed-off-by: varun-edachali-dbx <[email protected]> * use total_chunk_cound > 0 Signed-off-by: varun-edachali-dbx <[email protected]> * clarify that link has already been submitted on getting row_offset Signed-off-by: varun-edachali-dbx <[email protected]> * return None for out of range Signed-off-by: varun-edachali-dbx <[email protected]> * default link_fetcher to None Signed-off-by: varun-edachali-dbx <[email protected]> --------- Signed-off-by: varun-edachali-dbx <[email protected]>
* chunk download latency Signed-off-by: Sai Shree Pradhan <[email protected]> * formatting Signed-off-by: Sai Shree Pradhan <[email protected]> * test fixes Signed-off-by: Sai Shree Pradhan <[email protected]> * sea-migration static type checking fixes Signed-off-by: Sai Shree Pradhan <[email protected]> * check types fix Signed-off-by: Sai Shree Pradhan <[email protected]> * fix type issues Signed-off-by: varun-edachali-dbx <[email protected]> * type fix revert Signed-off-by: Sai Shree Pradhan <[email protected]> * - Signed-off-by: Sai Shree Pradhan <[email protected]> * statement id in get metadata functions Signed-off-by: Sai Shree Pradhan <[email protected]> * removed result set extractor Signed-off-by: Sai Shree Pradhan <[email protected]> * databricks client type Signed-off-by: Sai Shree Pradhan <[email protected]> * formatting Signed-off-by: Sai Shree Pradhan <[email protected]> * remove defaults, fix chunk id Signed-off-by: Sai Shree Pradhan <[email protected]> * added statement type to command id Signed-off-by: Sai Shree Pradhan <[email protected]> * check types fix Signed-off-by: Sai Shree Pradhan <[email protected]> * renamed chunk_id to num_downloaded_chunks Signed-off-by: Sai Shree Pradhan <[email protected]> * set statement type to query for chunk download Signed-off-by: Sai Shree Pradhan <[email protected]> * comment fix Signed-off-by: Sai Shree Pradhan <[email protected]> * removed dup check for trowset Signed-off-by: Sai Shree Pradhan <[email protected]> --------- Signed-off-by: Sai Shree Pradhan <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
varun-edachali-dbx
had a problem deploying
to
azure-prod
GitHub Actions
Failure
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
varun-edachali-dbx
had a problem deploying
to
azure-prod
GitHub Actions
Failure
Signed-off-by: varun-edachali-dbx <[email protected]>
varun-edachali-dbx
had a problem deploying
to
azure-prod
GitHub Actions
Failure
Signed-off-by: varun-edachali-dbx <[email protected]>
varun-edachali-dbx
had a problem deploying
to
azure-prod
GitHub Actions
Failure
Signed-off-by: varun-edachali-dbx <[email protected]>
varun-edachali-dbx
had a problem deploying
to
azure-prod
GitHub Actions
Failure
Signed-off-by: varun-edachali-dbx <[email protected]>
varun-edachali-dbx
had a problem deploying
to
azure-prod
GitHub Actions
Failure
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
varun-edachali-dbx
had a problem deploying
to
azure-prod
GitHub Actions
Failure
Signed-off-by: varun-edachali-dbx <[email protected]>
varun-edachali-dbx
temporarily deployed
to
azure-prod
GitHub Actions
Inactive
Signed-off-by: varun-edachali-dbx <[email protected]>
varun-edachali-dbx
temporarily deployed
to
azure-prod
GitHub Actions
Inactive
varun-edachali-dbx
requested review from
jayantsing-db,
vikrantpuppala and
saishreeeee
|
This PR incorporates #634 as well, so requesting @vikrantpuppala and @saishreeeee for review. Please give the telemetry-relevant files a look. Thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
Description
Introduce the fetch phase to round out the SEA backend. This PR is composed of the following:
INLINEdisposition andJSON_ARRAYformat) #594EXTERNAL_LINKSdisposition andARROWformat) #598Hybriddisposition #631It also incorporates chunk download latency logs as introduced by #634
How is this tested?
examples/experimental.Related Tickets & Documents
Design Doc