Skip to content

Conversation

NickCrews
Copy link
Contributor

I was getting confused as to how my pyarrow timestamps
were getting converted, and I think seeing this here would have
pointed me in the right direction sooner.

I was getting confused as to how my pyarrow timestamps
were getting converted, and I think seeing this here would have
pointed me in the right direction sooner.
@JovanVeljanoski
Copy link
Member

About the timestamp array in arrow.. so in vaex i believe it is stored as arrow, it is not really converted to numpy. However, since many of the datetime and timedelta operations supported come via pandas, it is true that on the fly the data is cast to numpy (well pandas series).

But the data is still stored as arrow.

I don't know how to state this exactly in the short text for the purpose of that table so what you've written probably is fine.. @maartenbreddels ?

@maartenbreddels
Copy link
Member

Indeed, it's being converted on the fly when doing many operations. How did it confused you @NickCrews ?

@NickCrews
Copy link
Contributor Author

I don't remember exactly now 😭, should have been more detailed.

But I think it was constructing a DF from a pyarrow timestamp array, doing some operations on it, and then re-exporting with .values, and I was surprised that it had turned into a numpy array.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants