Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

con.execute doesn't necessarily return proper dtype #214

Closed
randyzwitch opened this issue Apr 12, 2019 · 2 comments
Closed

con.execute doesn't necessarily return proper dtype #214

randyzwitch opened this issue Apr 12, 2019 · 2 comments

Comments

@randyzwitch
Copy link
Contributor

randyzwitch commented Apr 12, 2019

Update: filed internally as https://jira.omnisci.com/browse/BE-3467 for decimal downcast

When using the execute method to return a list of tuples, the following types lose precision:

In the case of integers, columns are returned as int, which is the system word size int, which is probably okay. For smaller width ints on OmniSci, this is ok, since there is no precision loss. Might be unexpected though to have a tinyint column return a 32- or 64-bit int column.

Bool is returned as 1/0, which is conventionally correct, but odd since Python itself has True/False

Modifying these behaviors are necessarily a breaking change, but in the case of losing precision, it seems like the correct thing to do.

@randyzwitch
Copy link
Contributor Author

Per https://docs.python.org/3/tutorial/floatingpoint.html:

Why is that? 1/10 is not exactly representable as a binary fraction. Almost all machines today (November 2000) use IEEE-754 floating point arithmetic, and almost all platforms map Python floats to IEEE-754 “double precision”. 754 doubles contain 53 bits of precision, so on input the computer strives to convert 0.1 to the closest fraction it can of the form J/2**N where J is an integer containing exactly 53 bits. Rewriting

@randyzwitch
Copy link
Contributor Author

Closing this in lieu of #212 for decimal issues. Bool and Int issues are moot I think, if someone else brings them up then we can revisit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant