You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've tried to use the BDouble for storage but at some point I need to use decimalExpansion to format the value to end users, and I was getting really weird values. In a debug session I got the following:
(lldb) po percent
▿ 7450921770987/7450936270987
- sign : false
▿ numerator : 1 element
- 0 : 7450921770987
▿ denominator : 1 element
- 0 : 7450936270987
- magnitude : 0.0
- _precision : 4
(lldb) po percent.decimalDescription
"0.99910"
(lldb) po percent.decimalExpansion(precisionAfterDecimalPoint: 2)
"0.910"
(lldb) po percent.decimalExpansion(precisionAfterDecimalPoint: 0)
"0.0"
(lldb) po percent.decimalExpansion(precisionAfterDecimalPoint: 7)
"0.9999981"
I'm pretty certain a value nearly 1 should not round to 0 when displayed to end users. Is decimalExpansion not the function I'm expecting and should I be using something else? I had expected the value to round to either 1.0 or 0.9.
The text was updated successfully, but these errors were encountered:
Okay I got it, in the case of precision=2, it computes 0.999 so since the third 9 is >= 5, the second 9 is rounded up, but there is a bug so it is replaced with 10 and the number becomes 0.910
Could be solved easily by checking the last digit of the derived BInt before converting to a string, if .divmod(10).remainder >= 5 then take .divmod(10).quotient + 1, else .divmod(10).quotient. Only after that take the Bint.description and prune 0's from the end if needed (so if 999 is rounded to 1000, then 1.000 has to remove the last zero
I've tried to use the BDouble for storage but at some point I need to use
decimalExpansion
to format the value to end users, and I was getting really weird values. In a debug session I got the following:I'm pretty certain a value nearly 1 should not round to 0 when displayed to end users. Is
decimalExpansion
not the function I'm expecting and should I be using something else? I had expected the value to round to either1.0
or0.9
.The text was updated successfully, but these errors were encountered: