Integer overflow in innocent looking linear interpolation. Should some warning be issued?

[hurak]

Overflow is encountered during and innocent looking linear interpolation/extrapolation:

Interpolate linearly two points (1,1) and (10000,10000) and then find the value at 10001 by linear extrapolation.
The following code does it

julia> itpi = LinearInterpolation([1,10000],[1,10000]; extrapolation_bc=Line())
2-element extrapolate(interpolate((::Vector{Int64},), ::Vector{Int64}, Gridded(Linear())), Line()) with element type Float64:
 Ratios.SimpleRatio{Int64}(99980001, 99980001)
 Ratios.SimpleRatio{Int64}(999800010000, 99980001)

julia> convert(Float64,itpi(10001))
773.9376918087789

The result is clearly incorrect because the correct answer is 10001. Looking at the displayed ratios, I guess that an integer overflow is what happened here. There are no problems with smaller numbers:

julia> itp = LinearInterpolation([1,2],[1,2]; extrapolation_bc=Line())
2-element extrapolate(interpolate((::Vector{Int64},), ::Vector{Int64}, Gridded(Linear())), Line()) with element type Float64:
 Ratios.SimpleRatio{Int64}(1, 1)
 Ratios.SimpleRatio{Int64}(2, 1)

julia> convert(Float64,itp(3))
3.0

or with the floating point data:

julia> itpf = LinearInterpolation([1.0,10000.0],[1.0,10000.0]; extrapolation_bc=Line())
2-element extrapolate(interpolate((::Vector{Float64},), ::Vector{Float64}, Gridded(Linear())), Line()) with element type Float64:
     1.0
 10000.0

julia> convert(Float64,itpf(10001.0))
10001.0

I am hesitating if this should be regarded (and filed) as an Issue. In general the responsibility for the overflow problems is on the application programmer, right? But shouldn’t the interpolation function give some warning here? The parameters of the problem look innocent, don’t they?

[mkitti]

If there is a bug here, it is that overflow occurs for such a simple case. If the user supplies an integer type, we stick with that integer type for the computation to take advantage of any performance gains along with any potential issues such as overflow.

Let's distill the bug to be a failure to simplify the SimpleRatio which makes us prematurely vulnerable to overflow in this case.

[timholy]

Can someone check whether it would be fixed with timholy/Ratios.jl#19 (comment)?

[mkitti]

Can someone check whether it would be fixed with timholy/Ratios.jl#19 (comment)?

julia> using Interpolations, Ratios

julia> itpi = LinearInterpolation([1,10000],[1,10000]; extrapolation_bc=Line())
2-element extrapolate(interpolate((::Vector{Int64},), ::Vector{Int64}, Gridded(Linear())), Line()) with element type Float64:
 SimpleRatio{Int64}(99980001, 99980001)
 SimpleRatio{Int64}(999800010000, 99980001)

julia> itpi(10001)
SimpleRatio{Int64}(7736281631652211921, 9996000599960001)

julia> convert(Float64, itpi(10001))
773.9376918087789

julia> Base.:+(x::SimpleRatio, y::SimpleRatio) = x.den == y.den ? SimpleRatio(x.num + y.num, x.den) : SimpleRatio(x.num*y.den + x.den*y.num, x.den*y.den)

julia> Base.:-(x::SimpleRatio, y::SimpleRatio) = x.den == y.den ? SimpleRatio(x.num - y.num, x.den) : SimpleRatio(x.num*y.den - x.den*y.num, x.den*y.den)

julia> itpi = LinearInterpolation([1,10000],[1,10000]; extrapolation_bc=Line())
2-element extrapolate(interpolate((::Vector{Int64},), ::Vector{Int64}, Gridded(Linear())), Line()) with element type Float64:
 SimpleRatio{Int64}(9999, 9999)
 SimpleRatio{Int64}(99990000, 9999)

julia> convert(Float64, itpi(10001))
10001.0

[mkitti]

Within Interpolations.jl, my thought would be to simplify the ratio before construction:

julia> using Interpolations, Ratios

julia> Interpolations.ratio(num::Integer, denom::Integer) = SimpleRatio(Base.divgcd(promote(num, denom)...)...)

julia> itpi = LinearInterpolation([1,10000],[1,10000]; extrapolation_bc=Line())
2-element extrapolate(interpolate((::Vector{Int64},), ::Vector{Int64}, Gridded(Linear())), Line()) with element type Float64:
 SimpleRatio{Int64}(1, 1)
 SimpleRatio{Int64}(10000, 1)

julia> itpi(10001)
SimpleRatio{Int64}(999899990001, 99980001)

julia> convert(Float64, itpi(10001))
10001.0

This also combines well with the above fix for Ratios.jl:

julia> Base.:+(x::SimpleRatio, y::SimpleRatio) = x.den == y.den ? SimpleRatio(x.num + y.num, x.den) : SimpleRatio(x.num*y.den + x.den*y.num, x.den*y.den)

julia> Base.:-(x::SimpleRatio, y::SimpleRatio) = x.den == y.den ? SimpleRatio(x.num - y.num, x.den) : SimpleRatio(x.num*y.den - x.den*y.num, x.den*y.den)

julia> itpi(10001)
SimpleRatio{Int64}(99999999, 9999)

Perhaps a simplify method would be generally useful?

julia> simplify(r::SimpleRatio) = (d = gcd(r.num, r.den); SimpleRatio(r.num ÷ d, r.den ÷ d))
simplify (generic function with 1 method)

julia> itpi(10001)
SimpleRatio{Int64}(99999999, 9999)

julia> simplify(itpi(10001))
SimpleRatio{Int64}(10001, 1)

At some point, it seems we are patching SimpleRatio until it becomes Rational.

[timholy]

At some point, it seems we are patching SimpleRatio until it becomes Rational.

Yes, and that would be a huge problem.

julia> using BenchmarkTools

julia> @btime Base.divgcd(7736281631652211921, 9996000599960001)
  127.862 ns (0 allocations: 0 bytes)
(7736281631652211921, 9996000599960001)

vs

julia> using Interpolations

julia> using Interpolations.Ratios

julia> Base.:+(x::SimpleRatio, y::SimpleRatio) = x.den == y.den ? SimpleRatio(x.num + y.num, x.den) : SimpleRatio(x.num*y.den + x.den*y.num, x.den*y.den)

julia> Base.:-(x::SimpleRatio, y::SimpleRatio) = x.den == y.den ? SimpleRatio(x.num - y.num, x.den) : SimpleRatio(x.num*y.den - x.den*y.num, x.den*y.den)

julia> itpi = LinearInterpolation([1,10000],[1,10000]; extrapolation_bc=Line())
2-element extrapolate(interpolate((::Vector{Int64},), ::Vector{Int64}, Gridded(Linear())), Line()) with element type Float64:
 SimpleRatio{Int64}(9999, 9999)
 SimpleRatio{Int64}(99990000, 9999)

julia> @btime $itpi(10001)
  7.331 ns (0 allocations: 0 bytes)
SimpleRatio{Int64}(99999999, 9999)

Do you see why I suggested that strategy first?

[mkitti]

Do you see why I suggested that strategy first?

It seems the denominator check seems to address this specific issue and is not too onerous. Do you want me to create an issue over at Ratios.jl? Would ifelse be better than a ternary operator to avoid branching logic?

[mkitti]

Regarding a warning or error that overflow occurred, perhaps working on the composition of this package, Ratios.jl, and SaferIntegers.jl would be useful.

Here's my attempt which requires some hot patching at the moment:

julia> using SaferIntegers, Interpolations, Ratios

julia> import SaferIntegers: ndigits0z, baseint, SaferIntegers

julia> for S in (:SafeSigned, :SafeUnsigned)
         @eval begin
               ndigits0z(x::T) where T<:$S = ndigits0z(baseint(x)) # do not reconvert
           ndigits0z(x::T1, b::T2) where T1<:$S where T2<:SafeInteger = ndigits0z(baseint(x), baseint(b)) # do not reconvert
           ndigits0z(x::T1, b::T2) where T1<:$S where T2<:Integer = ndigits0z(baseint(x), b) # do not reconvert
         end
       end

julia> Base.float(x::SafeInteger) = Base.float(SaferIntegers.baseint(x))

julia> Base.:-(x::SimpleRatio{T}) where {T<:SafeSigned} = SimpleRatio(-x.num, x.den)

julia> itpi = LinearInterpolation(SafeInt[1,10000],SafeInt[1,10000]; extrapolation_bc=Line())
2-element extrapolate(interpolate((::Vector{SafeInt64},), ::Vector{SafeInt64}, Gridded(Linear())), Line()) with element type Float64:
 SimpleRatio{SafeInt64}(99980001, 99980001)
 SimpleRatio{SafeInt64}(999800010000, 99980001)

julia> itpi(10001)
ERROR: OverflowError: 999800010000 * 99980001 overflowed for type Int64
Stacktrace:
 [1] throw_overflowerr_binaryop(op::Symbol, x::Int64, y::Int64)
   @ Base.Checked .\checked.jl:154
 [2] checked_mul
   @ .\checked.jl:288 [inlined]
 [3] *
   @ ~\.julia\packages\SaferIntegers\nOUgY\src\arith_ops.jl:21 [inlined]
 [4] +
   @ ~\.julia\packages\Ratios\uRs4y\src\Ratios.jl:31 [inlined]
 [5] extrapolate_axis
   @ ~\.julia\dev\Interpolations\src\extrapolation\extrapolation.jl:168 [inlined]
 [6] extrapolate_value
   @ ~\.julia\dev\Interpolations\src\extrapolation\extrapolation.jl:162 [inlined]
 [7] (::Interpolations.Extrapolation{Float64, 1, Interpolations.GriddedInterpolation{Float64, 1, SafeInt64, Gridded{Linear{Throw{OnGrid}}}, Tuple{Vector{SafeInt64}}}, Gridded{Linear{Throw{OnGrid}}}, Line{Nothing}})(x::Int64)
   @ Interpolations ~\.julia\dev\Interpolations\src\extrapolation\extrapolation.jl:52
 [8] top-level scope
   @ REPL[7]:1

[timholy]

I think before making any decisions, all the options should be benchmarked, including against the (buggy) implementation we have now. SaferIntegers is appealing in principle, but what are the performance implications? Interpolations is an extraordinarily performance-sensitive package.

[mkitti]

The SaferIntegers approach is orthogonal to the other approaches. Once we provide support for it, the user can make a choice whether to use a SafeInteger to detect an overflow or not.

The gcd approach is probably better handled by Rational. Perhaps we should have a Preferences.jl mechanism to use Rational instead of SimpleRatio when performance is not the primary need of the user?

To summarize, I think we have three orthogonal approaches:

1. Modify Ratios.jl to check if SimpleRatios has the same denominator during addition and subtraction. The added overhead of that check should be benchmarked carefully. Based on my informal experiments and the above times, the overhead looks reasonable to me.

2. Modify SaferIntegers.jl and Ratios.jl to work together with Interpolations.jl. While benchmarks would be informative, I do not think this influences whether we should do this or not since the choice to use SaferIntegers.jl lies with the user.

3. Modify Interpolations.jl to use Preferences.jl to modify the implementation of Interpolations.ratio between SimpleRatio or Rational.

[JeffreySarnoff]

How would SaferIntegers.jl need to be modified to work with Interpolations.jl?

[mkitti]

How would SaferIntegers.jl need to be modified to work with Interpolations.jl?

Hi @JeffreySarnoff , adding Base.float for SafeInteger would help.

Base.float(x::SafeInteger) = Base.float(SaferIntegers.baseint(x))

See JeffreySarnoff/SaferIntegers.jl#19

[JeffreySarnoff]

I incorporated this

Base.float(x::S) where S<:SafeSigned = Base.float(baseint(x))
Base.float(x::U) where U<:SafeUnsigned = Base.float(baseint(x))

into SaferIntegers/src/promote.jl, bumped the version and reregistered it
(I still need to add tests)

[mkitti]

On approach #2, we are then here:

julia> using Interpolations, SaferIntegers

julia> itpi = LinearInterpolation(SafeInt[1,10000],SafeInt[1,10000]; extrapolation_bc=Line())
2-element extrapolate(interpolate((::Vector{SafeInt64},), ::Vector{SafeInt64}, Gridded(Linear())), Line()) with element type Float64:
 Ratios.SimpleRatio{SafeInt64}(99980001, 99980001)
 Ratios.SimpleRatio{SafeInt64}(999800010000, 99980001)

julia> itpi(10001)
ERROR: MethodError: no method matching -(::Ratios.SimpleRatio{SafeInt64})
...

We just need to add the missing method to Ratios now since SafeInt64 is not Signed, but SafeSigned.

julia> using Interpolations.Ratios

julia> Base.:-(x::SimpleRatio{T}) where {T<:SafeSigned} = SimpleRatio(-x.num, x.den)

julia> itpi(10001)
ERROR: OverflowError: 999800010000 * 99980001 overflowed for type Int64
Stacktrace:
 [1] throw_overflowerr_binaryop(op::Symbol, x::Int64, y::Int64)
   @ Base.Checked .\checked.jl:154
 [2] checked_mul
   @ .\checked.jl:288 [inlined]
 [3] *
   @ ~\.julia\packages\SaferIntegers\zbJSp\src\arith_ops.jl:21 [inlined]
 [4] +
   @ ~\.julia\packages\Ratios\fgO34\src\Ratios.jl:35 [inlined]
 [5] extrapolate_axis
   @ ~\.julia\dev\Interpolations\src\extrapolation\extrapolation.jl:168 [inlined]
 [6] extrapolate_value
   @ ~\.julia\dev\Interpolations\src\extrapolation\extrapolation.jl:162 [inlined]
 [7] (::Interpolations.Extrapolation{Float64, 1, Interpolations.GriddedInterpolation{Float64, 1, SafeInt64, Gridded{Linear{Throw{OnGrid}}}, Tuple{Vector{SafeInt64}}}, Gridded{Linear{Throw{OnGrid}}}, Line{Nothing}})(x::Int64)
   @ Interpolations ~\.julia\dev\Interpolations\src\extrapolation\extrapolation.jl:52
 [8] top-level scope
   @ REPL[7]:1