From 13489f9ce924b519700c8fdd18dec5eb91335ea3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?G=C3=A1bor=20Lehel?= Date: Mon, 30 Jun 2014 02:44:05 +0200 Subject: [PATCH 01/18] RFC: Scoped attributes for checked arithmetic --- ...coped-attributes-for-checked-arithmetic.md | 423 ++++++++++++++++++ 1 file changed, 423 insertions(+) create mode 100644 active/0000-scoped-attributes-for-checked-arithmetic.md diff --git a/active/0000-scoped-attributes-for-checked-arithmetic.md b/active/0000-scoped-attributes-for-checked-arithmetic.md new file mode 100644 index 00000000000..4754e685cc5 --- /dev/null +++ b/active/0000-scoped-attributes-for-checked-arithmetic.md @@ -0,0 +1,423 @@ +- Start Date: 2014-06-30 +- RFC PR #: (leave this empty) +- Rust Issue #: (leave this empty) + + +# Summary + +Change the semantics of the built-in fixed-size integer types from being defined +as wrapping around on overflow to either returning an unspecified result or not +returning at all. Allow overflow checks to be turned on or off per-scope with an +attribute. Add a compiler option to force checking for testing and debugging +purposes. Add `Wrapping` traits to the standard library, with operations defined +as wrapping on overflow, for the limited number of cases where this is the +desired semantics, such as hash functions. + + +# Motivation + +The semantics of the basic arithmetic operators on the built-in fixed-size +integer types are currently defined to wrap around on overflow. Wrapping around +on overflow is well-defined behavior, which means that it's better than C. Yet +we should avoid falling prey to the soft bigotry of low expections. + +In the large majority of cases, wrapping around on overflow is not an +appropriate semantics: programs will generally not work correctly with the +wrapped-around value. Much of the time, programs are merely optimistic that +overflow won't happen, but this often turns out to be mistaken. When it does +happen, unexpected behavior and potentially even security bugs can result. It +should be emphasized that wrapping around on overflow is *not* a memory safety +issue in the safe subset of Rust, which greatly mitigates, but does not +eliminate, the harm that overflow bugs can cause. However, in `unsafe` blocks, +and when interfacing with existing C libraries, it can be a memory safety issue, +and furthermore, not all security bugs are memory safety bugs. By +indiscriminately using wraparound-on-overflow semantics in every case, whether +or not it is appropriate, it becomes difficult to impossible for programmers, +compilers, and analysis tools to reliably determine where overflow is the +expected behavior, and where the possibility of it should be considered a +defect. + +It is a fact that checked arithmetic poses an unacceptable performance burden in +many cases, especially in a performance-sensitive language like Rust. As such, a +perfect, one-size-fits-all solution is regrettably not possible. However, we can +make better compromises than we currently do. + +While in many cases, the performance cost of checking for overflow is not +acceptable, in many other cases, it is. Developers should be able to make the +tradeoff themselves in a convenient and granular way. + +The cases where wraparound on overflow is explicitly desired are comparatively +rare. The only use cases for wrapping arithmetic that are known to the author at +the time of writing are hashes, checksums, and emulation of processors with +wraparound arithmetic instructions. Therefore, it should be acceptable to not +provide symbolic operators, but require using named methods in +these cases. + + +## Goals of this proposal + + * Clearly distinguish the circumstances where overflow is expected behavior + from where it is not. + + * Provide programmers with the tools they need to flexibly make the tradeoff + between higher performance and catching more mistakes. + + * Be minimally disruptive to the language as it is and have a small surface + area. + + +## Non-goals of this proposal + + * Make checked arithmetic fast. Let me emphasize: *the actual performance of + checked arithmetic is wholly irrelevant to the content of this proposal*. It + will be relevant to the decisions programmers make when employing the tools + this proposal proposes to supply them with, but it is not to the tools + themselves. + + * Prepare for future processor architectures and/or compiler advances which may + improve the performance of checked arithmetic. If there were mathematical + proof that faster checked arithmetic is impossible, the proposal would be the + same. + + +## Acknowledgements and further reading + +Many aspects of this proposal and many of the ideas within it were influenced +and inspired by [a discussion on the rust-dev mailing list][GL18]. The author is +grateful to everyone who provided input, and would like to highlight the +following messages in particular as providing motivation for the proposal. + +On the limited use cases for wrapping arithmetic: + + * [Jerry Morrison on June 20][JM20] + +On the value of distinguishing where overflow is valid, and where it is not: + + * [Gregory Maxwell on June 18][GM18] + * [Gregory Maxwell on June 24][GM24] + * [Robert O'Callahan on June 24][ROC24] + * [Jerry Morrison on June 24][JM24] + +The idea of scoped attributes: + + * [Daniel Micay on June 23][DM23] + +On the drawbacks of a type-based approach: + + * [Daniel Micay on June 24][DM24] + +In general: + + * [John Regehr on June 23][JR23] + * [Lars Bergstrom on June 24][LB24] + +[GL18]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010363.html +[GM18]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010371.html +[JM20]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010410.html +[DM23]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010566.html +[JR23]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010558.html +[GM24]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010580.html +[ROC24]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010602.html +[DM24]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010598.html +[JM24]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010596.html +[LB24]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010579.html + + +# Detailed design + +## Semantics of overflow with the built-in types + +Currently, the built-in arithmetic operators `+`, `-`, `*`, and `/` on the +built-in types `i8`..`i64`, `u8`..`u64`, `int`, and `uint` are defined as +wrapping around on overflow. Change this to define them, on overflow, as either +returning an unspecified result, or not returning at all (i.e. terminating +execution in some fashion, "returning bottom"), instead. + +The implication is that overflow is considered to be an abnormal circumstance, +and the programmer expects it not to happen, resp. it is her goal to make sure +that it will not. + +Notes: + + * In practice, the unspecified result will most likely be the wraparound + result, but in theory, it's up to the implementation. + + * "Terminating execution in some fashion" will most likely mean failing the + task, but the defined semantics of the types do not foreclose on other + possibilities. + + * Most importantly: this is **not** undefined behavior in the C sense. Only the + result of the operation is left unspecified, as opposed to the entire + program's meaning, as in C. The programmer would not be allowed to rely on a + specific, or any, result being returned on overflow, but the compiler would + also not be allowed to assume that overflow won't happen. + + +## Scoped attributes to control checking + +This depends on [RFC PR 16][16] being accepted. + +Introduce an `overflow_checks` attribute which can be used to turn overflow +checks on or off in a given scope. `#[overflow_checks(on)]` turns them on, +`#[overflow_checks(off)]` turns them off. The attribute can be applied to a +whole `crate`, a `mod`ule, an `fn`, or (as per [RFC PR 16][16]) a given block or +a single expression. When applied to a block, this is analogous to the +`checked { }` blocks of C#. As with lint attributes, an `overflow_checks` +attribute on an inner scope or item will override the effects of any +`overflow_checks` attributes on outer scopes or items. (Overflow checks can in +fact be thought of as a kind of run-time lint.) Where overflow checks are in +effect, overflow with the basic arithmetic operations on the built-in fixed-size +integer types invokes `fail!()`. Where they are not, the checks are omitted, and +the result of the operations is left unspecified (but will most likely wrap). + +Illustration: + + // checks are on for this crate + #![overflow_checks(on)] + + // but they are off for this module + #[overflow_checks(off)] + mod some_stuff { + + // but they are on for this function + #[overflow_checks(on)] + fn do_thing() { + ... + + // but they are off for this block + #[overflow_checks(off)] { + ... + // but they are on for this expression + let n = #[overflow_checks(on)] (a * b + c); + ... + } + + ... + } + + ... + } + + ... + +[16]: https://github.com/rust-lang/rfcs/pull/16 + +### The default + +There is a significant decision to be made with respect to the default behavior +where neither `overflow_checks(on)` nor `off` has been explicitly specified. The +author does not presume to know the correct answer, and leaves this open to +debate. The following defaults are possible: + + 1. The default is `on`. This means that the default is to catch more mistakes. + + 2. The default is `off`. This means that the default is to be faster. (This + happens to be the current "default".) + + 3. There is no default, and a decision is forced. If the programmer neglects to + explicitly specify a behavior, the compiler will bail out and ask her to + specify one. + + 4. Combination of (1) and (3): The default is `on`, but the compiler emits a + warning when falling back to the default behavior. + + 5. Combination of (2) and (3): The default is `off`, but the compiler emits a + warning when falling back to the default behavior. + + +## A debugging switch to force checking + +The programmer has the option to turn `overflow_checks(off)` due to performance +considerations. However, when testing or debugging the program, for instance +when tracking down a difficult bug, it may be desired to throw performance to +the wind and enable as many checks as possible. For this purpose, provide a +compiler option, e.g. `--force-overflow-checks`, which causes overflow checks to +be considered `on` even where an attribute has turned them `off`. This is +somewhat analogous to the behavior of our current `--ndebug` flag and +`debug_assert!` macros. + + +## Traits for wrapping arithmetic + +For those use cases where explicit wraparound on overflow is required, such as +hash functions, we must provide operations with such semantics. Accomplish this +by providing the following traits in the `prelude`: + + pub trait WrappingAdd { + fn wrapping_add(self, rhs: Self) -> Self; + } + + pub trait WrappingSub { + fn wrapping_sub(self, rhs: Self) -> Self; + } + + pub trait WrappingMul { + fn wrapping_mul(self, rhs: Self) -> Self; + } + + pub trait WrappingDiv { + fn wrapping_div(self, rhs: Self) -> Self; + } + +Provide `impl`s of each of these traits for each of the built-in fixed-size +integer types, with the operations implemented to wrap around on overflow +unconditionally. + + +### `Wrapping` + +For convenience, also provide a `Wrapping` newtype for which the operator +overloads are implemented using the `Wrapping` traits: + + pub struct Wrapping(pub T); + + impl Add, Wrapping> for Wrapping { + fn add(&self, other: &Wrapping) -> Wrapping { + self.wrapping_add(*other) + } + } + + // Likewise for `Sub`, `Mul`, and `Div` + +Note that this is only for potential convenience. The type-based approach has the +drawback that e.g. `Vec` and `Vec>` are incompatible types. +The recommendation is to not use `Vec>`, but to use `Vec` and +the `wrapping_*` methods directly, instead. + + +# Drawbacks + + * Required implementation work: + + * Implement [RFC PR 16][16]. + + * Implement the `overflow_checks` attribute. + + * Port existing code which relies on wraparound semantics (primarily hash + functions) to use the `Wrapping` traits. + + * Code where `overflow_checks(off)` is in effect could end up accidentally + relying on overflow. Given the relative scarcity of cases where overflow is a + favorable circumstance, the risk of this happening seems minor. + + * Having to think about whether wraparound arithmetic is appropriate may + cause an increased cognitive burden. However, wraparound arithmetic is + almost never appropriate. Therefore, programmers should be able to keep using + the built-in integer types and to not think about it. Where wraparound + semantics are required, it is generally a specialized use case with the + implementor well aware of the requirement. + + * The built-in types become "special": the ability to control overflow checks + using scoped attributes doesn't extend to user-defined types. If you make a + `struct MyNum(int)` and `impl Add for MyNum` using the native `+` operation + on `int`s, whether overflow checks happen for `MyNum` is determined by + whether `overflow_checks` is `on` or `off` where `impl Add for MyNum` is + declared, not whether they are `on` or `off` where the overloaded operators + are used. + + The author considers this to be a serious shortcoming. *However*, it has the + saving grace of being no worse than the status quo, i.e. the change is still + a Pareto-improvement. Under the status quo, neither the built-in types nor + user-defined types can have overflow checks controlled by scoped attributes. + Under this proposal, the situation is improved with built-in types gaining + this capability. Under this light, making further improvements, namely + extending the capability to user-defined types, can be left to future work. + + * Someone may conduct a benchmark of Rust with overflow checks turned on, post + it to the Internet, and mislead the audience into thinking that Rust is a + slow language. + + +# Alternatives + +## Do nothing for now + +Defer any action until later, as suggested by: + +* [Patrick Walton on June 22][PW22] + +Reasons this was not pursued: The proposed changes are relatively well-contained. +Doing this after 1.0 would require either breaking existing programs which rely +on wraparound semantics, or introducing an entirely new set of integer types and +porting all code to use those types, whereas doing it now lets us avoid +needlessly proliferating types. Given the paucity of circumstances where +wraparound semantics is appropriate, having it be the default is defensible only +if better options aren't available. + +## Checks off means wrapping on + +Where overflow checks are turned off, instead of defining overflow as returning +an unspecified result, define it to wrap around. This would allow us to do +without the `Wrapping` traits and to avoid having unspecified results. See: + + * [Daniel Micay on June 24][DM24_2] + +Reasons this was not pursued: Having the declared semantics of a type change +based on context is weird. It should be possible to make the choice between +turning checks `on` or `off` solely based on performance considerations. It +should be possible to distinguish cases where checking was too expensive and +where wraparound was desired. Wraparound is not usually desired. + +## Different operators + +Have the usual arithmetic operators check for overflow, and introduce a new set +of operators with wraparound semantics, as done by Swift. Alternately, do the +reverse: make the normal operators wrap around, and introduce new ones which +check. + +Reasons this was not pursued: New, strange operators would pose an entrance +barrier to the language. The use cases for wraparound semantics are not common +enough to warrant having a separate set of symbolic operators. + +## Different types + +Have separate sets of fixed-size integer types which wrap around on overflow and +which are checked for overflow (e.g. `u8`, `u8c`, `i8`, `i8c`, ...). + +Reasons this was not pursued: Programmers might be confused by having to choose +among so many types. Using different types would introduce compatibility hazards +to APIs. `Vec` and `Vec` are incompatible. Wrapping arithmetic is not +common enough to warrant a whole separate set of types. + +## Just use `Checked*` + +Just use the existing `Checked` traits and a `Checked` type after the same +fashion as the `Wrapping` in this proposal. + +Reasons this was not pursued: Wrong defaults. Doesn't enable distinguishing +"checking is slow" from "wrapping is desired" from "it was the default". + +## Runtime-closed range types + +[As proposed by Bill Myers.][BM-RFC] + +Reasons this was not pursued: My brain melted. :( + + +# Unresolved questions + +"What should the default be where neither `overflow_checks(on)` nor `off` has +been explicitly specified?", as discussed in the main text. + + +# Future work + + * Extend the ability to make use of local `overflow_checks(on|off)` attributes + to user-defined types, as discussed under Drawbacks. (The author has some + preliminary ideas, however, they are preliminary.) + + * Look into adopting imprecise exceptions and a similar design to Ada's, and to + what is explored in the research on AIR (As Infinitely Ranged) semantics, to + improve the performance of checked arithmetic. See also: + + * [Cameron Zwarich on June 22][CZ22] + * [John Regehr on June 23][JR23_2] + + * Make it easier to use integer types of unbounded size, i.e. actual + mathematical integers and naturals. + +[BM-RFC]: https://github.com/bill-myers/rfcs/blob/no-integer-overflow/active/0000-no-integer-overflow.md +[PW22]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010494.html +[DM24_2]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010590.html +[CZ22]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010483.html +[JR23_2]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010527.html From ee6cc65d52216257f6cc85c5d682dc71a77000ca Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?G=C3=A1bor=20Lehel?= Date: Mon, 30 Jun 2014 02:48:11 +0200 Subject: [PATCH 02/18] typo --- active/0000-scoped-attributes-for-checked-arithmetic.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/active/0000-scoped-attributes-for-checked-arithmetic.md b/active/0000-scoped-attributes-for-checked-arithmetic.md index 4754e685cc5..3a6712dfcc6 100644 --- a/active/0000-scoped-attributes-for-checked-arithmetic.md +++ b/active/0000-scoped-attributes-for-checked-arithmetic.md @@ -19,7 +19,7 @@ desired semantics, such as hash functions. The semantics of the basic arithmetic operators on the built-in fixed-size integer types are currently defined to wrap around on overflow. Wrapping around on overflow is well-defined behavior, which means that it's better than C. Yet -we should avoid falling prey to the soft bigotry of low expections. +we should avoid falling prey to the soft bigotry of low expectations. In the large majority of cases, wrapping around on overflow is not an appropriate semantics: programs will generally not work correctly with the From 47ada301a4869b782d97e558f989589122418def Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?G=C3=A1bor=20Lehel?= Date: Thu, 3 Jul 2014 12:30:52 +0200 Subject: [PATCH 03/18] merge wrapping ops into a single `WrappingOps` trait, add `wrapping_rem` --- ...coped-attributes-for-checked-arithmetic.md | 59 ++++++++++--------- 1 file changed, 30 insertions(+), 29 deletions(-) diff --git a/active/0000-scoped-attributes-for-checked-arithmetic.md b/active/0000-scoped-attributes-for-checked-arithmetic.md index 3a6712dfcc6..ea8981b0c43 100644 --- a/active/0000-scoped-attributes-for-checked-arithmetic.md +++ b/active/0000-scoped-attributes-for-checked-arithmetic.md @@ -8,10 +8,10 @@ Change the semantics of the built-in fixed-size integer types from being defined as wrapping around on overflow to either returning an unspecified result or not returning at all. Allow overflow checks to be turned on or off per-scope with an -attribute. Add a compiler option to force checking for testing and debugging -purposes. Add `Wrapping` traits to the standard library, with operations defined -as wrapping on overflow, for the limited number of cases where this is the -desired semantics, such as hash functions. +attribute. Add a compiler option to force checking for testing and debugging +purposes. Add a `WrappingOps` trait to the standard library, with operations +defined as wrapping on overflow, for the limited number of cases where this is +the desired semantics, such as hash functions. # Motivation @@ -127,7 +127,7 @@ In general: ## Semantics of overflow with the built-in types -Currently, the built-in arithmetic operators `+`, `-`, `*`, and `/` on the +Currently, the built-in arithmetic operators `+`, `-`, `*`, `/`, and `%` on the built-in types `i8`..`i64`, `u8`..`u64`, `int`, and `uint` are defined as wrapping around on overflow. Change this to define them, on overflow, as either returning an unspecified result, or not returning at all (i.e. terminating @@ -237,52 +237,53 @@ somewhat analogous to the behavior of our current `--ndebug` flag and `debug_assert!` macros. -## Traits for wrapping arithmetic +## `WrappingOps` trait for explicit wrapping arithmetic For those use cases where explicit wraparound on overflow is required, such as hash functions, we must provide operations with such semantics. Accomplish this -by providing the following traits in the `prelude`: +by providing the following trait and impls in the `prelude`: - pub trait WrappingAdd { + pub trait WrappingOps { fn wrapping_add(self, rhs: Self) -> Self; - } - - pub trait WrappingSub { fn wrapping_sub(self, rhs: Self) -> Self; - } - - pub trait WrappingMul { fn wrapping_mul(self, rhs: Self) -> Self; - } - - pub trait WrappingDiv { fn wrapping_div(self, rhs: Self) -> Self; + fn wrapping_rem(self, rhs: Self) -> Self; } -Provide `impl`s of each of these traits for each of the built-in fixed-size -integer types, with the operations implemented to wrap around on overflow -unconditionally. + impl WrappingOps for int + impl WrappingOps for uint + impl WrappingOps for i8 + impl WrappingOps for u8 + impl WrappingOps for i16 + impl WrappingOps for u16 + impl WrappingOps for i32 + impl WrappingOps for u32 + impl WrappingOps for i64 + impl WrappingOps for u64 + +These are implemented to wrap around on overflow unconditionally. -### `Wrapping` +### `Wrapping` type for convenience For convenience, also provide a `Wrapping` newtype for which the operator -overloads are implemented using the `Wrapping` traits: +overloads are implemented using the `WrappingOps` trait: pub struct Wrapping(pub T); - impl Add, Wrapping> for Wrapping { + impl Add, Wrapping> for Wrapping { fn add(&self, other: &Wrapping) -> Wrapping { self.wrapping_add(*other) } } - // Likewise for `Sub`, `Mul`, and `Div` + // Likewise for `Sub`, `Mul`, `Div`, and `Rem` Note that this is only for potential convenience. The type-based approach has the drawback that e.g. `Vec` and `Vec>` are incompatible types. The recommendation is to not use `Vec>`, but to use `Vec` and -the `wrapping_*` methods directly, instead. +the `wrapping_`* methods directly, instead. # Drawbacks @@ -294,7 +295,7 @@ the `wrapping_*` methods directly, instead. * Implement the `overflow_checks` attribute. * Port existing code which relies on wraparound semantics (primarily hash - functions) to use the `Wrapping` traits. + functions) to use the `wrapping_`* methods. * Code where `overflow_checks(off)` is in effect could end up accidentally relying on overflow. Given the relative scarcity of cases where overflow is a @@ -320,7 +321,7 @@ the `wrapping_*` methods directly, instead. a Pareto-improvement. Under the status quo, neither the built-in types nor user-defined types can have overflow checks controlled by scoped attributes. Under this proposal, the situation is improved with built-in types gaining - this capability. Under this light, making further improvements, namely + this capability. In light of this, making further improvements, namely extending the capability to user-defined types, can be left to future work. * Someone may conduct a benchmark of Rust with overflow checks turned on, post @@ -334,7 +335,7 @@ the `wrapping_*` methods directly, instead. Defer any action until later, as suggested by: -* [Patrick Walton on June 22][PW22] + * [Patrick Walton on June 22][PW22] Reasons this was not pursued: The proposed changes are relatively well-contained. Doing this after 1.0 would require either breaking existing programs which rely @@ -348,7 +349,7 @@ if better options aren't available. Where overflow checks are turned off, instead of defining overflow as returning an unspecified result, define it to wrap around. This would allow us to do -without the `Wrapping` traits and to avoid having unspecified results. See: +without the `WrappingOps` trait and to avoid having unspecified results. See: * [Daniel Micay on June 24][DM24_2] From 797dcadbaa5620ab550649d4fb34baaf3c3a6bdf Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?G=C3=A1bor=20Lehel?= Date: Tue, 26 Aug 2014 18:10:03 +0200 Subject: [PATCH 04/18] update to reflect points from discussion --- ...coped-attributes-for-checked-arithmetic.md | 287 ++++++++++++------ 1 file changed, 194 insertions(+), 93 deletions(-) diff --git a/active/0000-scoped-attributes-for-checked-arithmetic.md b/active/0000-scoped-attributes-for-checked-arithmetic.md index ea8981b0c43..7bcaaf6e468 100644 --- a/active/0000-scoped-attributes-for-checked-arithmetic.md +++ b/active/0000-scoped-attributes-for-checked-arithmetic.md @@ -6,52 +6,56 @@ # Summary Change the semantics of the built-in fixed-size integer types from being defined -as wrapping around on overflow to either returning an unspecified result or not -returning at all. Allow overflow checks to be turned on or off per-scope with an -attribute. Add a compiler option to force checking for testing and debugging -purposes. Add a `WrappingOps` trait to the standard library, with operations -defined as wrapping on overflow, for the limited number of cases where this is +as wrapping around on overflow to it being considered a program error (but *not* +undefined behavior in the C sense), with whether to check for it at compile +and/or runtime being left up to the implementation. Add an attribute to allow +runtime overflow checks to be turned on or off per-scope, similarly to lints. +Add a compiler option to force checking to be on for testing and debugging +purposes. Add a `WrappingOps` trait to the standard library with operations +defined as wrapping on overflow for the limited number of cases where this is the desired semantics, such as hash functions. # Motivation -The semantics of the basic arithmetic operators on the built-in fixed-size -integer types are currently defined to wrap around on overflow. Wrapping around -on overflow is well-defined behavior, which means that it's better than C. Yet -we should avoid falling prey to the soft bigotry of low expectations. +The semantics of the basic arithmetic operators and casts on the built-in +fixed-size integer types are currently defined to wrap around on overflow. This +is is well-defined behavior, so it is already an improvement over C. Yet we +should avoid falling into the trap of low expectations. -In the large majority of cases, wrapping around on overflow is not an +In the large majority of cases, wrapping around on overflow is not a useful or appropriate semantics: programs will generally not work correctly with the wrapped-around value. Much of the time, programs are merely optimistic that -overflow won't happen, but this often turns out to be mistaken. When it does -happen, unexpected behavior and potentially even security bugs can result. It -should be emphasized that wrapping around on overflow is *not* a memory safety -issue in the safe subset of Rust, which greatly mitigates, but does not -eliminate, the harm that overflow bugs can cause. However, in `unsafe` blocks, -and when interfacing with existing C libraries, it can be a memory safety issue, -and furthermore, not all security bugs are memory safety bugs. By +overflow won't happen, but this frequently turns out to be mistaken. When +overflow does happen, unexpected behavior and potentially even security bugs can +result. + +It should be emphasized that wrapping around on overflow is *not* a memory +safety issue in the safe subset of Rust, which greatly mitigates, but does not +eliminate, the harm that overflow bugs can cause. However, in `unsafe` blocks +and when interfacing with existing C libraries, it *can* be a memory safety +issue; and furthermore, not all security bugs are memory safety bugs. By indiscriminately using wraparound-on-overflow semantics in every case, whether -or not it is appropriate, it becomes difficult to impossible for programmers, -compilers, and analysis tools to reliably determine where overflow is the -expected behavior, and where the possibility of it should be considered a -defect. +or not it is sensible, it becomes difficult to impossible for programmers, +compilers, and analysis tools to reliably determine in which cases overflow is +the expected behavior, and in which cases it should be considered a defect. -It is a fact that checked arithmetic poses an unacceptable performance burden in -many cases, especially in a performance-sensitive language like Rust. As such, a -perfect, one-size-fits-all solution is regrettably not possible. However, we can -make better compromises than we currently do. +In many cases, runtime overflow checks unfortunately present an unacceptable +performance burden, especially in a performance-conscious language like Rust. +As such, a perfect, one-size-fits-all solution is regrettably not possible. +However, it is very much within our power to make better compromises than we +currently do. -While in many cases, the performance cost of checking for overflow is not -acceptable, in many other cases, it is. Developers should be able to make the -tradeoff themselves in a convenient and granular way. +While the performance cost of checking for overflow is not acceptable in many +cases, in many others, it is. Developers should be able to make the tradeoff +themselves in a convenient and granular way. -The cases where wraparound on overflow is explicitly desired are comparatively -rare. The only use cases for wrapping arithmetic that are known to the author at -the time of writing are hashes, checksums, and emulation of processors with -wraparound arithmetic instructions. Therefore, it should be acceptable to not -provide symbolic operators, but require using named methods in -these cases. +The circumstances where wraparound on overflow is explicitly desired are +comparatively rare. The only use cases for wrapping arithmetic that are known to +the author at the time of writing are hashes, checksums, and emulation of +hardware instructions with wraparound semantics. Therefore, it should be +acceptable to only provide named methods for wraparound arithmetic, and not +symbolic operators. ## Goals of this proposal @@ -60,24 +64,23 @@ these cases. from where it is not. * Provide programmers with the tools they need to flexibly make the tradeoff - between higher performance and catching more mistakes. + between higher performance and more reliably catching mistakes. - * Be minimally disruptive to the language as it is and have a small surface - area. + * Be well-contained and minimally disruptive to the language as it is. -## Non-goals of this proposal +## *Non*-goals of this proposal * Make checked arithmetic fast. Let me emphasize: *the actual performance of checked arithmetic is wholly irrelevant to the content of this proposal*. It will be relevant to the decisions programmers make when employing the tools - this proposal proposes to supply them with, but it is not to the tools - themselves. + this proposal proposes to supply them with, but to the tools themselves it + is not. * Prepare for future processor architectures and/or compiler advances which may improve the performance of checked arithmetic. If there were mathematical - proof that faster checked arithmetic is impossible, the proposal would be the - same. + proof that faster checked arithmetic is impossible, the proposal would still + be the same. ## Acknowledgements and further reading @@ -91,7 +94,7 @@ On the limited use cases for wrapping arithmetic: * [Jerry Morrison on June 20][JM20] -On the value of distinguishing where overflow is valid, and where it is not: +On the value of distinguishing where overflow is valid from where it is not: * [Gregory Maxwell on June 18][GM18] * [Gregory Maxwell on June 24][GM24] @@ -127,20 +130,28 @@ In general: ## Semantics of overflow with the built-in types -Currently, the built-in arithmetic operators `+`, `-`, `*`, `/`, and `%` on the -built-in types `i8`..`i64`, `u8`..`u64`, `int`, and `uint` are defined as -wrapping around on overflow. Change this to define them, on overflow, as either -returning an unspecified result, or not returning at all (i.e. terminating -execution in some fashion, "returning bottom"), instead. +Currently, the built-in arithmetic operators `+`, `-`, `*`, `/`, and `%`, as +well as casts using `as` on the built-in types `i8`..`i64`, `u8`..`u64`, `int`, +and `uint` are defined as wrapping around on overflow. Change this to define +them, on overflow, as either returning an unspecified result, or not returning +at all (i.e. terminating execution in some fashion, "returning bottom") instead. The implication is that overflow is considered to be an abnormal circumstance, -and the programmer expects it not to happen, resp. it is her goal to make sure -that it will not. +a program error, and the programmer expects it not to happen, resp. it is her +goal to make sure that it will not. + +However, the compiler is *not* allowed to assume that overflow cannot happen, +nor to optimize based on this assumption. Where overflow or underflow can be +statically detected, the implementation is free to diagnose it with a warning +or an error at compile time, and/or to represent the overflowing value at +runtime with some form of bottom (e.g. task failure). Notes: - * In practice, the unspecified result will most likely be the wraparound - result, but in theory, it's up to the implementation. + * In theory, the implementation returns an unspecified result. In practice, + however, this will most likely be the same as the wraparound result. + Implementations should avoid unnecessarily exacerbating program errors + with further unpredictability or surprising behavior. * "Terminating execution in some fashion" will most likely mean failing the task, but the defined semantics of the types do not foreclose on other @@ -150,27 +161,34 @@ Notes: result of the operation is left unspecified, as opposed to the entire program's meaning, as in C. The programmer would not be allowed to rely on a specific, or any, result being returned on overflow, but the compiler would - also not be allowed to assume that overflow won't happen. + also not be allowed to assume that overflow won't happen and optimize based + on this assumption. +See also Appendix A. -## Scoped attributes to control checking +## Scoped attributes to control runtime checking -This depends on [RFC PR 16][16] being accepted. +This depends on [RFC 40][40] being implemented. -Introduce an `overflow_checks` attribute which can be used to turn overflow -checks on or off in a given scope. `#[overflow_checks(on)]` turns them on, -`#[overflow_checks(off)]` turns them off. The attribute can be applied to a -whole `crate`, a `mod`ule, an `fn`, or (as per [RFC PR 16][16]) a given block or -a single expression. When applied to a block, this is analogous to the +Introduce an `overflow_checks` attribute which can be used to turn runtime +overflow checks on or off in a given scope. `#[overflow_checks(on)]` turns them +on, `#[overflow_checks(off)]` turns them off. The attribute can be applied to a +whole `crate`, a `mod`ule, an `fn`, or (as per [RFC 40][40]) a given block or +a single expression. When applied to a block, this is analogous to the `checked { }` blocks of C#. As with lint attributes, an `overflow_checks` -attribute on an inner scope or item will override the effects of any -`overflow_checks` attributes on outer scopes or items. (Overflow checks can in -fact be thought of as a kind of run-time lint.) Where overflow checks are in -effect, overflow with the basic arithmetic operations on the built-in fixed-size -integer types invokes `fail!()`. Where they are not, the checks are omitted, and -the result of the operations is left unspecified (but will most likely wrap). +attribute on an inner scope or item will override the effects of any +`overflow_checks` attributes on outer scopes or items. Overflow checks can, in +fact, be thought of as a kind of run-time lint. Where overflow checks are in +effect, overflow with the basic arithmetic operations and casts on the built-in +fixed-size integer types will invoke task failure. Where they are not, the +checks are omitted, and the result of the operations is left unspecified (but +will most likely wrap). + +Significantly, turning `overflow_checks` on or off should only produce an +observable difference in the behavior of the program, beyond the time it takes +to execute, if the program has an overflow bug. -Illustration: +Illustration of use: // checks are on for this crate #![overflow_checks(on)] @@ -200,11 +218,11 @@ Illustration: ... -[16]: https://github.com/rust-lang/rfcs/pull/16 +[40]: https://github.com/rust-lang/rfcs/blob/master/active/0040-more-attributes.md ### The default -There is a significant decision to be made with respect to the default behavior +There is an important decision to be made with respect to the default behavior where neither `overflow_checks(on)` nor `off` has been explicitly specified. The author does not presume to know the correct answer, and leaves this open to debate. The following defaults are possible: @@ -286,46 +304,73 @@ The recommendation is to not use `Vec>`, but to use `Vec` and the `wrapping_`* methods directly, instead. -# Drawbacks +# Required implementation work + +## Backwards incompatible parts + + * Amend the language definition to remove the guarantee that over- and + underflow wraps around when using the built-in arithmetic operators and casts + on the built-in types. Specify that such over- and underflow counts as a + program error, and that it is up to the implementation whether to diagnose it + at compile time where possible, and whether to insert checks at runtime, or + to return an unspecified result (but without optimizing based on the + assumption that overflow cannot happen). + + * Add the `WrappingOps` trait and implementations of it for the built-in types. + + * Port existing code which relies on wraparound semantics - primarily hash + functions - to use the `wrapping_`* methods. - * Required implementation work: +Any code which does not explicitly depend on wraparound semantics, which is the +large majority of existing code, is not affected. - * Implement [RFC PR 16][16]. - * Implement the `overflow_checks` attribute. +## Backwards compatible parts - * Port existing code which relies on wraparound semantics (primarily hash - functions) to use the `wrapping_`* methods. + * Implement [RFC 40][40] for attributes on statements and expressions. + + * Implement the `overflow_checks` attribute and the `--force-overflow-checks` + compiler flag. + + * Potentially emit warnings or errors at compile time when static analysis can + show that over- or underflow would happen. + +After the backwards incompatible changes have been made, these can be done at +any point without breaking the semantics of existing programs. The only effect +would be to make it easier to discover bugs. + + +# Drawbacks * Code where `overflow_checks(off)` is in effect could end up accidentally relying on overflow. Given the relative scarcity of cases where overflow is a favorable circumstance, the risk of this happening seems minor. * Having to think about whether wraparound arithmetic is appropriate may - cause an increased cognitive burden. However, wraparound arithmetic is + cause an increased cognitive burden. However, wraparound arithmetic is almost never appropriate. Therefore, programmers should be able to keep using - the built-in integer types and to not think about it. Where wraparound + the built-in integer types and to not think about this. Where wraparound semantics are required, it is generally a specialized use case with the implementor well aware of the requirement. * The built-in types become "special": the ability to control overflow checks using scoped attributes doesn't extend to user-defined types. If you make a `struct MyNum(int)` and `impl Add for MyNum` using the native `+` operation - on `int`s, whether overflow checks happen for `MyNum` is determined by - whether `overflow_checks` is `on` or `off` where `impl Add for MyNum` is - declared, not whether they are `on` or `off` where the overloaded operators - are used. + on `int`s, whether overflow checks happen for `MyNum` is determined by + whether `overflow_checks` is `on` or `off` where `impl Add for MyNum` is + declared, not whether they are `on` or `off` where the overloaded operators + are used. The author considers this to be a serious shortcoming. *However*, it has the saving grace of being no worse than the status quo, i.e. the change is still - a Pareto-improvement. Under the status quo, neither the built-in types nor + a Pareto-improvement. Under the status quo, neither the built-in types nor user-defined types can have overflow checks controlled by scoped attributes. - Under this proposal, the situation is improved with built-in types gaining - this capability. In light of this, making further improvements, namely + Under this proposal, the situation is improved with built-in types gaining + this capability. In light of this, making further improvements, namely extending the capability to user-defined types, can be left to future work. * Someone may conduct a benchmark of Rust with overflow checks turned on, post - it to the Internet, and mislead the audience into thinking that Rust is a + it to the Internet, and mislead the audience into thinking that Rust is a slow language. @@ -333,7 +378,7 @@ the `wrapping_`* methods directly, instead. ## Do nothing for now -Defer any action until later, as suggested by: +Defer any action until later, as advocated by: * [Patrick Walton on June 22][PW22] @@ -353,11 +398,11 @@ without the `WrappingOps` trait and to avoid having unspecified results. See: * [Daniel Micay on June 24][DM24_2] -Reasons this was not pursued: Having the declared semantics of a type change -based on context is weird. It should be possible to make the choice between -turning checks `on` or `off` solely based on performance considerations. It -should be possible to distinguish cases where checking was too expensive and -where wraparound was desired. Wraparound is not usually desired. +Reasons this was not pursued: The official semantics of a type should not change +based on the context. It should be possible to make the choice between turning +checks `on` or `off` solely based on performance considerations. It should be +possible to distinguish cases where checking was too expensive from where +wraparound was desired. (Wraparound is not usually desired.) ## Different operators @@ -400,12 +445,40 @@ Reasons this was not pursued: My brain melted. :( "What should the default be where neither `overflow_checks(on)` nor `off` has been explicitly specified?", as discussed in the main text. +Instead of a single `WrappingOps` trait, should there perhaps be a separate +trait for each operation, as with the built-in arithmetic traits `Add`, `Sub`, +and so on? The only purpose of `WrappingOps` is to be implemented for the +built-in numeric types, so it's not clear that there would be a benefit to doing +so. + +C and C++ define `INT_MIN / -1` and `INT_MIN % -1` to be undefined behavior, to +make it possible to use a division and remainder instruction to compute them. +Mathematically, however, modulus can never overflow and `INT_MIN % -1` has value, +`0`. Should Rust consider these to be instances of overflow, or should it +guarantee that they return the correct mathematical result, at the cost of a +runtime branch? Division is already slow, so a branch here may be an affordable +cost. If we do this, it would obviate the need for the `wrapping_div` and +`wrapping_rem` methods, and they could be removed. This isn't intrinsically tied +to the current RFC, and could be discussed separately. + +It is not clear whether, or how, overflowing casts between types should be +provided. For casts between signed and unsigned integers of the same size, +the [`Transmute` trait for safe transmutes][transmute] would be appropriate in +the future, or the unsafe `transmute()` function in the present. For other casts +(between different sized-types, between floats and integers), special operations +might have to be provided if supporting them is desirable. + +[transmute]: https://github.com/rust-lang/rfcs/pull/91 + # Future work * Extend the ability to make use of local `overflow_checks(on|off)` attributes - to user-defined types, as discussed under Drawbacks. (The author has some - preliminary ideas, however, they are preliminary.) + to user-defined types, as discussed under Drawbacks. For newtypes of the + built-in types which simply want their arithmetic operations to forward to + the built-in ones, this could be accomplished by just adding a newtype + deriving feature, which we would eventually like to do anyways. For more + involved cases, there may not be an easy solution. * Look into adopting imprecise exceptions and a similar design to Ada's, and to what is explored in the research on AIR (As Infinitely Ranged) semantics, to @@ -422,3 +495,31 @@ been explicitly specified?", as discussed in the main text. [DM24_2]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010590.html [CZ22]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010483.html [JR23_2]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010527.html + + +# Appendix A: On the precise meaning of unspecified results. + +*In theory*, the below loop would be allowed to print "hello" any number of +times from 127 to infinity (inclusive), and may print "hello" a different number +of times on different runs of the program, and may print "hedgehogs", but may +not print "penguins". + + fn main() { + let mut i: i8 = 1; + + while i > 0 { + println!("hello"); + i = i + 1; + if i != i { println!("penguins"); } + if i + 1 != i + 1 { println!("hedgehogs"); } + } + } + +Thus, for example, LLVM's `undef` could not be used to represent the value of +`100i8 + 100`, because `undef` can be two different values in different usage +points. + +*In practice*, implementations should avoid unnecessarily surprising behavior, +and the above loop should almost always print "hello" 127 times, and nothing +else. Implementations would also be encouraged to diagnose at compile time the +fact that this loop, as well as `100i8 + 100`, will always overflow. From c2a842e9eb72ac1b54ac006338292c10fc76e189 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?G=C3=A1bor=20Lehel?= Date: Tue, 26 Aug 2014 19:25:20 +0200 Subject: [PATCH 05/18] minor changes --- ...coped-attributes-for-checked-arithmetic.md | 23 ++++++++++++++----- 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/active/0000-scoped-attributes-for-checked-arithmetic.md b/active/0000-scoped-attributes-for-checked-arithmetic.md index 7bcaaf6e468..22f0873f45e 100644 --- a/active/0000-scoped-attributes-for-checked-arithmetic.md +++ b/active/0000-scoped-attributes-for-checked-arithmetic.md @@ -114,6 +114,8 @@ In general: * [John Regehr on June 23][JR23] * [Lars Bergstrom on June 24][LB24] +Further credit is due to the commenters in the [GitHub discussion thread][GH]. + [GL18]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010363.html [GM18]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010371.html [JM20]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010410.html @@ -124,6 +126,7 @@ In general: [DM24]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010598.html [JM24]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010596.html [LB24]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010579.html +[GH]: https://github.com/rust-lang/rfcs/pull/146 # Detailed design @@ -142,16 +145,16 @@ goal to make sure that it will not. However, the compiler is *not* allowed to assume that overflow cannot happen, nor to optimize based on this assumption. Where overflow or underflow can be -statically detected, the implementation is free to diagnose it with a warning -or an error at compile time, and/or to represent the overflowing value at -runtime with some form of bottom (e.g. task failure). +statically detected, the implementation is free, and encouraged, to diagnose +it with a warning or an error at compile time, and/or to represent the +overflowing value at runtime with some form of bottom (e.g. task failure). Notes: * In theory, the implementation returns an unspecified result. In practice, however, this will most likely be the same as the wraparound result. - Implementations should avoid unnecessarily exacerbating program errors - with further unpredictability or surprising behavior. + Implementations should avoid needlessly exacerbating program errors with + additional unpredictability or surprising behavior. * "Terminating execution in some fashion" will most likely mean failing the task, but the defined semantics of the types do not foreclose on other @@ -164,6 +167,9 @@ Notes: also not be allowed to assume that overflow won't happen and optimize based on this assumption. +To state it plainly: This is for the programmer's benefit, and not the +optimizer's. + See also Appendix A. ## Scoped attributes to control runtime checking @@ -188,6 +194,11 @@ Significantly, turning `overflow_checks` on or off should only produce an observable difference in the behavior of the program, beyond the time it takes to execute, if the program has an overflow bug. +It should also be emphasized that `overflow_checks(off)` only disables *runtime* +overflow checks. Compile-time analysis can and should still be performed where +possible. Perhaps the name could be chosen to make this more obvious, such as +`runtime_overflow_checks`, but that starts to get overly verbose. + Illustration of use: // checks are on for this crate @@ -453,7 +464,7 @@ so. C and C++ define `INT_MIN / -1` and `INT_MIN % -1` to be undefined behavior, to make it possible to use a division and remainder instruction to compute them. -Mathematically, however, modulus can never overflow and `INT_MIN % -1` has value, +Mathematically, however, modulus can never overflow and `INT_MIN % -1` has value `0`. Should Rust consider these to be instances of overflow, or should it guarantee that they return the correct mathematical result, at the cost of a runtime branch? Division is already slow, so a branch here may be an affordable From 9351afb502694584094de0ed803b492345f78266 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Wed, 7 Jan 2015 17:09:09 -0500 Subject: [PATCH 06/18] Move from active to text --- {active => text}/0000-scoped-attributes-for-checked-arithmetic.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename {active => text}/0000-scoped-attributes-for-checked-arithmetic.md (100%) diff --git a/active/0000-scoped-attributes-for-checked-arithmetic.md b/text/0000-scoped-attributes-for-checked-arithmetic.md similarity index 100% rename from active/0000-scoped-attributes-for-checked-arithmetic.md rename to text/0000-scoped-attributes-for-checked-arithmetic.md From 3ece24be6f85513991f7bd672a720266f3eac66c Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Wed, 7 Jan 2015 17:09:56 -0500 Subject: [PATCH 07/18] Rename to represet new focus --- ...ributes-for-checked-arithmetic.md => 0000-integer-overflow.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename text/{0000-scoped-attributes-for-checked-arithmetic.md => 0000-integer-overflow.md} (100%) diff --git a/text/0000-scoped-attributes-for-checked-arithmetic.md b/text/0000-integer-overflow.md similarity index 100% rename from text/0000-scoped-attributes-for-checked-arithmetic.md rename to text/0000-integer-overflow.md From d4ed2d10c947dc1bd39348105f5f57d12e7e9760 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Wed, 7 Jan 2015 17:10:57 -0500 Subject: [PATCH 08/18] Edit test to move scoped attributes into future designs and mandate checking in debug mode. --- text/0000-integer-overflow.md | 246 ++++++++++++++++------------------ 1 file changed, 117 insertions(+), 129 deletions(-) diff --git a/text/0000-integer-overflow.md b/text/0000-integer-overflow.md index 22f0873f45e..ec42beb0208 100644 --- a/text/0000-integer-overflow.md +++ b/text/0000-integer-overflow.md @@ -5,16 +5,15 @@ # Summary -Change the semantics of the built-in fixed-size integer types from being defined -as wrapping around on overflow to it being considered a program error (but *not* -undefined behavior in the C sense), with whether to check for it at compile -and/or runtime being left up to the implementation. Add an attribute to allow -runtime overflow checks to be turned on or off per-scope, similarly to lints. -Add a compiler option to force checking to be on for testing and debugging -purposes. Add a `WrappingOps` trait to the standard library with operations -defined as wrapping on overflow for the limited number of cases where this is -the desired semantics, such as hash functions. - +Change the semantics of the built-in fixed-size integer types from +being defined as wrapping around on overflow to it being considered a +program error (but *not* undefined behavior in the C +sense). Implementations are *permitted* to check for overflow at any +time (statically or dynamically). Implementations are *required* to at +least check dynamically when `debug_assert!` assertions are +enabled. Add a `WrappingOps` trait to the standard library with +operations defined as wrapping on overflow for the limited number of +cases where this is the desired semantics, such as hash functions. # Motivation @@ -57,11 +56,17 @@ hardware instructions with wraparound semantics. Therefore, it should be acceptable to only provide named methods for wraparound arithmetic, and not symbolic operators. - ## Goals of this proposal * Clearly distinguish the circumstances where overflow is expected behavior from where it is not. + + * Ensure that pervasive checking for overflow is performed *at some + point* in the development cycle, even if it does not take place in + shipping code for performance reasons. The goal of this is to + prevent "lock-in" where code has a de-facto reliance on wrapping + semantics, and thus incorrectly breaks when stricter checking is + enabled. * Provide programmers with the tools they need to flexibly make the tradeoff between higher performance and more reliably catching mistakes. @@ -82,13 +87,20 @@ symbolic operators. proof that faster checked arithmetic is impossible, the proposal would still be the same. - ## Acknowledgements and further reading -Many aspects of this proposal and many of the ideas within it were influenced -and inspired by [a discussion on the rust-dev mailing list][GL18]. The author is -grateful to everyone who provided input, and would like to highlight the -following messages in particular as providing motivation for the proposal. +This RFC was [initially written by Gábor Lehel][GH] and was since +edited by Nicholas Matsakis into its current form. The changes were +primarily to introduce a requirement that overflow be checked in debug +builds and move discussion of scoped attributes to the "future +directions" section. + +Many aspects of this proposal and many of the ideas within it were +influenced and inspired by +[a discussion on the rust-dev mailing list][GL18]. The author is +grateful to everyone who provided input, and would like to highlight +the following messages in particular as providing motivation for the +proposal. On the limited use cases for wrapping arithmetic: @@ -128,26 +140,27 @@ Further credit is due to the commenters in the [GitHub discussion thread][GH]. [LB24]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010579.html [GH]: https://github.com/rust-lang/rfcs/pull/146 - # Detailed design ## Semantics of overflow with the built-in types -Currently, the built-in arithmetic operators `+`, `-`, `*`, `/`, and `%`, as -well as casts using `as` on the built-in types `i8`..`i64`, `u8`..`u64`, `int`, -and `uint` are defined as wrapping around on overflow. Change this to define -them, on overflow, as either returning an unspecified result, or not returning -at all (i.e. terminating execution in some fashion, "returning bottom") instead. +Currently, the built-in arithmetic operators `+`, `-`, `*`, `/`, and +`%`, as well as casts using `as` on the built-in types `i8`..`i64`, +`u8`..`u64`, `isize`, and `usize` are defined as wrapping around on +overflow. Change this to define them, on overflow, as either returning +an unspecified result, or task panic, depending on whether the +overflow is checked. The implication is that overflow is considered to be an abnormal circumstance, a program error, and the programmer expects it not to happen, resp. it is her goal to make sure that it will not. -However, the compiler is *not* allowed to assume that overflow cannot happen, -nor to optimize based on this assumption. Where overflow or underflow can be -statically detected, the implementation is free, and encouraged, to diagnose -it with a warning or an error at compile time, and/or to represent the -overflowing value at runtime with some form of bottom (e.g. task failure). +However, the compiler is *not* allowed to assume that overflow cannot +happen, nor to optimize based on this assumption. Where overflow or +underflow can be statically detected, the implementation is free, and +encouraged, to diagnose it with an error at compile time, and/or +to represent the overflowing value at runtime with some form of bottom +(e.g. task failure). Notes: @@ -156,10 +169,6 @@ Notes: Implementations should avoid needlessly exacerbating program errors with additional unpredictability or surprising behavior. - * "Terminating execution in some fashion" will most likely mean failing the - task, but the defined semantics of the types do not foreclose on other - possibilities. - * Most importantly: this is **not** undefined behavior in the C sense. Only the result of the operation is left unspecified, as opposed to the entire program's meaning, as in C. The programmer would not be allowed to rely on a @@ -172,99 +181,25 @@ optimizer's. See also Appendix A. -## Scoped attributes to control runtime checking - -This depends on [RFC 40][40] being implemented. - -Introduce an `overflow_checks` attribute which can be used to turn runtime -overflow checks on or off in a given scope. `#[overflow_checks(on)]` turns them -on, `#[overflow_checks(off)]` turns them off. The attribute can be applied to a -whole `crate`, a `mod`ule, an `fn`, or (as per [RFC 40][40]) a given block or -a single expression. When applied to a block, this is analogous to the -`checked { }` blocks of C#. As with lint attributes, an `overflow_checks` -attribute on an inner scope or item will override the effects of any -`overflow_checks` attributes on outer scopes or items. Overflow checks can, in -fact, be thought of as a kind of run-time lint. Where overflow checks are in -effect, overflow with the basic arithmetic operations and casts on the built-in -fixed-size integer types will invoke task failure. Where they are not, the -checks are omitted, and the result of the operations is left unspecified (but -will most likely wrap). - -Significantly, turning `overflow_checks` on or off should only produce an -observable difference in the behavior of the program, beyond the time it takes -to execute, if the program has an overflow bug. - -It should also be emphasized that `overflow_checks(off)` only disables *runtime* -overflow checks. Compile-time analysis can and should still be performed where -possible. Perhaps the name could be chosen to make this more obvious, such as -`runtime_overflow_checks`, but that starts to get overly verbose. - -Illustration of use: - - // checks are on for this crate - #![overflow_checks(on)] - - // but they are off for this module - #[overflow_checks(off)] - mod some_stuff { - - // but they are on for this function - #[overflow_checks(on)] - fn do_thing() { - ... - - // but they are off for this block - #[overflow_checks(off)] { - ... - // but they are on for this expression - let n = #[overflow_checks(on)] (a * b + c); - ... - } - - ... - } - - ... - } - - ... - -[40]: https://github.com/rust-lang/rfcs/blob/master/active/0040-more-attributes.md - ### The default -There is an important decision to be made with respect to the default behavior -where neither `overflow_checks(on)` nor `off` has been explicitly specified. The -author does not presume to know the correct answer, and leaves this open to -debate. The following defaults are possible: - - 1. The default is `on`. This means that the default is to catch more mistakes. - - 2. The default is `off`. This means that the default is to be faster. (This - happens to be the current "default".) - - 3. There is no default, and a decision is forced. If the programmer neglects to - explicitly specify a behavior, the compiler will bail out and ask her to - specify one. +In general, implementations are always free to insert dynamic checks +for overflow if they so choose. However, when running in debug mode +(in other words, whenever a `debug_assert!` would be compiled in), +they are *required* to emit code to perform dynamic checks on all +arithmetic operations that may potentially overflow. - 4. Combination of (1) and (3): The default is `on`, but the compiler emits a - warning when falling back to the default behavior. - - 5. Combination of (2) and (3): The default is `off`, but the compiler emits a - warning when falling back to the default behavior. - - -## A debugging switch to force checking - -The programmer has the option to turn `overflow_checks(off)` due to performance -considerations. However, when testing or debugging the program, for instance -when tracking down a difficult bug, it may be desired to throw performance to -the wind and enable as many checks as possible. For this purpose, provide a -compiler option, e.g. `--force-overflow-checks`, which causes overflow checks to -be considered `on` even where an attribute has turned them `off`. This is -somewhat analogous to the behavior of our current `--ndebug` flag and -`debug_assert!` macros. +The goal of this rule is to ensure that, during debugging and normal +development, overflow detection is on, so that users can be alerted to +potential overflow (and, in particular, for code where overflow is +expected and normal, they will be immediately guided to use the +`WrappingOps` traits introduced below). However, because these checks +will be compiled out whenever an optimized build is produced, final +code wilil not pay a performance penalty. +In the future, we may add additional means to control when overflow is +checked, such as scoped attributes or a global, independent +compile-time switch. ## `WrappingOps` trait for explicit wrapping arithmetic @@ -280,8 +215,8 @@ by providing the following trait and impls in the `prelude`: fn wrapping_rem(self, rhs: Self) -> Self; } - impl WrappingOps for int - impl WrappingOps for uint + impl WrappingOps for isize + impl WrappingOps for usize impl WrappingOps for i8 impl WrappingOps for u8 impl WrappingOps for i16 @@ -293,7 +228,6 @@ by providing the following trait and impls in the `prelude`: These are implemented to wrap around on overflow unconditionally. - ### `Wrapping` type for convenience For convenience, also provide a `Wrapping` newtype for which the operator @@ -311,9 +245,6 @@ overloads are implemented using the `WrappingOps` trait: Note that this is only for potential convenience. The type-based approach has the drawback that e.g. `Vec` and `Vec>` are incompatible types. -The recommendation is to not use `Vec>`, but to use `Vec` and -the `wrapping_`* methods directly, instead. - # Required implementation work @@ -335,7 +266,6 @@ the `wrapping_`* methods directly, instead. Any code which does not explicitly depend on wraparound semantics, which is the large majority of existing code, is not affected. - ## Backwards compatible parts * Implement [RFC 40][40] for attributes on statements and expressions. @@ -384,8 +314,7 @@ would be to make it easier to discover bugs. it to the Internet, and mislead the audience into thinking that Rust is a slow language. - -# Alternatives +# Alternatives and possible future directions ## Do nothing for now @@ -415,6 +344,65 @@ checks `on` or `off` solely based on performance considerations. It should be possible to distinguish cases where checking was too expensive from where wraparound was desired. (Wraparound is not usually desired.) +## Scoped attributes to control runtime checking + +This depends on [RFC 40][40] being implemented. + +Introduce an `overflow_checks` attribute which can be used to turn runtime +overflow checks on or off in a given scope. `#[overflow_checks(on)]` turns them +on, `#[overflow_checks(off)]` turns them off. The attribute can be applied to a +whole `crate`, a `mod`ule, an `fn`, or (as per [RFC 40][40]) a given block or +a single expression. When applied to a block, this is analogous to the +`checked { }` blocks of C#. As with lint attributes, an `overflow_checks` +attribute on an inner scope or item will override the effects of any +`overflow_checks` attributes on outer scopes or items. Overflow checks can, in +fact, be thought of as a kind of run-time lint. Where overflow checks are in +effect, overflow with the basic arithmetic operations and casts on the built-in +fixed-size integer types will invoke task failure. Where they are not, the +checks are omitted, and the result of the operations is left unspecified (but +will most likely wrap). + +Significantly, turning `overflow_checks` on or off should only produce an +observable difference in the behavior of the program, beyond the time it takes +to execute, if the program has an overflow bug. + +It should also be emphasized that `overflow_checks(off)` only disables *runtime* +overflow checks. Compile-time analysis can and should still be performed where +possible. Perhaps the name could be chosen to make this more obvious, such as +`runtime_overflow_checks`, but that starts to get overly verbose. + +Illustration of use: + + // checks are on for this crate + #![overflow_checks(on)] + + // but they are off for this module + #[overflow_checks(off)] + mod some_stuff { + + // but they are on for this function + #[overflow_checks(on)] + fn do_thing() { + ... + + // but they are off for this block + #[overflow_checks(off)] { + ... + // but they are on for this expression + let n = #[overflow_checks(on)] (a * b + c); + ... + } + + ... + } + + ... + } + + ... + +[40]: https://github.com/rust-lang/rfcs/blob/master/active/0040-more-attributes.md + ## Different operators Have the usual arithmetic operators check for overflow, and introduce a new set From e1af4982980696061a03c871f1e864e6dae5c871 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Thu, 8 Jan 2015 09:59:41 -0500 Subject: [PATCH 09/18] Add a paragraph about bit shifts. --- text/0000-integer-overflow.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/text/0000-integer-overflow.md b/text/0000-integer-overflow.md index ec42beb0208..54e9d94a3d2 100644 --- a/text/0000-integer-overflow.md +++ b/text/0000-integer-overflow.md @@ -151,6 +151,10 @@ overflow. Change this to define them, on overflow, as either returning an unspecified result, or task panic, depending on whether the overflow is checked. +The semantics of bitshifts are also changed. The number of bits to +shift by must be within the interval +`[0..N)` where `N` is the number of bits in the value being shifted, or else the result is panic/undefined, as above. + The implication is that overflow is considered to be an abnormal circumstance, a program error, and the programmer expects it not to happen, resp. it is her goal to make sure that it will not. From 630dd70a51c0c7e166be78cd3bc8f1247664db28 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 9 Jan 2015 18:32:47 -0500 Subject: [PATCH 10/18] Clarify that overflow panics are not required to be precise, and that panics may occur only on some branches. --- text/0000-integer-overflow.md | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/text/0000-integer-overflow.md b/text/0000-integer-overflow.md index 54e9d94a3d2..bd7d660c27c 100644 --- a/text/0000-integer-overflow.md +++ b/text/0000-integer-overflow.md @@ -152,8 +152,9 @@ an unspecified result, or task panic, depending on whether the overflow is checked. The semantics of bitshifts are also changed. The number of bits to -shift by must be within the interval -`[0..N)` where `N` is the number of bits in the value being shifted, or else the result is panic/undefined, as above. +shift by must be within the interval `0..N` (using Rust semantics) +where `N` is the number of bits in the value being shifted, or else +the result is panic/undefined, as above. The implication is that overflow is considered to be an abnormal circumstance, a program error, and the programmer expects it not to happen, resp. it is her @@ -172,20 +173,34 @@ Notes: however, this will most likely be the same as the wraparound result. Implementations should avoid needlessly exacerbating program errors with additional unpredictability or surprising behavior. - * Most importantly: this is **not** undefined behavior in the C sense. Only the result of the operation is left unspecified, as opposed to the entire program's meaning, as in C. The programmer would not be allowed to rely on a specific, or any, result being returned on overflow, but the compiler would also not be allowed to assume that overflow won't happen and optimize based on this assumption. + * Even when checking for an overflow, compilers are not required to + panic at the precise point of overflow. They are free to coalesce + checks from adjacent operations, for example, or otherwise move the + point of panic around. However, when checks are enabled (e.g., + during a debug build, by default), then any operation which + overflows must result in a panic some finite number of steps after + the operation that caused the overflow. (For example, the panic + cannot be delayed until after a loop, unless the number of + iterations in that loop can be statically bounded.) + * When overflow checking is not explicitly enabled, the matter of + whether an overflow results in panic or an undefined value is not + required to be consistent across executions. This means that, for + example, the compiler could insert code which results in an + overflow only if the value that was produced is actually consumed + (which may occur on only one arm of a branch). To state it plainly: This is for the programmer's benefit, and not the optimizer's. See also Appendix A. -### The default +## The default In general, implementations are always free to insert dynamic checks for overflow if they so choose. However, when running in debug mode From 6dbbc70637c3ea550d0458bc7da65348462302f2 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Thu, 29 Jan 2015 14:50:14 -0500 Subject: [PATCH 11/18] Substantial rework of the text: 1. Clarify the aims of the RFC and generally streamline the text. 2. In the event that no panic occurs, define the results of overflow. 3. Be less generous about the compiler's ability to defer panics. 4. Move the wrapping operations out of the prelude and into `std::num`. 5. Elaborate on the objections raised to the previous draft and respond to them. --- text/0000-integer-overflow.md | 650 +++++++++++++++++----------------- 1 file changed, 318 insertions(+), 332 deletions(-) diff --git a/text/0000-integer-overflow.md b/text/0000-integer-overflow.md index bd7d660c27c..b68810fcee7 100644 --- a/text/0000-integer-overflow.md +++ b/text/0000-integer-overflow.md @@ -17,196 +17,133 @@ cases where this is the desired semantics, such as hash functions. # Motivation -The semantics of the basic arithmetic operators and casts on the built-in -fixed-size integer types are currently defined to wrap around on overflow. This -is is well-defined behavior, so it is already an improvement over C. Yet we -should avoid falling into the trap of low expectations. - -In the large majority of cases, wrapping around on overflow is not a useful or -appropriate semantics: programs will generally not work correctly with the -wrapped-around value. Much of the time, programs are merely optimistic that -overflow won't happen, but this frequently turns out to be mistaken. When -overflow does happen, unexpected behavior and potentially even security bugs can -result. - -It should be emphasized that wrapping around on overflow is *not* a memory -safety issue in the safe subset of Rust, which greatly mitigates, but does not -eliminate, the harm that overflow bugs can cause. However, in `unsafe` blocks -and when interfacing with existing C libraries, it *can* be a memory safety -issue; and furthermore, not all security bugs are memory safety bugs. By -indiscriminately using wraparound-on-overflow semantics in every case, whether -or not it is sensible, it becomes difficult to impossible for programmers, -compilers, and analysis tools to reliably determine in which cases overflow is -the expected behavior, and in which cases it should be considered a defect. - -In many cases, runtime overflow checks unfortunately present an unacceptable -performance burden, especially in a performance-conscious language like Rust. -As such, a perfect, one-size-fits-all solution is regrettably not possible. -However, it is very much within our power to make better compromises than we -currently do. - -While the performance cost of checking for overflow is not acceptable in many -cases, in many others, it is. Developers should be able to make the tradeoff -themselves in a convenient and granular way. - -The circumstances where wraparound on overflow is explicitly desired are -comparatively rare. The only use cases for wrapping arithmetic that are known to -the author at the time of writing are hashes, checksums, and emulation of -hardware instructions with wraparound semantics. Therefore, it should be -acceptable to only provide named methods for wraparound arithmetic, and not -symbolic operators. - -## Goals of this proposal - - * Clearly distinguish the circumstances where overflow is expected behavior - from where it is not. +Numeric overflow prevents a difficult situation. On the one hand, +overflow (and [underflow]) is known to be a common source of error in +other languages. Rust, at least, does not have to worry about memory +safety violations, but it is still possible for overflow to lead to +bugs. Moreover, Rust's safety guarantees do not apply to `unsafe` +code, which carries the +[same risks as C code when it comes to overflow][phrack]. Unfortunately, +banning overflow outright is not feasible at this time. Detecting +overflow statically is not practical, and detecting it dynamically can +be costly. Therefore, we have to steer a middle ground. + +[phrack]: http://phrack.org/issues/60/10.html#article +[underflow]: http://google-styleguide.googlecode.com/svn/trunk/cppguide.html#Integer_Types + +The RFC has several major goals: + +1. Ensure that code which intentionally uses wrapping semantics is + clearly identified. +2. Help users to identify overflow problems and help those who wish to + be careful about overflow to do so. +3. Ensure that users who wish to detect overflow can safely enable + overflow checks and dynamic analysis, both on their code and on + libraries they use, with a minimal risk of "false positives" + (intentional overflows leading to a panic). +4. To the extent possible, leave room in the future to move towards + universal overflow checking if it becomes feasible. This may require + opt-in from end-users. - * Ensure that pervasive checking for overflow is performed *at some - point* in the development cycle, even if it does not take place in - shipping code for performance reasons. The goal of this is to - prevent "lock-in" where code has a de-facto reliance on wrapping - semantics, and thus incorrectly breaks when stricter checking is - enabled. - - * Provide programmers with the tools they need to flexibly make the tradeoff - between higher performance and more reliably catching mistakes. - - * Be well-contained and minimally disruptive to the language as it is. - - -## *Non*-goals of this proposal - - * Make checked arithmetic fast. Let me emphasize: *the actual performance of - checked arithmetic is wholly irrelevant to the content of this proposal*. It - will be relevant to the decisions programmers make when employing the tools - this proposal proposes to supply them with, but to the tools themselves it - is not. - - * Prepare for future processor architectures and/or compiler advances which may - improve the performance of checked arithmetic. If there were mathematical - proof that faster checked arithmetic is impossible, the proposal would still - be the same. - -## Acknowledgements and further reading - -This RFC was [initially written by Gábor Lehel][GH] and was since -edited by Nicholas Matsakis into its current form. The changes were -primarily to introduce a requirement that overflow be checked in debug -builds and move discussion of scoped attributes to the "future -directions" section. - -Many aspects of this proposal and many of the ideas within it were -influenced and inspired by -[a discussion on the rust-dev mailing list][GL18]. The author is -grateful to everyone who provided input, and would like to highlight -the following messages in particular as providing motivation for the -proposal. - -On the limited use cases for wrapping arithmetic: - - * [Jerry Morrison on June 20][JM20] - -On the value of distinguishing where overflow is valid from where it is not: - - * [Gregory Maxwell on June 18][GM18] - * [Gregory Maxwell on June 24][GM24] - * [Robert O'Callahan on June 24][ROC24] - * [Jerry Morrison on June 24][JM24] - -The idea of scoped attributes: - - * [Daniel Micay on June 23][DM23] - -On the drawbacks of a type-based approach: - - * [Daniel Micay on June 24][DM24] - -In general: - - * [John Regehr on June 23][JR23] - * [Lars Bergstrom on June 24][LB24] - -Further credit is due to the commenters in the [GitHub discussion thread][GH]. - -[GL18]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010363.html -[GM18]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010371.html -[JM20]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010410.html -[DM23]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010566.html -[JR23]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010558.html -[GM24]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010580.html -[ROC24]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010602.html -[DM24]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010598.html -[JM24]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010596.html -[LB24]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010579.html -[GH]: https://github.com/rust-lang/rfcs/pull/146 +To that end the RFC proposes two mechanisms: + +1. Optional, dynamic overflow checking. Ordinary arithmetic operations + (e.g., `a+b`) would *optionally* check for overflow. If checking is + enabled, and an overflow occurs, a thread panic will be + signaled. Specific intrinsics and library support are provided to + permit either explicit overflow checks or explicit wrapping. +2. Overflow checking would be, by default, tied to debug assertions + (`debug_assert!`). It can be seen as analogous to a debug + assertion: an important safety check that is too expensive to + perform on all code. + +We expect that additional and finer-grained mechanisms for enabling +overflows will be added in the future. One easy option is a +command-line switch to enable overflow checking universally or within +specific crates. Another option might be lexically scoped annotations +to enable overflow (or perhaps disable) checking in specific +blocks. Neither mechanism is detailed in this RFC at this time. + +## Why tie overflow checking to debug assertions + +The reasoning behind connecting overflow checking and debug assertion +is that it ensures that pervasive checking for overflow is performed +*at some point* in the development cycle, even if it does not take +place in shipping code for performance reasons. The goal of this is to +prevent "lock-in" where code has a de-facto reliance on wrapping +semantics, and thus incorrectly breaks when stricter checking is +enabled. + +We would like to allow people to switch "pervasive" overflow checks on +by default, for example. However, if the default is not to check for +overflow, then it seems likely that a pervasive check like that could +not be used, because libraries are sure to come to rely on wrapping +semantics, even if accidentally. + +By making the default for debugging code be checked overflow, we help +ensure that users will encounter overflow errors in practice, and thus +become aware that overflow in Rust is not the norm. It will also help +debug simple errors, like signed underflow leading to an infinite +loop. # Detailed design -## Semantics of overflow with the built-in types - -Currently, the built-in arithmetic operators `+`, `-`, `*`, `/`, and -`%`, as well as casts using `as` on the built-in types `i8`..`i64`, -`u8`..`u64`, `isize`, and `usize` are defined as wrapping around on -overflow. Change this to define them, on overflow, as either returning -an unspecified result, or task panic, depending on whether the -overflow is checked. - -The semantics of bitshifts are also changed. The number of bits to -shift by must be within the interval `0..N` (using Rust semantics) -where `N` is the number of bits in the value being shifted, or else -the result is panic/undefined, as above. - -The implication is that overflow is considered to be an abnormal circumstance, -a program error, and the programmer expects it not to happen, resp. it is her -goal to make sure that it will not. - -However, the compiler is *not* allowed to assume that overflow cannot -happen, nor to optimize based on this assumption. Where overflow or -underflow can be statically detected, the implementation is free, and -encouraged, to diagnose it with an error at compile time, and/or -to represent the overflowing value at runtime with some form of bottom -(e.g. task failure). - -Notes: - - * In theory, the implementation returns an unspecified result. In practice, - however, this will most likely be the same as the wraparound result. - Implementations should avoid needlessly exacerbating program errors with - additional unpredictability or surprising behavior. - * Most importantly: this is **not** undefined behavior in the C sense. Only the - result of the operation is left unspecified, as opposed to the entire - program's meaning, as in C. The programmer would not be allowed to rely on a - specific, or any, result being returned on overflow, but the compiler would - also not be allowed to assume that overflow won't happen and optimize based - on this assumption. - * Even when checking for an overflow, compilers are not required to - panic at the precise point of overflow. They are free to coalesce - checks from adjacent operations, for example, or otherwise move the - point of panic around. However, when checks are enabled (e.g., - during a debug build, by default), then any operation which - overflows must result in a panic some finite number of steps after - the operation that caused the overflow. (For example, the panic - cannot be delayed until after a loop, unless the number of - iterations in that loop can be statically bounded.) - * When overflow checking is not explicitly enabled, the matter of - whether an overflow results in panic or an undefined value is not - required to be consistent across executions. This means that, for - example, the compiler could insert code which results in an - overflow only if the value that was produced is actually consumed - (which may occur on only one arm of a branch). - -To state it plainly: This is for the programmer's benefit, and not the -optimizer's. - -See also Appendix A. - -## The default - -In general, implementations are always free to insert dynamic checks -for overflow if they so choose. However, when running in debug mode -(in other words, whenever a `debug_assert!` would be compiled in), -they are *required* to emit code to perform dynamic checks on all -arithmetic operations that may potentially overflow. +## Arithmetic operations with error conditions + +There are various operations which can sometimes produce error +conditions (detailed below). Typically these error conditions +correspond to under/overflow but not exclusively. It is the +programmers responsibility to avoid these error conditions: any +failure to do so can be considered a bug, and hence can be flagged by +a static/dynamic analysis tools as an error. This is largerly a +semantic distinction, though. + +The result of an error condition depends upon the state of overflow +checking, which can be either *enabled* or *optional* (this RFC does +not describe a way to disable overflow checking completely). If +overflow checking is *enabled*, then an error condition always results +in a panic. For efficiency reasons, this panic may be delayed over +some number of pure operations, as described below. + +If overflow checking is *optional*, then an error condition may result +in a panic, but the erroneous operation may also simply produce a +value. The permitted results for each error condition are specified +below. When checking is optional, panics are not required to be +consistent across executions. This means that, for example, the +compiler could insert code which results in an overflow only if the +value that was produced is actually consumed (which may occur on only +one arm of a branch). + +In the future, we may add some way to explicit *disable* overflow +checking (probably in a scoped fashion). In that case, the result of +each error condition would simply be the same as the optional state +when no panic occurs. + +The error conditions that can arise, and their defined results, are as +follows. In all cases, the defined results are the same as the defined +results today. The only change is that now a panic may result. + +- The operations `+`, `-`, `*`, `/`, and `%` can underflow and overflow. If no panic occurs, + the result of such an operation is defined to be the same as wrapping. +- When truncating, the casting operation `as` can overflow if the truncated bits contain + non-zero values. If no panic occurs, the result of such an operation is defined to + be the same as wrapping. +- Shift operations (`<<`, `>>`) can shift a value of width `N` by more + than `N` bits. If no panic occurs, the result of such an operation + is an *undefined value* (note that this is not the same as + *undefined behavior* in C). + - The reason that there is no defined result here is because + different CPUs have different behavior, and we wish to be able to translate shifting + operations directly to the corresponding machine instructions. + +## Enabling overflow checking + +Compilers should present a command-line option to enable overflow +checking universally. Additionally, when building in a default "debug" +configuration (i.e., whenever `debug_assert` would be enabled), +overflow checking should be enabled by default, unless the user +explicitly requests otherwise. The precise control of these settings +is not detailed in this RFC. The goal of this rule is to ensure that, during debugging and normal development, overflow detection is on, so that users can be alerted to @@ -220,37 +157,52 @@ In the future, we may add additional means to control when overflow is checked, such as scoped attributes or a global, independent compile-time switch. -## `WrappingOps` trait for explicit wrapping arithmetic +## Delayed panics -For those use cases where explicit wraparound on overflow is required, such as -hash functions, we must provide operations with such semantics. Accomplish this -by providing the following trait and impls in the `prelude`: +If an error condition should occur and a thread panic should result, +the compiler is not required to signal the panic at the precise point +of overflow. It is free to coalesce checks from adjacent pure +operations. Panics may never be delayed across an unsafe block nor +delayed by an indefinite number of steps, however. The precise +definition of a pure operation can be hammered out over time, but the +intention here is that, at minimum, overflow checks for adjacent +numeric operations like `a+b-c` can be coallesced into a single check. - pub trait WrappingOps { - fn wrapping_add(self, rhs: Self) -> Self; - fn wrapping_sub(self, rhs: Self) -> Self; - fn wrapping_mul(self, rhs: Self) -> Self; - fn wrapping_div(self, rhs: Self) -> Self; - fn wrapping_rem(self, rhs: Self) -> Self; - } +## `WrappingOps` trait for explicit wrapping arithmetic - impl WrappingOps for isize - impl WrappingOps for usize - impl WrappingOps for i8 - impl WrappingOps for u8 - impl WrappingOps for i16 - impl WrappingOps for u16 - impl WrappingOps for i32 - impl WrappingOps for u32 - impl WrappingOps for i64 - impl WrappingOps for u64 +For those use cases where explicit wraparound on overflow is required, +such as hash functions, we must provide operations with such +semantics. Accomplish this by providing the following trait and impls +in the `std::num` module. + +```rust +pub trait WrappingOps { + fn wrapping_add(self, rhs: Self) -> Self; + fn wrapping_sub(self, rhs: Self) -> Self; + fn wrapping_mul(self, rhs: Self) -> Self; + fn wrapping_div(self, rhs: Self) -> Self; + fn wrapping_rem(self, rhs: Self) -> Self; +} + +impl WrappingOps for isize +impl WrappingOps for usize +impl WrappingOps for i8 +impl WrappingOps for u8 +impl WrappingOps for i16 +impl WrappingOps for u16 +impl WrappingOps for i32 +impl WrappingOps for u32 +impl WrappingOps for i64 +impl WrappingOps for u64 +``` These are implemented to wrap around on overflow unconditionally. ### `Wrapping` type for convenience -For convenience, also provide a `Wrapping` newtype for which the operator -overloads are implemented using the `WrappingOps` trait: +For convenience, the `std::num` module also provides a `Wrapping` +newtype for which the operator overloads are implemented using the +`WrappingOps` trait: pub struct Wrapping(pub T); @@ -265,73 +217,78 @@ overloads are implemented using the `WrappingOps` trait: Note that this is only for potential convenience. The type-based approach has the drawback that e.g. `Vec` and `Vec>` are incompatible types. -# Required implementation work - -## Backwards incompatible parts - - * Amend the language definition to remove the guarantee that over- and - underflow wraps around when using the built-in arithmetic operators and casts - on the built-in types. Specify that such over- and underflow counts as a - program error, and that it is up to the implementation whether to diagnose it - at compile time where possible, and whether to insert checks at runtime, or - to return an unspecified result (but without optimizing based on the - assumption that overflow cannot happen). - - * Add the `WrappingOps` trait and implementations of it for the built-in types. - - * Port existing code which relies on wraparound semantics - primarily hash - functions - to use the `wrapping_`* methods. - -Any code which does not explicitly depend on wraparound semantics, which is the -large majority of existing code, is not affected. - -## Backwards compatible parts - - * Implement [RFC 40][40] for attributes on statements and expressions. - - * Implement the `overflow_checks` attribute and the `--force-overflow-checks` - compiler flag. - - * Potentially emit warnings or errors at compile time when static analysis can - show that over- or underflow would happen. - -After the backwards incompatible changes have been made, these can be done at -any point without breaking the semantics of existing programs. The only effect -would be to make it easier to discover bugs. - - # Drawbacks - * Code where `overflow_checks(off)` is in effect could end up accidentally - relying on overflow. Given the relative scarcity of cases where overflow is a - favorable circumstance, the risk of this happening seems minor. - - * Having to think about whether wraparound arithmetic is appropriate may - cause an increased cognitive burden. However, wraparound arithmetic is - almost never appropriate. Therefore, programmers should be able to keep using - the built-in integer types and to not think about this. Where wraparound - semantics are required, it is generally a specialized use case with the - implementor well aware of the requirement. - - * The built-in types become "special": the ability to control overflow checks - using scoped attributes doesn't extend to user-defined types. If you make a - `struct MyNum(int)` and `impl Add for MyNum` using the native `+` operation - on `int`s, whether overflow checks happen for `MyNum` is determined by - whether `overflow_checks` is `on` or `off` where `impl Add for MyNum` is - declared, not whether they are `on` or `off` where the overloaded operators - are used. - - The author considers this to be a serious shortcoming. *However*, it has the - saving grace of being no worse than the status quo, i.e. the change is still - a Pareto-improvement. Under the status quo, neither the built-in types nor - user-defined types can have overflow checks controlled by scoped attributes. - Under this proposal, the situation is improved with built-in types gaining - this capability. In light of this, making further improvements, namely - extending the capability to user-defined types, can be left to future work. - - * Someone may conduct a benchmark of Rust with overflow checks turned on, post - it to the Internet, and mislead the audience into thinking that Rust is a - slow language. +**Making choices is hard.** Having to think about whether wraparound +arithmetic is appropriate may cause an increased cognitive +burden. However, wraparound arithmetic is almost never +appropriate. Therefore, programmers should be able to keep using the +built-in integer types and to not think about this. Where wraparound +semantics are required, it is generally a specialized use case with +the implementor well aware of the requirement. + +**Loss of additive commutativity and benign overflows.** In some +cases, overflow behavior can be benign. For example, given an +expression like `a+b-c`, intermediate overflows are not harmful so +long as the final result is within the range of the integral type. To +take advantage of this property, code would have to be written to use +the wrapping constructs, such as `a.wrapping_add(b).wrapping_sub(c)`. +However, this drawback is counterbalanced by the large number of +arithmetic expressions which do not have the same behavior when +overflow occurs. A common example is `(max+min)/2`, which is a typical +ingredient for [binary searches and the like][BS] and can lead to very +surprising behavior. Moreover, the use of `wrapping_add` and +`wrapping_sub` to highlight the fact that the intermediate result may +overflow seems potentially useful to an end-reader. + +[BS]: http://googleresearch.blogspot.com/2006/06/extra-extra-read-all-about-it-nearly.html + +**Danger of triggering additional panics from within unsafe code.** +This proposal creates more possibility for panics to occur, at least +when checks are enabled. As usual, a panic at an inopportune time can +lead to bugs if code is not exception safe. This is particularly +worrisome in unsafe code, where crucial safety guarantees can be +violated. However, this danger already exists, as there are numerous +ways to trigger a panic, and hence unsafe code must be written with +this in mind. It seems like the best advice is for unsafe code to +eschew the plain `+` and `-` operators, and instead prefer explicit +checked or wrapping operations as appropriate (this would be a good +opportunity for a lint). Furthermore, the danger of an unexpected +panic occurring in unsafe code must be weighed against the danger of a +(silent) overflow, which can also lead to unsafety. + +**Divergence of debug and optimized code.** The proposal here causes +additional divergence of debug and optimized code, since optimized +code will not include overflow checking. It would therefore be +recommended that robust applications run tests both with and without +optimizations (and debug assertions). That said, this state of affairs +already exists. First, the use of `debug_assert!` causes +debug/optimized code to diverge, but also, optimizations are known to +cause non-trivial changes in behavior. For example, recursive (but +pure) functions may be optimized away entirely by LLVM. Therefore, it +always makes sense to run tests in both modes. This situation is not +unique to Rust; most major projects do something similar. Moreover, in +most languages, `debug_assert!` is in fact the only (or at least +predominant) kind of of assertion, and hence the need to run tests +both with and without assertions enabled is even stronger. + +**Benchmarking.** Someone may conduct a benchmark of Rust with +overflow checks turned on, post it to the Internet, and mislead the +audience into thinking that Rust is a slow language. The choice of +defaults minimizes this risk, however, since doing an optimized build +in cargo (which ought to be a prerequisite for any benchmark) also +disables debug assertions (or ought to). + +**Impact of overflow checking on optimization.** In addition to the +direct overhead of checking for overflow, there is some additional +overhead when checks are enabled because compilers may have to forego +other optimizations or code motion that might have been legal. This +concern seems minimal since, in optimized builds, overflow checking +will be considered *optional* and hence the compiler is free to ignore +it. Certainly if we ever decided to change the default for overflow +checking from *optional* to *enabled* in optimized builds, we would +want to measure carefully and likely include some means of disabling +checks in particularly hot paths. # Alternatives and possible future directions @@ -349,36 +306,30 @@ needlessly proliferating types. Given the paucity of circumstances where wraparound semantics is appropriate, having it be the default is defensible only if better options aren't available. -## Checks off means wrapping on - -Where overflow checks are turned off, instead of defining overflow as returning -an unspecified result, define it to wrap around. This would allow us to do -without the `WrappingOps` trait and to avoid having unspecified results. See: - - * [Daniel Micay on June 24][DM24_2] - -Reasons this was not pursued: The official semantics of a type should not change -based on the context. It should be possible to make the choice between turning -checks `on` or `off` solely based on performance considerations. It should be -possible to distinguish cases where checking was too expensive from where -wraparound was desired. (Wraparound is not usually desired.) - ## Scoped attributes to control runtime checking -This depends on [RFC 40][40] being implemented. - -Introduce an `overflow_checks` attribute which can be used to turn runtime -overflow checks on or off in a given scope. `#[overflow_checks(on)]` turns them -on, `#[overflow_checks(off)]` turns them off. The attribute can be applied to a -whole `crate`, a `mod`ule, an `fn`, or (as per [RFC 40][40]) a given block or -a single expression. When applied to a block, this is analogous to the -`checked { }` blocks of C#. As with lint attributes, an `overflow_checks` -attribute on an inner scope or item will override the effects of any -`overflow_checks` attributes on outer scopes or items. Overflow checks can, in -fact, be thought of as a kind of run-time lint. Where overflow checks are in -effect, overflow with the basic arithmetic operations and casts on the built-in -fixed-size integer types will invoke task failure. Where they are not, the -checks are omitted, and the result of the operations is left unspecified (but +The [original RFC][GH] proposed a system of scoped attributes for +enabling/disabling overflow checking. Nothing in the current RFC +precludes us from going in this direction in the future. Rather, this +RFC is attempting to answer the question (left unanswered in the +original RFC) of what the behavior ought to be when no attribute is in +scope. + +The proposal for scoped attributes in the original RFC was as follows. +Introduce an `overflow_checks` attribute which can be used to turn +runtime overflow checks on or off in a given +scope. `#[overflow_checks(on)]` turns them on, +`#[overflow_checks(off)]` turns them off. The attribute can be applied +to a whole `crate`, a `mod`ule, an `fn`, or (as per [RFC 40][40]) a +given block or a single expression. When applied to a block, this is +analogous to the `checked { }` blocks of C#. As with lint attributes, +an `overflow_checks` attribute on an inner scope or item will override +the effects of any `overflow_checks` attributes on outer scopes or +items. Overflow checks can, in fact, be thought of as a kind of +run-time lint. Where overflow checks are in effect, overflow with the +basic arithmetic operations and casts on the built-in fixed-size +integer types will invoke task failure. Where they are not, the checks +are omitted, and the result of the operations is left unspecified (but will most likely wrap). Significantly, turning `overflow_checks` on or off should only produce an @@ -422,6 +373,21 @@ Illustration of use: [40]: https://github.com/rust-lang/rfcs/blob/master/active/0040-more-attributes.md +## Checks off means wrapping on + +If we adopted a model of overflow checks, one could use an explicit +request to turn overflow checks *off* as a signal that wrapping is +desirted. This would allow us to do without the `WrappingOps` trait +and to avoid having unspecified results. See: + + * [Daniel Micay on June 24][DM24_2] + +Reasons this was not pursued: The official semantics of a type should not change +based on the context. It should be possible to make the choice between turning +checks `on` or `off` solely based on performance considerations. It should be +possible to distinguish cases where checking was too expensive from where +wraparound was desired. (Wraparound is not usually desired.) + ## Different operators Have the usual arithmetic operators check for overflow, and introduce a new set @@ -457,7 +423,6 @@ Reasons this was not pursued: Wrong defaults. Doesn't enable distinguishing Reasons this was not pursued: My brain melted. :( - # Unresolved questions "What should the default be where neither `overflow_checks(on)` nor `off` has @@ -488,16 +453,8 @@ might have to be provided if supporting them is desirable. [transmute]: https://github.com/rust-lang/rfcs/pull/91 - # Future work - * Extend the ability to make use of local `overflow_checks(on|off)` attributes - to user-defined types, as discussed under Drawbacks. For newtypes of the - built-in types which simply want their arithmetic operations to forward to - the built-in ones, this could be accomplished by just adding a newtype - deriving feature, which we would eventually like to do anyways. For more - involved cases, there may not be an easy solution. - * Look into adopting imprecise exceptions and a similar design to Ada's, and to what is explored in the research on AIR (As Infinitely Ranged) semantics, to improve the performance of checked arithmetic. See also: @@ -514,30 +471,59 @@ might have to be provided if supporting them is desirable. [CZ22]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010483.html [JR23_2]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010527.html +## Acknowledgements and further reading -# Appendix A: On the precise meaning of unspecified results. +This RFC was [initially written by Gábor Lehel][GH] and was since +edited by Nicholas Matsakis into its current form. Although the text +has changed significantly, the spirit of the original is preserved (at +least in our opinion). The primary changes from the original are: -*In theory*, the below loop would be allowed to print "hello" any number of -times from 127 to infinity (inclusive), and may print "hello" a different number -of times on different runs of the program, and may print "hedgehogs", but may -not print "penguins". +1. Define the results of errors in some cases rather than using undefined values. +2. Move discussion of scoped attributes to the "future directions" section. +3. Define defaults for when overflow checking is enabled. - fn main() { - let mut i: i8 = 1; +Many aspects of this proposal and many of the ideas within it were +influenced and inspired by +[a discussion on the rust-dev mailing list][GL18]. The author is +grateful to everyone who provided input, and would like to highlight +the following messages in particular as providing motivation for the +proposal. - while i > 0 { - println!("hello"); - i = i + 1; - if i != i { println!("penguins"); } - if i + 1 != i + 1 { println!("hedgehogs"); } - } - } +On the limited use cases for wrapping arithmetic: -Thus, for example, LLVM's `undef` could not be used to represent the value of -`100i8 + 100`, because `undef` can be two different values in different usage -points. + * [Jerry Morrison on June 20][JM20] + +On the value of distinguishing where overflow is valid from where it is not: + + * [Gregory Maxwell on June 18][GM18] + * [Gregory Maxwell on June 24][GM24] + * [Robert O'Callahan on June 24][ROC24] + * [Jerry Morrison on June 24][JM24] + +The idea of scoped attributes: + + * [Daniel Micay on June 23][DM23] + +On the drawbacks of a type-based approach: + + * [Daniel Micay on June 24][DM24] + +In general: + + * [John Regehr on June 23][JR23] + * [Lars Bergstrom on June 24][LB24] + +Further credit is due to the commenters in the [GitHub discussion thread][GH]. + +[GL18]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010363.html +[GM18]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010371.html +[JM20]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010410.html +[DM23]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010566.html +[JR23]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010558.html +[GM24]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010580.html +[ROC24]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010602.html +[DM24]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010598.html +[JM24]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010596.html +[LB24]: https://mail.mozilla.org/pipermail/rust-dev/2014-June/010579.html +[GH]: https://github.com/rust-lang/rfcs/pull/146 -*In practice*, implementations should avoid unnecessarily surprising behavior, -and the above loop should almost always print "hello" 127 times, and nothing -else. Implementations would also be encouraged to diagnose at compile time the -fact that this loop, as well as `100i8 + 100`, will always overflow. From 9681e603239031401a844dced75ce54295d4f905 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 30 Jan 2015 11:07:38 -0500 Subject: [PATCH 12/18] Add a suggestion for a lint. --- text/0000-integer-overflow.md | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/text/0000-integer-overflow.md b/text/0000-integer-overflow.md index b68810fcee7..057ef8f3cbf 100644 --- a/text/0000-integer-overflow.md +++ b/text/0000-integer-overflow.md @@ -217,6 +217,16 @@ newtype for which the operator overloads are implemented using the Note that this is only for potential convenience. The type-based approach has the drawback that e.g. `Vec` and `Vec>` are incompatible types. +## Lint + +In general it seems inadvisable to use operations with error +conditions (like a naked `+` or `-`) in unsafe code. It would be +better to use explicit `checked` or `wrapped` operations as +appropriate. The same holds for destructors, since unwinding in +destructors is inadvisable. Therefore, the RFC recommends a lint be +added against such operations, defaulting to warn, though the details +(such as the name of this lint) are not spelled out. + # Drawbacks **Making choices is hard.** Having to think about whether wraparound @@ -252,10 +262,10 @@ violated. However, this danger already exists, as there are numerous ways to trigger a panic, and hence unsafe code must be written with this in mind. It seems like the best advice is for unsafe code to eschew the plain `+` and `-` operators, and instead prefer explicit -checked or wrapping operations as appropriate (this would be a good -opportunity for a lint). Furthermore, the danger of an unexpected -panic occurring in unsafe code must be weighed against the danger of a -(silent) overflow, which can also lead to unsafety. +checked or wrapping operations as appropriate (hence the proposed +lint). Furthermore, the danger of an unexpected panic occurring in +unsafe code must be weighed against the danger of a (silent) overflow, +which can also lead to unsafety. **Divergence of debug and optimized code.** The proposal here causes additional divergence of debug and optimized code, since optimized From 865eae64f376ae4804a52467ebbce31707f3a784 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 30 Jan 2015 11:11:55 -0500 Subject: [PATCH 13/18] Rephrase the *optional* mode so it is deterministic; but make it clear that not having checks enabled is not the same as having them explicitly *disabled*. --- text/0000-integer-overflow.md | 71 +++++++++++++++++------------------ 1 file changed, 34 insertions(+), 37 deletions(-) diff --git a/text/0000-integer-overflow.md b/text/0000-integer-overflow.md index 057ef8f3cbf..71807d9ecd6 100644 --- a/text/0000-integer-overflow.md +++ b/text/0000-integer-overflow.md @@ -48,12 +48,12 @@ The RFC has several major goals: To that end the RFC proposes two mechanisms: 1. Optional, dynamic overflow checking. Ordinary arithmetic operations - (e.g., `a+b`) would *optionally* check for overflow. If checking is - enabled, and an overflow occurs, a thread panic will be + (e.g., `a+b`) would conditionally check for overflow. If an + overflow occurs when checking is enabled, a thread panic will be signaled. Specific intrinsics and library support are provided to permit either explicit overflow checks or explicit wrapping. 2. Overflow checking would be, by default, tied to debug assertions - (`debug_assert!`). It can be seen as analogous to a debug + (`debug_assert!`). It can be seen as analogous to a debug assertion: an important safety check that is too expensive to perform on all code. @@ -99,42 +99,40 @@ a static/dynamic analysis tools as an error. This is largerly a semantic distinction, though. The result of an error condition depends upon the state of overflow -checking, which can be either *enabled* or *optional* (this RFC does +checking, which can be either *enabled* or *default* (this RFC does not describe a way to disable overflow checking completely). If overflow checking is *enabled*, then an error condition always results in a panic. For efficiency reasons, this panic may be delayed over some number of pure operations, as described below. -If overflow checking is *optional*, then an error condition may result -in a panic, but the erroneous operation may also simply produce a -value. The permitted results for each error condition are specified -below. When checking is optional, panics are not required to be -consistent across executions. This means that, for example, the -compiler could insert code which results in an overflow only if the -value that was produced is actually consumed (which may occur on only -one arm of a branch). - -In the future, we may add some way to explicit *disable* overflow -checking (probably in a scoped fashion). In that case, the result of -each error condition would simply be the same as the optional state -when no panic occurs. +If overflow checking is *default*, that means that erroneous +operations will produce a value as specified below. Note though that +code which encounters an error condition is still considered buggy. +In particular, Rust source code (in particular library code) cannot +rely on wrapping semantics, and should always be written with the +assumption that overflow checking *may* be enabled. This is because +overflow checking may be enabled by a downstream consumer of the +library. + +In the future, we could add some way to explicitly *disable* overflow +checking in a scoped fashion. In that case, the result of each error +condition would simply be the same as the optional state when no panic +occurs, and this would requests for override checking specified +elsewhere. However, no mechanism for disabling overflow checks is +provided by this RFC: instead, it is recommended that authors use the +wrapped primitives. The error conditions that can arise, and their defined results, are as -follows. In all cases, the defined results are the same as the defined -results today. The only change is that now a panic may result. - -- The operations `+`, `-`, `*`, `/`, and `%` can underflow and overflow. If no panic occurs, - the result of such an operation is defined to be the same as wrapping. -- When truncating, the casting operation `as` can overflow if the truncated bits contain - non-zero values. If no panic occurs, the result of such an operation is defined to - be the same as wrapping. -- Shift operations (`<<`, `>>`) can shift a value of width `N` by more - than `N` bits. If no panic occurs, the result of such an operation - is an *undefined value* (note that this is not the same as - *undefined behavior* in C). - - The reason that there is no defined result here is because - different CPUs have different behavior, and we wish to be able to translate shifting - operations directly to the corresponding machine instructions. +follows. The intention is that the defined results are the same as the +defined results today. The only change is that now a panic may result. + +- The operations `+`, `-`, `*`, `/`, `%` can underflow and + overflow. Shift operations (`<<`, `>>`) can shift a value of width `N` by more + than `N` bits. In these cases, the result is the same as the pre-existing, + wrapping semantics. +- When truncating, the casting operation `as` can overflow if the + truncated bits contain non-zero values. If no panic occurs, the + result of such an operation is defined to be the same as wrapping. ## Enabling overflow checking @@ -294,11 +292,10 @@ direct overhead of checking for overflow, there is some additional overhead when checks are enabled because compilers may have to forego other optimizations or code motion that might have been legal. This concern seems minimal since, in optimized builds, overflow checking -will be considered *optional* and hence the compiler is free to ignore -it. Certainly if we ever decided to change the default for overflow -checking from *optional* to *enabled* in optimized builds, we would -want to measure carefully and likely include some means of disabling -checks in particularly hot paths. +will not be enabled. Certainly if we ever decided to change the +default for overflow checking to *enabled* in optimized builds, we +would want to measure carefully and likely include some means of +disabling checks in particularly hot paths. # Alternatives and possible future directions From af776c65f0cbf3c35fba5a0b99210955ba65b72f Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 30 Jan 2015 11:13:01 -0500 Subject: [PATCH 14/18] Add methods for wrapping versions of the `as xx` conversions. --- text/0000-integer-overflow.md | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/text/0000-integer-overflow.md b/text/0000-integer-overflow.md index 71807d9ecd6..0bd1282db36 100644 --- a/text/0000-integer-overflow.md +++ b/text/0000-integer-overflow.md @@ -180,6 +180,18 @@ pub trait WrappingOps { fn wrapping_mul(self, rhs: Self) -> Self; fn wrapping_div(self, rhs: Self) -> Self; fn wrapping_rem(self, rhs: Self) -> Self; + + fn wrapping_as_u8(self, rhs: Self) -> u8; + fn wrapping_as_u16(self, rhs: Self) -> u16; + fn wrapping_as_u32(self, rhs: Self) -> u32 + fn wrapping_as_u64(self, rhs: Self) -> u64; + fn wrapping_as_usize(self, rhs: Self) -> usize; + + fn wrapping_as_i8(self, rhs: Self) -> i8; + fn wrapping_as_i16(self, rhs: Self) -> i16; + fn wrapping_as_i32(self, rhs: Self) -> i32 + fn wrapping_as_i64(self, rhs: Self) -> i64; + fn wrapping_as_isize(self, rhs: Self) -> isize; } impl WrappingOps for isize @@ -451,15 +463,6 @@ cost. If we do this, it would obviate the need for the `wrapping_div` and `wrapping_rem` methods, and they could be removed. This isn't intrinsically tied to the current RFC, and could be discussed separately. -It is not clear whether, or how, overflowing casts between types should be -provided. For casts between signed and unsigned integers of the same size, -the [`Transmute` trait for safe transmutes][transmute] would be appropriate in -the future, or the unsafe `transmute()` function in the present. For other casts -(between different sized-types, between floats and integers), special operations -might have to be provided if supporting them is desirable. - -[transmute]: https://github.com/rust-lang/rfcs/pull/91 - # Future work * Look into adopting imprecise exceptions and a similar design to Ada's, and to From 22ecff8301393655746f6c48b1ac1abff3ddcfb9 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 30 Jan 2015 11:13:16 -0500 Subject: [PATCH 15/18] Refine the text on delayed panics somewhat. --- text/0000-integer-overflow.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/text/0000-integer-overflow.md b/text/0000-integer-overflow.md index 0bd1282db36..16d7766dfd6 100644 --- a/text/0000-integer-overflow.md +++ b/text/0000-integer-overflow.md @@ -160,11 +160,14 @@ compile-time switch. If an error condition should occur and a thread panic should result, the compiler is not required to signal the panic at the precise point of overflow. It is free to coalesce checks from adjacent pure -operations. Panics may never be delayed across an unsafe block nor -delayed by an indefinite number of steps, however. The precise -definition of a pure operation can be hammered out over time, but the -intention here is that, at minimum, overflow checks for adjacent -numeric operations like `a+b-c` can be coallesced into a single check. +operations. Panics may never be delayed across an unsafe block nor may +they be skipped entirely, however. The precise details of how panics +may be deferred -- and the definition of a pure operation -- can be +hammered out over time, but the intention here is that, at minimum, +overflow checks for adjacent numeric operations like `a+b-c` can be +coallesced into a single check. Another useful example might be that, +when summing a vector, the final overflow check could be deferred +until the summation is complete. ## `WrappingOps` trait for explicit wrapping arithmetic From 341ff45bcc94c7096c7ecf9640e3d6d36b281eca Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 30 Jan 2015 11:21:42 -0500 Subject: [PATCH 16/18] Small suggestions from @mkaufmann. --- text/0000-integer-overflow.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/text/0000-integer-overflow.md b/text/0000-integer-overflow.md index 16d7766dfd6..541dddc2c59 100644 --- a/text/0000-integer-overflow.md +++ b/text/0000-integer-overflow.md @@ -83,7 +83,7 @@ semantics, even if accidentally. By making the default for debugging code be checked overflow, we help ensure that users will encounter overflow errors in practice, and thus become aware that overflow in Rust is not the norm. It will also help -debug simple errors, like signed underflow leading to an infinite +debug simple errors, like unsigned underflow leading to an infinite loop. # Detailed design @@ -209,7 +209,8 @@ impl WrappingOps for i64 impl WrappingOps for u64 ``` -These are implemented to wrap around on overflow unconditionally. +These are implemented to preserve the pre-existing, wrapping semantics +unconditionally. ### `Wrapping` type for convenience @@ -244,8 +245,8 @@ added against such operations, defaulting to warn, though the details **Making choices is hard.** Having to think about whether wraparound arithmetic is appropriate may cause an increased cognitive -burden. However, wraparound arithmetic is almost never -appropriate. Therefore, programmers should be able to keep using the +burden. However, wraparound arithmetic is almost never the intended +behavior. Therefore, programmers should be able to keep using the built-in integer types and to not think about this. Where wraparound semantics are required, it is generally a specialized use case with the implementor well aware of the requirement. From 648e1b1531773d01a526b26c73089ec23530c05f Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 30 Jan 2015 11:21:59 -0500 Subject: [PATCH 17/18] Rewrite the unresolved questions to clearly highlight that the semantics of division and shift are out of scope for this RFC. --- text/0000-integer-overflow.md | 29 +++++++++++------------------ 1 file changed, 11 insertions(+), 18 deletions(-) diff --git a/text/0000-integer-overflow.md b/text/0000-integer-overflow.md index 541dddc2c59..ce9463df1fe 100644 --- a/text/0000-integer-overflow.md +++ b/text/0000-integer-overflow.md @@ -448,24 +448,17 @@ Reasons this was not pursued: My brain melted. :( # Unresolved questions -"What should the default be where neither `overflow_checks(on)` nor `off` has -been explicitly specified?", as discussed in the main text. - -Instead of a single `WrappingOps` trait, should there perhaps be a separate -trait for each operation, as with the built-in arithmetic traits `Add`, `Sub`, -and so on? The only purpose of `WrappingOps` is to be implemented for the -built-in numeric types, so it's not clear that there would be a benefit to doing -so. - -C and C++ define `INT_MIN / -1` and `INT_MIN % -1` to be undefined behavior, to -make it possible to use a division and remainder instruction to compute them. -Mathematically, however, modulus can never overflow and `INT_MIN % -1` has value -`0`. Should Rust consider these to be instances of overflow, or should it -guarantee that they return the correct mathematical result, at the cost of a -runtime branch? Division is already slow, so a branch here may be an affordable -cost. If we do this, it would obviate the need for the `wrapping_div` and -`wrapping_rem` methods, and they could be removed. This isn't intrinsically tied -to the current RFC, and could be discussed separately. +The C semantics of wrapping operations in some cases are undefined: + +- `INT_MIN / -1`, `INT_MIN % -1` +- Shifts by an excessive number of bits + +This RFC takes no position on the correct semantics of these +operations, simply preserving the existing semantics. However, it may +be worth trying to define the wrapping semantics of these operations +in a portable way, even if that implies some runtime cost. Since these +are all error conditions, this is an orthogonal topic to the matter of +overflow. # Future work From 9fb9ad698cf83e77b45cfbe9845962f5a8ca6d35 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 30 Jan 2015 11:22:15 -0500 Subject: [PATCH 18/18] Add methods for lshift and rshift to the wrapping trait. --- text/0000-integer-overflow.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/text/0000-integer-overflow.md b/text/0000-integer-overflow.md index ce9463df1fe..81746ed99cc 100644 --- a/text/0000-integer-overflow.md +++ b/text/0000-integer-overflow.md @@ -183,7 +183,10 @@ pub trait WrappingOps { fn wrapping_mul(self, rhs: Self) -> Self; fn wrapping_div(self, rhs: Self) -> Self; fn wrapping_rem(self, rhs: Self) -> Self; - + + fn wrapping_lshift(self, amount: u32) -> Self; + fn wrapping_rshift(self, amount: u32) -> Self; + fn wrapping_as_u8(self, rhs: Self) -> u8; fn wrapping_as_u16(self, rhs: Self) -> u16; fn wrapping_as_u32(self, rhs: Self) -> u32