Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Gather] Clamp the indices to prevent out-of-bound reading #82

Open
shiyi9801 opened this issue Jan 9, 2025 · 1 comment
Open

[Gather] Clamp the indices to prevent out-of-bound reading #82

shiyi9801 opened this issue Jan 9, 2025 · 1 comment
Labels

Comments

@shiyi9801
Copy link
Owner

shiyi9801 commented Jan 9, 2025

If the index is out-of-bound, CPU EP will throw runtime error like: Non-zero status code returned while running Gather node. Name:'gather_0' Status Message: indices element out of data bounds, idx=3 must be within the inclusive range [-3,2].

OpenVINO GPU EP won't throw error but get zeros for these indices.

Edited:
Strangely, I tested this case again and the result changed. Not sure if it's caused by the environment change or I made a mistake🤔

OV GPU would fail with runtime error

[28140:15376:0218/141513.353:ERROR:graph_impl_ort.cc(308)] : [WebNN] Failed to call GetOrtModelBuilderApi()->CreateSessionFromModel( env, model_info->model, session_options, session.GetAddressOf()): Exception during initialization: D:\onnxruntime\onnxruntime\core\providers\openvino\backend_manager.cc:163 __cdecl onnxruntime::openvino_ep::BackendManager::BackendManager(const struct onnxruntime::openvino_ep::GlobalContext &,const struct onnxruntime::Node &,const class onnxruntime::GraphViewer &,const struct onnxruntime::logging::Logger &,class onnxruntime::openvino_ep::EPCtxHandler &) D:\onnxruntime\onnxruntime\core\providers\openvino\ov_interface.cc:106 class onnxruntime::openvino_ep::OVExeNetwork __cdecl onnxruntime::openvino_ep::OVCore::CompileModel(const class std::basic_string<char,struct std::char_traits,class std::allocator > &,class std::basic_string<char,struct std::char_traits,class std::allocator > &,class std::map<class std::basic_string<char,struct std::char_traits,class std::allocator >,class ov::Any,struct std::less<class std::basic_string<char,struct std::char_traits,class std::allocator > >,class std::allocator<struct std::pair<class std::basic_string<char,struct std::char_traits,class std::allocator > const ,class ov::Any> > > &,const class std::basic_string<char,struct std::char_traits,class std::allocator > &) [OpenVINO-EP] Exception while Loading Network for graph: OpenVINOExecutionProvider_OpenVINO-EP-subgraph_21_0Exception from src/inference/src/cpp/core.cpp:142:
Exception from src/inference/src/dev/plugin.cpp:53:
src/plugins/intel_gpu/src/graph/crop.cpp at line: 253
Error has occured for: gather:Gather_79724_reshape_output/crop_10
input sizes - offsets sizes: [b:-8, f:12, x:1, y:1, z:0, w:0, u:0, v:0, g:0]
reference input sizes sizes: [b:1, f:12, x:1, y:1, z:1, w:1, u:1, v:1, g:1]
All input sizes - offsets dimensions should not be less than reference input sizes dimensions.
Mismatching dimensions: Batch
Invalid Batch offset: exceeds data for output!

And OV CPU would fail due to wrong results

Fail | gather float32 2D tensor and int32 0D out-of-bound positive indices default options | assert_true: assert_array_approx_equals_ulp: test gather float32 actual 0 should be close enough to expected 90.2870101928711 by the acceptable 0 ULP distance, but they have 1119130355 ULP distance expected true got false
Fail | gather float32 2D tensor and int32 0D out-of-bound negative indices default options | assert_true: assert_array_approx_equals_ulp: test gather float32 actual 0 should be close enough to expected -66.05901336669922 by the acceptable 0 ULP distance, but they have 1031528905 ULP distance expected true got false

OV version: 2024.6.0.17404, Debug version

@shiyi9801
Copy link
Owner Author

ORT CPU EP also fails with error

[5128:21800:0220/152128.313:FATAL:graph_impl_ort.cc(122)] : NOTREACHED hit. [WebNN] Ort Status: Non-zero status code returned while running Gather node. Name:'_0' Status Message: indices element out of data bounds, idx=-10 must be within the inclusive range [-2,1]
base::debug::CollectStackTrace [0x00007FFAAE72A055+21]
base::debug::StackTrace::StackTrace [0x00007FFAAE73B2FC+236]
logging::LogMessage::Flush [0x00007FFAAE7DF542+210]
logging::LogMessage::~LogMessage [0x00007FFAAE7DF409+25]
logging::NotReachedNoreturnError::NotReachedNoreturnError [0x00007FFAAE7FF3EE+1518]
logging::CheckNoreturnError::~CheckNoreturnError [0x00007FFAAE7FEC5B+11]
webnn::ort::GraphImplOrt::ComputeResources::OrtRunSync [0x00007FFAB6895B36+2198]
base::internal::Invoker<base::internal::FunctorTraits<void (__cdecl webnn::ort::GraphImplOrt::ComputeResources::&& __ptr64)(std::__Cr::vector<std::__Cr::pair<std::__Cr::basic_string<char,std::__Cr::char_traits,std::__Cr::allocator >,OrtValue [0x00007FFAB6895BB7+119]
base::OnceCallback<void __cdecl(void)>::Run [0x00007FFAA6E7CF3D+141]
base::internal::PostTaskAndReplyRelay::RunTaskAndPostReply [0x00007FFAAE75D9B7+103]
base::internal::Invoker<base::internal::FunctorTraits<void (__cdecl
&& __ptr64)(base::internal::PostTaskAndReplyRelay),base::internal::PostTaskAndReplyRelay && __ptr64>,base::internal::BindState<0,1,0,void (__cdecl*)(base::internal::PostTaskAndReplyRelay) [0x00007FFAAE75DB46+54]
base::OnceCallback<void __cdecl(void)>::Run [0x00007FFAA6E7CF3D+141]
base::TaskAnnotator::RunTaskImpl [0x00007FFAAE7737A4+340]
base::internal::TaskTracker::RunSkipOnShutdown [0x00007FFAB2922C78+200]
base::internal::TaskTracker::RunTask [0x00007FFAB2922057+1159]
base::internal::TaskTracker::RunAndPopNextTask [0x00007FFAB2921844+772]
base::internal::WorkerThread::RunWorker [0x00007FFAB50B5301+1073]
base::internal::WorkerThread::RunPooledWorker [0x00007FFAB50B4D58+24]
base::PlatformThreadBase::GetCurrentThreadPriorityForTest [0x00007FFAAE705634+660]
BaseThreadInitThunk [0x00007FFB62CE259D+29]
RtlUserThreadStart [0x00007FFB6494AF38+40]

@huningxin huningxin added the ort label Feb 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants