Repository navigation
Batch size greater than 1 results in memory allocation error #742
|
Tried first with the TextRowSet example in the docs, and again with a RowVec, and both fail with a memory allocation error if the batch size is greater than 1. The data type being detected is correct per the source documentation and both work with batch set to one. Is this a data source problem or could there be an issue with the library? Error: Relevant excerpt of code used for testing: Some(mut cursor) => {
// Write the column names to stdout
let headline : Vec<String> = cursor.column_names()?.collect::<Result<_,_>>()?;
for (index, name) in headline.iter().enumerate() {
println!("Column: {}, Data Type: {:?}", name, cursor.col_data_type(index as u16+1)?);
}
writer.write_record(headline)?;
let max_rows_in_batch = 1;
type Row = (Bit,);
let buffer = RowVec::<Row>::new(max_rows_in_batch);
let mut block_cursor = cursor.bind_buffer(buffer)?;
while let Some(batch) = block_cursor.fetch()? {
for (value,) in batch.iter() {
println!("row: {value:?}");
}
}
// // Use schema in cursor to initialize a text buffer large enough to hold the largest
// // possible strings for each column up to an upper limit of 4KiB.
// let mut buffers = TextRowSet::for_cursor(BATCH_SIZE, &mut cursor, None)?;
// // Bind the buffer to the cursor. It is now being filled with every call to fetch.
// let mut row_set_cursor = cursor.bind_buffer(&mut buffers)?;
//
// // Iterate over batches
// while let Some(batch) = row_set_cursor.fetch()? {
// // Within a batch, iterate over every row
// for row_index in 0..batch.num_rows() {
// // Within a row iterate over every column
// let record = (0..batch.num_cols()).map(|col_index| {
// batch
// .at(col_index, row_index)
// .unwrap_or(&[])
// });
// // Writes row as csv
// writer.write_record(record)?;
// }
// }
} |
Replies: 9 comments 4 replies
|
Thanks for raising the issue, May I ask, what platform are you on? What driver manager are you using and what is the database and its driver you are using to connect? Normally if I see allocation errors my mind jumps to columns with really large upper bounds for individual elements. This does not seem to be the case here. Honestly the first time I see an allocation error within the driver manager. Overall more than one row has been fetched at the time in plenty of use-cases using this library. This does not imply by far that there is no bug. Yet at least it means I need more information to reproduce the error. Answering the questions above might help. Best, Markus |
|
It does not seem to be a problem with the size of the returned data, which was my first thought as well. To check that I tried a query that only returned about 15 rows of a single Bit value and observed the same behavior. I'm not sure how the driver works so possibly the data returned could be different from the data it needs to allocate memory for. This is running on a Windows 10 system, using QODBC to query data from Quickbooks Desktop. I have run the same queries on that system with a Go application without issues, however if Go it has a similar configuration option for batch size it is whatever the default is. |
|
What batch size did you configure? Did you try |
|
The only value I know works is Code for batch_size in [1, 2, 3, 4, 5, 10, 15, 100, 1000] {
println!("------- Batch size: {batch_size} ---------");
match connection.execute("SELECT IsManuallyClosed FROM SalesOrderLine WHERE SalesOrderLineQuantity IS NOT NULL AND SalesOrderLineItemRefFullName IS NOT NULL AND IsFullyInvoiced = false AND IsManuallyClosed = false", (), None)? {
Some(mut cursor) => {
let headline : Vec<String> = cursor.column_names()?.collect::<Result<_,_>>()?;
for (index, name) in headline.iter().enumerate() {
println!("Column: {}, Data Type: {:?}", name, cursor.col_data_type(index as u16+1)?);
}
type Row = (Bit,);
let buffer = RowVec::<Row>::new(batch_size);
let mut block_cursor = cursor.bind_buffer(buffer)?;
match block_cursor.fetch() {
Ok(Some(batch)) => {
for (value, ) in batch.iter() {
println!("row: {value:?}");
}
},
Ok(None) => println!("no rows available"),
Err(e) => eprintln!("{e}")
}
}
None => {
eprintln!(
"Query came back empty. No output has been created."
);
}
}
} |
|
Another data point is this api causes the latest version of QODBC to crash, while the Go library still works. The only information I could get from it is: If you have any ideas on where I should go poking around first let me know. I don't know currently know much about the ODBC driver. |
|
I have tests with more than one row in a batch ;-). If you are on Windows, that also ships with a builtin ODBC driver manager. So the most likely culprit seems to me the driver QODBC itself. I would assume that your Go code is currently transfers the data row by row, which could be equivalent to a batch size of Best, Markus |
|
Enabled logging to check the return from the If the new size is less than or equal to the buffer capacity it seems this could be handled automatically, although maybe it would be best to just return an error from |
|
Hello @5061726b6572 , thanks for activating logging. This uncovers part of the mystery. It seems QODBC indeed does not support bulk fetching. It even answers with the correct status code in the diagnostic record. Now as I read the standard it should set the value to something similar which is still supported (i.e. leave it at In that case what is supposed to happen is that while the application may bind a larger buffer, only the first row of it is ever filled. Not as fast as a bulk fetch, but it would still work. I wonder if it really sets the value to one after, or if something else happens. We do not have to guess though. We can ask the statement handle about the // Given a connection `conn`
let query = "SELECT IsManuallyClosed FROM SalesOrderLine WHERE SalesOrderLineQuantity IS NOT NULL AND SalesOrderLineItemRefFullName IS NOT NULL AND IsFullyInvoiced = false AND IsManuallyClosed = false"
let mut statement = conn.prepare(query).unwrap();
let mut stmt = statement.as_stmt_ref();
let row_array_size = unsafe {
stmt.set_row_array_size(2).unwrap();
stmt.row_array_size().unwrap()
};
println("ROW_ARRAY_SIZE: {row_array_size}"); |
|
|
No, that is actually about the version of the ODBC API used. Your driver does not support
3.8.odbc-apialso allows you to set the earlier3.5version. You need to deactivate default features and setodbc_version_3_5in order to do so.That's it right there. That is a rather weird number. Does not seem like an integer overflow or anything to me. However, it is large! Maybe large enough to cause a memory allocation error, if somebody would allocate an array of that size? I think so.
If we request a change of
ROW_ARRAY_SIZEto2. And the driver does no…