Crashing with big prompt when streaming.

[JG-Adams]

I need to be able to send some large prompt, but it appear to crash.
It happen consistently for large prompt but not for small one.
I tried to make it not happen with try and catch exception, but it didn't work either.

However, it does not crash if I use no stream.

So, I attempted to find the issue and tried turning off allow_exception and show replies because it only threw Ollama exception without any more info. I discovered that it end like this:

{"model":"llama3.1:latest","created_at":"2024-08-12T04:18:00.108913513Z","response":"","done":true,"done_reason":"stop","context":[128006,9125,128007,271,1687,2351,304,264,15861,1405,1274,527,41416,71017,28296,10269,2209,437,346,645,39131,449,11977,389,872,11314,13,11205,6548,527,4251,12960,449,6437,61072,26432,11,323,264,3254,4251,28029,11013,627,32,8762,39131,8965,617,264,2385,1933,315,1193,279,35044,7106,1933,1093,11,3816,11,22725,11,26541,11,7997,11,8828,6437,11,8868,13,2030,11,814,1101,617,10824,1093,5929,11,14198,11,14531,627,32,8954,39131,4869,11,617,279,10824,315,2579,323,6437,8146,11,7231,1124,3816,11,41489,11,27211,11,7023,16985,11,323,8868,13,2435,617,18460,11813,369,682,220,18,3585,11,20444,19960,527,17676,14618,627,2675,527,279,10877,13,1472,23846,279,3446,358,3371,499,1093,433,596,264,5818,13,128009,128006,9125,128007,271,1687,2351,304,264,15861,1405,1274,527,41416,71017,28296,10269,2209,437,346,645,39131,449,11977,389,872,11314,13,11205,6548,527,4251,12960,449,6437,61072,26432,11,323,264,3254,4251,28029,11013,627,32,8762,39131,8965,617,264,2385,1933,315,1193,279,35044,7106,1933,1093,11,3816,11,22725,11,26541,11,7997,11,8828,6437,11,8868,13,2030,11,814,1101,617,10824,1093,5929,11,14198,11,14531,627,32,8954,39131,4869,11,617,279,10824,315,2579,323,6437,8146,11,7231,1124,3816,11,41489,11,27211,11,7023,16985,11,323,8868,13,2435,617,18460,11813,369,682,220,18,3585,11,20444,19960,527,17676,14618,627,2675,527,279,10877,13,1472,23846,279,3446,358,3371,499,1093,433,596,264,5818,13,128009,128006,882,128007,1038,10224,25,622,638,70408,271,2170,279,7160,6137,311,743,389,279,13057,16763,1974,11,264,47766,26541,39131,7086,622,638,1903,813,1648,1203,311,813,13691,11,264,3254,1515,1776,2234,927,813,17308,627,16366,6548,15541,839,449,264,5647,315,24617,11,719,24923,430,11203,264,13310,315,87163,5849,323,264,19662,41328,922,279,1917,7953,279,35174,627,1548,71771,264,14351,11,323,1243,4780,311,9762,8356,505,813,13863,627,2170,568,15203,11,813,36496,41398,704,311,279,61838,11,653,50009,506,95519,315,279,16763,8329,11,264,5647,315,79422,323,31398,28786,927,1461,627,2409,813,14351,2539,11,622,638,6052,311,813,13691,11,5675,304,3463,439,568,11427,3093,279,64550,315,264,50999,628,40458,86188,627,1548,5900,311,1373,1063,315,813,7075,2363,11,568,1047,4101,315,2999,18025,345,1,11787,12639,17694,2744,1314,36818,72059,264,1314,22217,3041,499,264,58768,34536,1,52522,279,1314,39131,34536,1,96621,264,69934,1994,39131,994,499,1518,832,34536,1,5530,6296,13,1314,34536,1,96621,701,14469,1182,9135,627,1548,2503,304,813,10716,627,438,3940,5403,627,791,1828,1938,9522,41,638,4321,704,315,813,3838,3411,264,2697,12703,13,320,18433,568,6439,7636,29275,1548,2010,10411,311,813,1866,6246,1828,311,13952,627,2181,706,264,4686,12960,315,65442,2212,264,36670,304,279,6278,449,5496,3245,1093,37085,24269,304,433,627,1548,2503,389,813,23162,10716,627,23274,12703,627,1548,16008,264,3070,315,37085,627,1548,53914,627,98331,568,21423,68,3059,627,32,938,11975,80608,311,430,28485,7162,627,31082,11,622,638,374,86015,813,1515,449,813,1450,627,3112,568,53006,433,449,813,40902,1139,4027,6798,627,1548,5900,4871,323,923,1515,311,813,40511,627,1548,2800,304,813,10716,369,279,3814,14624,311,4027,17944,2785,627,2181,2543,6453,627,791,1828,6693,11,568,1427,704,279,3321,627,16366,3663,374,1437,41133,449,65875,627,32,1317,892,4227,994,568,574,264,1716,13,5414,18233,3309,1461,11,330,40,2846,1022,311,990,11,358,3358,387,1203,18396,10246,41,638,20592,11,330,6854,956,358,2586,449,499,48469,7189,6562,499,649,1210,568,6013,11,330,4071,11,358,2011,5944,34504,2533,433,596,264,1317,11879,13,30070,499,649,11722,11,499,2011,4822,1618,10246,7009,65682,47555,323,568,3952,11213,14836,11226,627,1548,2646,6052,627,41,638,14980,1070,555,5678,627,3112,706,3596,2533,11,439,568,25735,64033,9522,1548,4321,927,311,813,2363,55050,3131,810,627,47,92122,704,264,3361,2363,627,10227,15457,315,264,48085,39131,702,1548,6227,813,18233,11890,1461,11,330,40,1390,499,311,6865,264,3446,922,4423,1633,3361,627,2181,574,264,28812,38490,889,12439,1690,1667,4227,13,9176,1274,3463,568,574,1120,264,2184
terminate called after throwing an instance of 'nlohmann::json_abi_v3_11_3::detail::type_error'
what(): [json.exception.type_error.305] cannot use operator[] with a string argument with null

It fail to complete the context and ended up missing some integral info that an operator have tried and expected to access and thus crash.
What we would expect to have after the context is the end bracket and: ],"total_duration":1157301429,"load_duration":29868219,"prompt_eval_count":11,"prompt_eval_duration":27732000,"eval_count":51,"eval_duration":1049088000}

So, it must be guaranteed that the information come through entirely or otherwise we must check whether an object exist before trying to access it.

Or, it could be how it gather the information?
Anyway, this need to be fixed and I'm not sure how I can provide you something to replicate the error except make a large prompt.

[jmont-dev]

Thanks for the info. I think I understand the problem and will put in handling for partial responses from the server. I'm assuming you were using the generate endpoint? Does it only crash when streaming is true and the response is large?

[JG-Adams]

Yes. Generate, and only crash when streaming is true and prompt is large. (Not sure about the response.)
I haven't tried chat on this yet. But, maybe it has it as well.

I'm curious what context is used for. Can it actually be used into another prompt?

[jmont-dev]

Context is the list of the tokens from the past conversation history, eg the prompt you provided and the model response in vector form. If you feed this as an input to another prompt the model will have the entire conversation history when producing a response. Most models only support a context window of a fixed size (usually 2048 - 8096 tokens) but this can be used for longer conversations or generations with longer inputs.

The Ollama team has added a few things to the generate endpoint. I'll add those and also an input for context so that it can be fed back into the function.

[JG-Adams]

I tried something really silly.
auto stream_callback = [on_receive_token](const char *data, size_t data_length)->bool{

        std::string message(data, data_length);
        if (ollama::log_replies) std::cout << message << std::endl;
        if (message.substr(message.size()-1, 1) != "\n") {std::cout << "\nDetected!\n"; message += "0],\"total_duration\":1157301429,\"load_duration\":29868219,\"prompt_eval_count\":11,\"prompt_eval_duration\":27732000,\"eval_count\":51,\"eval_duration\":1049088000}";
        std::cout << message << "\n";}
        ollama::response response(message);
        on_receive_token(response);

        return true;
    };
    
    By appending the missing parameter it enables it to continue for one more message which turns out to be the remainder.

Message 1.
{"model":"llama3:latest","created_at":"2024-08-12T05:22:40.323485736Z","response":"","done":true,"done_reason":"stop","context":[lots of numbers... ,16366,18427,706,0],"total_duration":1157301429,"load_duration":29868219,"prompt_eval_count":11,"prompt_eval_duration":27732000,"eval_count":51,"eval_duration":1049088000}
message2
1120,22088,11,449,279,13057,16763,8329,42949,704,1603,1461,1093,459,26762,10247,8748,311,387,36131,382,3648,17952,1473,2170,622,638,82294,1555,279,13057,1825,16763,1974,11,568,6137,311,2733,5675,323,40186,505,813,40190,627,1548,3940,13633,389,279,5015,11,4560,311,1893,264,10723,2472,315,1405,568,574,323,1268,311,636,1203,389,3839,627,32,6926,293,10616,10616,10616,12248,10791,813,6666,11,7982,62896,449,1855,12579,4545,627,41,638,596,41328,2751,279,2731,315,1461,11,720,438,568,6656,311,1518,264,23464,50644,6437,4047,12585,449,264,3776,25879,389,1202,3185,31047,627,41,638,1047,2646,3970,4205,1093,420,1603,11,720,438,813,22047,6656,311,17035,439,279,12585,53634,264,13587,12970,2624,449,1202,25879,323,14219,433,520,1461,627,791,2624,11,3967,439,264,37943,4047,11,44913,304,264,5041,315,35678,5304,5536,11,14718,622,638,311,30305,704,323,48973,433,627,791,25176,596,40696,57780,291,279,12585,596,11376,11,720,7331,287,622,638,311,10477,4920,264,7091,4315,3892,3885,13,578,12585,31527,291,2212,11,15389,369,1202,2218,11,323,622,638,11,304,264,1614,315,8850,11,22982,264,7091,311,64917,433,627,791,12585,3952,279,49160,11,323,622,638,38949,3201,11,719,813,18338,48946,994,568,2463,7069,627,2170,568,7111,1203,11,568,5602,279,12585,449,1202,25879,55060,520,1461,627,1548,32189,3640,279,3440,323,30418,1202,25879,11,379,33434,433,311,279,5015,26,1243,51249,433,311,279,50393,1405,433,312,33536,5196,13,49407,311,21818,704,315,11682,596,1648,11,622,638,10837,1578,11,1193,311,617,279,12585,4027,3892,37943,46618,520,1461,627,5001,80213,369,3504,11,568,2751,4920,264,7091,13,16910,433,387,904,11201,30,362,49499,4047,574,14219,520,1461,11,568,20858,5678,505,16706,35098,627,1548,5602,430,433,3782,505,279,6138,11,2500,11678,315,279,29807,25,264,2678,2579,13192,449,264,52524,2385,323,49499,11,23464,50644,1120,3485,279,5015,439,433,8738,311,4027,98014,520,1461,627,41,638,7882,311,2500,7091,430,2682,1461,505,279,1403,64490,627,644,264,16763,1974,449,23902,11,323,568,1047,912,4623,1405,311,733,627,1548,16696,311,2339,813,11745,323,1781,627,4071,11,568,596,539,6220,369,1317,439,568,6865,279,6690,18748,86424,14373,627,12487,11,568,30652,264,6307,22217,389,279,24898,627,1548,2543,505,46996,11,330,32,22217,1093,757,19318,568,44543,449,16267,627,27131,65437,11,622,638,38949,291,7119,279,6307,22217,11,11125,14850,505,279,28013,14932,1461,627,1548,2663,704,369,1520,11,323,304,2077,11,279,6307,22217,11684,449,64159,11,79529,264,37943,4047,304,813,1450,627,2645,19787,8850,1139,622,638,596,4851,439,568,15393,568,574,31691,1990,2380,15086,15676,304,389,1461,627,2170,622,638,272,4234,304,8850,11,279,6690,18748,14219,832,520,622,638,11,719,279,6307,22217,15187,514,10395,4741,323,4224,35344,433,505,5209,38635,627,1548,22982,813,1176,37943,4047,520,279,12585,11,902,7882,311,279,3185,13,4452,11,568,19698,420,323,38067,433,1139,9863,449,813,2132,4999,41,638,574,2163,304,51517,439,279,6307,22217,30418,279,12585,596,25879,323,11684,6089,520,279,13192,627,791,13192,14219,3892,49499,46618,520,1461,11,889,514,10395,2212,11,86912,93774,3252,3156,568,2751,3345,3403,311,16363,1022,279,13192,596,49499,627,12487,11,568,51280,433,449,279,25879,311,6381,433,1022,0,1666,568,27096,704,323,75095,4285,291,11,622,638,1766,5678,304,4686,51517,315,279,6307,22217,596,7512,627,791,16905,27212,38586,18007,26,568,14980,709,323,7111,2212,449,264,3152,1427,389,813,3663,369,810,18208,627,2170,568,574,3815,420,11,622,638,15203,927,323,14980,4920,1461,7422,568,2643,387,264,4333,627,1548,14000,420,323,6656,311,622,638,11,813,7645,9922,58644,11,3339,622,638,596,6548,94415,304,11196,345,8248,4920,279,7645,374,264,21655,315,22999,488,323,28930,13,364,9906,2965,568,1071,11,813,7899,3428,323,1524,13,330,3923,12716,499,1618,3432,48469,41,638,596,6548,1051,2103,7029,449,51517,439,568,45135,520,279,6307,22217,13,330,4438,1131,2650,1550,499,656,430,30,9135,568,80608,11,449,264,8982,315,264,1716,627,1548,9408,832,8071,304,32854,1001,11,330,40,617,16572,13,358,1097,264,53529,315,279,10913,7771,1210,627,1,55112,1291,47944,622,638,296,26902,382,22336,2351,539,505,2212,1618,11,527,499,7673,279,6307,22217,3488,449,264,7626,7899,4286,41,638,20365,33337,11,44003,1268,311,6013,627,4071,813,19428,36496,9508,311,39786,1139,1461,11,3339,1461,2733,1093,568,1047,912,5873,719,311,4320,627,93466,29466,1359,622,638,91637,291,11,813,7899,20025,3485,264,35955,382,791,6307,22217,36065,3463,3725,627,7189,2846,93188,1359,568,1071,11,813,16630,7626,719,539,653,15674,13,330,3112,499,527,1131,12241,41,638,596,6548,29447,12616,709,311,3449,93188,596,36496,627,1548,6612,264,559,1553,1629,1523,813,35776,439,872,6548,16447,11,279,13168,1990,872,14071,323,6307,8146,38246,2403,279,4092,382,7189,22197,2846,622,638,1359,568,357,67699,291,11,279,836,8430,15234,389,813,23726,13,2030,433,574,682,568,1047,311,733,389,382,41,638,1486,520,279,12585,11,330,72891,0,3639,374,430,30,9135,93188,1427,11,330,3923,30,2564,3011,48469,77955,527,814,779,3152,30,19318,622,638,4691,11,330,7009,2351,56168,430,1648,11,555,622,30969,13,1283,1390,311,51630,279,1917,10684,16846,93188,627,46639,12825,32111,4560,311,3619,279,3492,568,1120,12570,627,7189,3782,3411,369,54594,11,323,499,617,1766,433,369,757,13,3494,2162,1359,330,39,12825,0,32111,622,638,55841,304,22047,1606,568,1120,2163,311,1505,1274,11,330,438,8985,5127,9135,627,1,34124,30,29308,0,32111,568,1071,7706,38210,330,40,1097,539,58134,11,358,2846,6129,10684,93188,58644,398,14564,11,330,12174,0,358,1518,19318,622,638,80608,449,5655,4747,627,27806,499,527,5675,11,1120,1833,1405,701,12737,3063,13,578,7160,374,743,369,433,10246,2170,622,638,1427,311,3619,1148,568,8967,11,93188,32122,3201,13,330,39,12825,30,28653,0,14144,19318,39169,622,638,13,1283,59581,1139,279,6138,627,41,638,596,6548,86988,304,10988,439,93188,29496,505,1684,11,9564,1461,8430,23838,3131,810,13,5414,20801,5352,21958,81700,311,51978,11,323,24014,550,19011,520,279,24359,315,813,6548,439,568,584,418,627,2014,19121,2055,41551,757,1148,374,701,13010,311,420,1510,6237,30,3041,1855,8712,264,4478,449,220,15,12,1041,14697,128009,128006,78191,128007,271,29815,389,279,2317,3984,11,1618,596,856,13010,311,279,1510,6237,1473,334,2321,41964,11278,68063,220,1490,15804,40,2733,264,3831,14604,3717,311,622,638,596,3752,11,5423,994,568,596,8430,43206,323,23838,13,578,4096,315,813,24014,550,16671,520,279,24359,315,813,6548,13750,390,50369,813,34104,382,334,2573,14,90198,68063,220,2031,15804,791,6237,706,1063,2294,1957,14269,11,1778,439,93188,596,16358,3113,315,7512,2403,279,29807,13,4452,11,358,2733,430,279,83240,1436,387,13241,555,7999,810,24408,477,72930,311,1521,16451,382,334,12686,11050,68063,220,1954,15804,41,373,596,3752,374,27387,11,323,813,58644,94115,58704,264,19662,23965,13,578,21976,1990,622,638,323,93188,21667,872,12742,44908,11,3339,1124,2225,7185,323,1375,15436,382,334,10343,7826,1526,287,68063,220,5313,15804,791,1917,52499,304,420,6237,374,16358,11,449,5016,12585,14769,323,264,64979,950,18921,13,358,2846,22999,311,4048,810,922,279,10913,12542,323,622,30969,596,65931,382,334,47,4628,68063,220,1399,15804,8142,1070,527,1063,13548,14269,11,279,83240,11321,264,2766,6435,304,7634,13,578,6237,1436,8935,505,264,64062,4600,311,2567,279,1957,323,24408,7366,520,264,24981,18338,382,334,18902,488,68063,220,2721,15804,791,7434,315,41416,71017,43546,449,11977,374,5016,323,41765,13,358,2846,12304,311,1518,1405,420,1917,5900,2268,28589,11,358,2846,17045,555,279,3446,323,5885,11,719,1070,596,3130,369,16048,304,3878,315,83240,323,24408,13],"total_duration":6543566949,"load_duration":26968705,"prompt_eval_count":1036,"prompt_eval_duration":23427000,"eval_count":296,"eval_duration":6450068000}

terminate called after throwing an instance of 'nlohmann::json_abi_v3_11_3::detail::type_error'
what(): [json.exception.type_error.305] cannot use operator[] with a string argument with null

So yeah... It came in part for the whole message. That's pretty wild to me.

That's pretty interesting about the use of context. What is difference between chat and context then?
Another way for me to give context, would be to provide it in prompt and then say "context complete" and it work.
I would love to know the best way to provide something like a worldbuilding context.
system are nice for defining behavior.

[JG-Adams]

Streaming is useless until this is fixed. If you could resolve the handling of partial messages that would be fantastic! :)

[jmont-dev]

I merged in a solution testing this with the generate endpoint with #21. This looks good from my tests; Let me know if this resolves your specific use case.

It looks like any replies from the server are being limited to 4090 bytes. Anything larger than that size is being fragmented into multiple messages over HTTP. This patch keeps an intermediate buffer and stores all received messages until they form a valid JSON message.

If you can confirm resolved performance I'll also apply this change to the chat and embedding endpoints.

[JG-Adams]

It work! Yay!