Big output values crowd out other columns

[ashiklom]

When SIPNET produces really large values for some variables, the fixed width of the columns means these values can get crowded out by adjacent columns (there are no spaces on which to separate the lines).

Here's an excerpt:

nUptake      ch4  nppStorage  bcdeltaC  bcdeltaN
   -nan     -nan-100031.5842      -nan      -nan
   -nan     -nan-100250.3078      -nan      -nan
   -nan     -nan-100560.4010      -nan      -nan
   -nan     -nan-100886.9315      -nan      -nan
   -nan     -nan-101101.1909      -nan      -nan

Notice how there's no space between ch4 and nppStorage values. (A more complete excerpt is below).

Conceptually, the simple fix is to replace fixed-width outputs with comma- or tab-separated values. That will require coincident fixes in PEcAn.SIPNET::model2netcdf.SIPNET, but the fix is straightforward (just an argument to read.table).

(I realize this shouldn't happen and the values here are probably unrealistically large indicating some configuration issue...but failure to parse the output file is not how one should find out about this).

Details

year day  time plantWoodC plantLeafC woodCreation     soil coarseRootC fineRootC   litter  soilWater soilWetnessFrac     snow      npp      nee   cumNEE      gpp rAboveground    rSoil    rRoot       ra       rh     rtot evapotranspiration fluxestranspiration     minN  soilOrgN    litterN       n2o nLeaching  nFixation  nUptake      ch4  nppStorage  bcdeltaC  bcdeltaN
2016 258 16.41 -100031.58   77298.51         0.00     -nan     5910.69   5783.85     -nan      6.913           0.581     0.00 -126.823      nan      nan    2.810      128.112     -nan    1.521  129.633     -nan     -nan         0.10872744              0.6179     -nan      -nan       -nan      -nan      -nan       -nan     -nan     -nan-100031.5842      -nan      -nan
2016 258 19.41 -100250.31   77421.07         0.00     -nan     5910.57   5783.57     -nan      6.807           0.572     0.00 -218.724      nan      nan    1.263      218.457     -nan    1.529  219.986     -nan     -nan         0.10567717              0.6084     -nan      -nan       -nan      -nan      -nan       -nan     -nan     -nan-100250.3078      -nan      -nan
2016 258 22.41 -100560.40   77543.63         0.00     -nan     5910.46   5783.30     -nan      6.703           0.563     0.00 -310.093      nan      nan    0.795      309.359     -nan    1.529  310.888     -nan     -nan         0.10487523              0.5991     -nan      -nan       -nan      -nan      -nan       -nan     -nan     -nan-100560.4010      -nan      -nan
2016 259  1.41 -100886.93   77666.17         0.00     -nan     5910.35   5783.03     -nan      6.597           0.554     0.00 -326.531      nan      nan    0.738      325.746     -nan    1.522  327.269     -nan     -nan         0.10575730              0.5898     -nan      -nan       -nan      -nan      -nan       -nan     -nan     -nan-100886.9315      -nan      -nan
2016 259  4.41 -101101.19   77788.72         0.00     -nan     5910.23   5782.76     -nan      6.498           0.546     0.00 -214.259      nan      nan    1.386      214.131     -nan    1.515  215.646     -nan     -nan         0.09916346              0.5805     -nan      -nan       -nan      -nan      -nan       -nan     -nan     -nan-101101.1909      -nan      -nan

[dlebauer]

I'm wondering if the bug is in the writing or in the numbers being written, ie:

• Would this be fixed by appropriate precision?
• Are these realistic magnitudes of NPP, or would this be fixed if the model was constrained to realistic values (Add range constraints to model inputs and throw error for out of range inputs #278)?

[ashiklom]

I mean, the numbers here are almost certainly really bad because I put zero effort into proper initialization of SIPNET here; I was just trying to get some workflows working. So we shouldn't encounter this during "real" use.

That said, this is a very roundabout way to discover the model is producing bad values. If there's a threshold beyond which we want SIPNET to error, we should throw an informative error outside of those thresholds (in particular, I wonder if we should throw errors for nan values in any of the output states, since those are very likely indicative of bad output).

But if the philosophy is "let the model run and let the user decide if values are bad", then the data should be readable by "normal" means.

To me, this is straightforwardly a bug in output formatting. The simplest resolution is to bump up the width of the fixed width columns (which "buys" us a few more orders of magnitude before we hit the same error). A better solution is to replace fixed width output with tab- or comma-separated output.

[dlebauer]

@ashiklom is #278 consistent with your suggestion (it is explicitly trying to capture bad errors)? Is it a matter of increasing its priority?

Other than basic range checks for plausible values, I don't think there is a way for sipnet to know if numbers are "bad" - that sounds like a validation task.

I agree it seems like nan should error. That would be easier to implement than range checks.

[ANAMASGARD]

Sir @dlebauer I would I like to work on this issue could you please assign it to me , Thank You .

[dlebauer]

@ANAMASGARD Could you please outline the approach that you would take?

@Alomir Do you have any guidance?

[Alomir]

@dlebauer

To me, this is a requirements question, which seems above my level. Specifically:

• How does this fit with the ticket for enabling sipnet to output NetCDF?
  • Do we still need text output when that is in place?
• How much of a change can we (i.e., PEcAn) handle at this point? Especially given upcoming deadlines...
  • Converting to CSV seems the safest, but that would require updating all downstream tools as well as a lot of tests
• Is binary output possible/desirable?
• How about scientific notation? (%g format)

If all we want is a short-term easing of the "we might hit too large numbers in rather extreme edge cases" worry, then I'd suggest:

• Analyze the output data to see which values could get large
• Add one or two spaces as padding for those columns

[Alomir]

Hmmm, that's not terribly helpful. I believe there is a higher-level "what do we want to do with text output" question that can be separated from this ticket. Given that, though, I think doing the simplest change here is warranted (the last part of the above comment).

[Alomir]

Created #323 for the higher-level question.

[ANAMASGARD]

Hi @dlebauer,
After digging into the code, the issue lives in src/sipnet/sipnet.c — specifically in outputState() (around line 453). The format strings rely on the padding from printf width specifiers to separate columns (e.g. %12.4f), but width is a minimum, not a maximum. When a value actually needs all 12 characters (like -100031.5842), the padding disappears and the column smashes into its neighbor. The bug is most visible between ch4 and nppStorage because the fprintf for ch4 doesn't end with a trailing space, and the next fprintf starts directly with %12.4f — so the only separator is padding that isn't always there.
Aligned with @Alomir's and @ashiklom's suggestion of the minimal-change path, my plan is:

• 1. Add explicit space separators between adjacent columns in outputState() so we don't rely on padding alone.
• 2. Widen format specifiers for columns prone to large magnitudes (starting with plantWoodCStorageDelta).
• 3. Update the header strings in outputHeader() to stay aligned.
• 4. Regenerate the smoke test reference outputs in tests/smoke/*/sipnet.out and verify only whitespace changes (no numeric drift).
• 5. Add a small regression test that calls outputState() with extreme values and asserts the line is parseable by \s+ splitting (matching what tools/smoke_check.py does).

A couple of questions before I start:

• Are you okay with me keeping the NaN-as-error change (Add range constraints to model inputs and throw error for out of range inputs #278-adjacent) as a separate follow-up PR to keep this one focused?
• For columns I should widen — would you like me to audit all storage/delta-tracked variables, or keep it strictly to plantWoodCStorageDelta for now?

Happy to adjust based on your feedback. Thanks!