Using TriScale for ETX

[arurke]

Hi,

I was playing around with ETX as metric. I define ETX as attempted transmissions per successful transmission, i.e. num. TX attempts divided by num. successful TXs. Calculating the total ETX for an entire run is thus trivial. However, calculating it over time such that I could utilize TriScale metric-analysis, I met some challenges: The core issue is that transmissions may not succeed, leading to 0 in the divisor. For example, an intuitive approach would be to use ETX per transaction (MAC-layer transmissions and retransmissions of the same packet). Yet the MAC will typically give up, e.g. after 8 attempts - what ETX to set for such a transaction? Similar situations may occur if calculating ETX e.g. per minute.

One option could be to exclude such transactions, and rather catch them in a different metric (like transaction-loss) - this would be somewhat analogous to end-to-end latency and delivery ratio. This does seem precise, but it disperses the information across two values, making comparisons harder.
Another option could be to calculate it across a number of transactions which more or less guarantees there is at least one success, e.g. "ETX per 10 transactions". But introducing such an artificial value may create issues, e.g. it might not be portable to future experiments, making comparison more difficult.
A third option could be to set a fixed high ETX for the transactions, yet as above, such artificial values may be challenging.
Or finally, just calculate ETX across the entire run - losing TriScale features such as the convergence test.

Would love to hear thoughts on this as I suspect there is some angle on the entire problem I am missing. Note that the issue is probably transferable to other metrics as well.

[romain-jacob]

TL;DR
If you struggle when using a given metric, then you should consider changing it.

---

Indeed, you touch upon something interesting here: How to deal with unbounded metric values?

As you noted, it is the same problem with the delay for packets that don't get delivered. How is this dealt with in practice? Simply by using two different metrics: packet delay, and packet reception rate. It's simple, and clean from an analysis point of view, but the drawback is that you now have two different performance dimensions (delay and reliability) which leads to a set of Pareto-optimal solutions--that is, protocols what will perform better than the others in one of the two dimensions. It makes head-to-head comparison more precise, but less conclusive.

Another approach is to do some data transformation. Keeping the example of delay, one could analyze 1/d instead of d. Since delay is strictly positive, you would get rid of the infinity problem, however you could still get arbitrarily high values. This is not ideal, as it will bias the data scaling that is performed before the convergence test. Also, one should think twice before introducing "yet another metric" that readers might have difficulty to interpret. Though this can still be okay if you use the transformation for the convergence test only.

More generally, I'd (strongly) recommend asking oneself, "what is the performance dimension that I really want to capture?" In your case, why are you looking at the ETX? What statement do you want to make with it?

• Is it about energy efficiency? If so, could you measure the current/power draw instead?
• Is it about reliability? Then why not a simple packet reception ratio, where each attempt is counted as one packet?
• Is it about something else? IIRC, ETX was introduced to capture loss burstiness. If that's what you are interested in, why not flip things around, and measure the "number of losses between two successes" or "the number of retransmissions per packets"? That would give you 0 in the perfect case (no loss), 1 is the packet is lost once, etc.

Hope that helps!

[arurke]

Thanks for the reply, and apologies for the late follow-up - I had an encounter with a corona-virus.

If you struggle when using a given metric, then you should consider changing it.

This is probably hitting the nail the on the head 😄 In my case it is about reliability. I am running two scenarios with everything equal except in one there is a interference source. The aim is to measure how the interference impact transmission reliability.

Then why not a simple packet reception ratio, where each attempt is counted as one packet?

I did actually consider this early on, but found it hard to wrap my head around how to do comparison in these cases the results are binary. I am writing this without doing any testing so please bear with me: My raw-data from a run would be a series of 0's (failed attempt) and 1's (successful attempts). Let's say the success-ratio is somewhere around 70-75 % for one scenario and 90-95 % for another, and my goal is to highlight that difference for comparison. How do I go about doing it with TriScale? Using mean it seems trivial as I would get a number for each run that I could use to compare between the scenarios. However, using percentiles seems more tricky as the output from the run is either 0 or 1 which does not make for comparison directly. If I were to use the 80-percentile as a metric it would bring out the difference but surely I would want to be more precise. The only approaches I can imagine will involve some trial-and-error trying to find the most informative percentiles for metric and then compare different percentiles of KPIs. Or conversely, find a common KPI percentile and then work backwards to the percentile of the metrics. Am I on the right track, or am I missing something?

[romain-jacob]

I think your problem comes from the confusion between the metric (the aggregate statistic of one run) and the KPI (the aggregate statistic of a series of runs).

In your case, the "measurements" are binary so, as you say, using a percentile as your metric does not make much sense. You'd rather take the mean. If you want to test for convergence, you can use the packet reception ratio (which is, by definition, the mean) over a sliding window, etc.

Then, to capture the uncertainty in your results, you will repeat it, e.g. 10 times. That gives you 10 metric values (the PRRs), and you compute your KPI over those. Following TriScale recommendations, you could compute the one-sided confidence interval for the median, with a 95% confidence level. Important here! This is the median of the metric values, not the median reception success within one run.

Assuming your KPI returns 75%, the conclusion you can draw will be:
"with 95% confidence, the PRR is greater than 75% in at least 50% of the runs."

Does that make sense?

[arurke]

I was trying to avoid using mean as a metric, and I see now my clumsy description caused more confusion. I don't think I was on a fruitful direction, or perhaps simply wrong, so I'll just leave it at that. To sum up, you are indeed making sense - this was a very helpful discussion, thank you very much!

[romain-jacob]

May I ask why you were trying to avoid using the mean as a metric?

(in any case, happy that it was useful to you already!)

[arurke]

No solid rationale really: Spurred by the discussion in Section 3 of your paper I have been leaning more and more on median as opposed to mean if I cant find any good reasons opposing it. With regards to PRR I was also "inspired" by Section 5.3, making me think how I could use percentiles as metric and do comparison with such binary data. The more I think about how I would do it, the less sense it makes :)