Transfer Entropy Test Question

[jwhowse4]

I have done what I thought was a simple test of your transfer entropy code, but I do not understand the answers that I am getting.

Generate two realizations from the same uniform distribution.
unif_1 = scipy.stats.uniform.rvs( size=15000, loc=-0.25, scale=1.0 )
unif_2 = scipy.stats.uniform.rvs( size=15000, loc=-0.25, scale=1.0 )

Compute the transfer entropy using the 'ordinal' method. First compute the transfer entropy where both samples are 'unit_1'.

infomeasure.transfer_entropy( unif_1, unif_1, approach='ordinal', embedding_dim=4, base='e', step_size=1, prop_time=0, src_hist_len=5, dest_hist_len=5 )
Out: np.float64(0.0)

The answer is 0 which is what I expected.

Next compute the transfer entropy where one sample is 'unif_1' and the other is 'unit_2'.

infomeasure.transfer_entropy( unif_1, unif_2, approach='ordinal', embedding_dim=4, base='e', step_size=1, prop_time=0, src_hist_len=5, dest_hist_len=5 )
Out: np.float64(0.8017785056845319)

I expected this answer to be very small if not zero. Why is this value not near zero?

The situation becomes more confusing for me when I compute transfer entropy using the 'metric' method. First compute the transfer entropy where both samples are 'unit_1'.

infomeasure.transfer_entropy( unif_1, unif_1, approach='metric', minkowski_p=numpy.inf, k=4, base='e', noise_level=1.0e-8, step_size=1, prop_time=0, src_hist_len=5, dest_hist_len=5 )
Out: np.float64(2.6662238315844906)

I expected this answer to be zero as it was for the 'ordinal' case. Why is this value not zero?

Next compute the transfer entropy where one sample is 'unif_1' and the other is 'unit_2'.

infomeasure.transfer_entropy( unif_1, unif_2, approach='metric', minkowski_p=numpy.inf, k=4, base='e', noise_level=1.0e-8, step_size=1, prop_time=0, src_hist_len=5, dest_hist_len=5 )
Out: np.float64(3.1566700132055603)

Why is this value not near zero?

I would really appreciate any thoughts or suggestions. Am I doing something wrong in using these functions?

[cbueth]

Hey @jwhowse4, thanks for using infomeasure and for checking in. With the detailed description I was able to easily reproduce your results.
To get the most important off the table, you are using the package correct and it produces correct outputs. To determine if a TE value is significantly different from zero, you should use the statistical tools provided, see Statistical Tests with the Example for Transfer Entropy, but more specifically the background on Effective Transfer Entropy which you can also find an example for on the Ordinal Transfer Entropy Page which has to be used instantiating an estimator:

>>> import scipy
>>> import infomeasure as im
>>> unif_1 = scipy.stats.uniform.rvs( size=15000, loc=-0.25, scale=1.0 )
>>> unif_2 = scipy.stats.uniform.rvs( size=15000, loc=-0.25, scale=1.0 )
>>> te_estimator_ordinal = im.estimator(unif_1, unif_2, measure="te", approach='ordinal', embedding_dim=4, base='e', step_size=1, prop_time=0, src_hist_len=5, dest_hist_len=5 )
>>> te_estimator_ordinal.effective_val()
np.float64(0.00027740682252919147)

Then with te_estimator_ordinal.statistical_test(n_tests=100) you can calculate a $p$-value to figure out if the significance of the initial result.

I will explain the outputs you got step by step:

Ordinal/Permutation/Symbolic Estimator

In your first test with unif_1 and unif_1, the result is 0.0. This is because the estimator sees identical permutation patterns in both the source and the destination. For the 'ordinal' method, Transfer Entropy (TE) is calculated as: TE = CMI(target_future; source_past | target_past). When source == target, the source_past and target_past contain exactly the same information (just shifted). In discrete/symbolic terms, if the history length and embedding are sufficient, the source_past provides no new information about target_future that isn't already contained in target_past, leading to a theoretical and numerical TE of zero.
In your second test with independent realizations of unif_1 and unif_2, you get ~0.80. This is a contribution of a finite-size bias. Even if two processes are independent, a finite sample of symbolic patterns will almost always show some coincidental "information transfer" because the frequencies of patterns don't perfectly match their theoretical probabilities. It seems high, but with an embedding_dim=4, you have $4! = 24$ possible symbols, and with src_hist_len=5 and dest_hist_len=5, the joint space of (future, past_target, past_source) becomes extremely large ($24^{11}$ possible states). With only 15,000 samples, the contingency tables are extremely sparse. In discrete entropy estimation, sparsity leads to a significant overestimation of mutual information and thus TE.
Lastly, even for truly i.i.d. (independent and identically distributed) uniform data, finite samples won't perfectly represent uniform distribution over ordinal patterns.

Metric (KSG) Estimator

Unlike discrete methods, with the "metric" Kraskov-Stoegbauer-Grassberger (KSG) estimator, which is a K-nearest neighbors (KNN) approach designed for continuous data, does per-se not automatically return zero for identical signals.
When the source and destination are identical, the data points lie on a lower-dimensional manifold within the high-dimensional joint space. KNN estimators struggle with this singularity because many points may have identical coordinates or distances. The noise_level=1.0e-8 also plays an important role for this. This noise is added to break ties and handle the manifold problem. However, the resulting TE value depends heavily on the relationship between this noise and the local density of points. For identical signals, the "self-information" is picked up, but it doesn't cancel out perfectly in the KSG formula due to how distances are calculated in different dimensions.

For unif_1 to unif_2, similarly to the ordinal case, this is not zero due to statistical bias. KSG is designed to be low-bias, but it is never zero-bias for finite $N$. For independent variables, the expected value of the estimator is close to zero, but for high-dimensional embeddings (11 dimensions in total: 1 future + 5 target past + 5 source past), the curse of dimensionality causes the neighbour searches to become less meaningful, usually resulting in a positive bias.
This bias was already initially discussed by Kraskov et al. (2004, https://doi.org/10.1103/PhysRevE.69.066138).

You might also increase $k$ to e.g. $6$ to combat overestimation.

---

Looking into this again, I found a recent preprint that suggests an approach of reducing this bias https://arxiv.org/abs/2506.16215. For many discrete approaches and especially for entropy there are already bias corrections (Discrete Entropy Estimators Guide Page), and infomeasure also supports their MI and TE variants. In your case I'd still suggest just using the effective value with the estimating approaches you are using.
For large histories/state spaces the need for more data is increased, so if you have more data this might help, or reducing the history length.
With the given keywords you should be able to find other literature and I hope this can point you in the right direction, with statistical testing or effective values.

Please let us know if this helped you and if there is any specific issue you see or a place in the documentation you would have found this information useful. If all is clear, please mark this issue as closed 🙌

[jwhowse4]

Thank you for your timely response. I understand your arguments, but I find myself wishing that the errors were not so large.

In my actual problem the two processes X and Y are time series sampled from variables that are continuous valued in a bounded range. This is one reason I chose uniform random variables as an example. Another reason I chose uniform random variables is because it is my understanding that the differential entropy for random variables with finite support is maximal for the uniform distribution. Since I started working on this problem I have been wrestling with three issues about the transfer entropy estimates.

1. Removing the estimation error (the variance more or less) due to finite sample effects.
2. Removing the approximation error (the bias more or less) due to the choice of function to approximate the probability distribution.
3. Normalizing the transfer entropy in order to obtain a value in the interval [0, 1].

It seems to me, perhaps incorrectly, that the effective transfer entropy calculation is about addressing item 1 in the above list. It seems to me that ideally the estimation error is the difference between the value obtained for a single sample of size N and the mean over all samples of size N. I do not understand how randomly permuting samples from a time series gives a good estimate of the mean over all samples of size N, since all the permuted sample sets will most likely have a different autocorrelation structure than the original time series. For time series, it seems to me that using a set of block bootstrap samples would give a better estimate of the mean over all samples of size N. For time series with a long range autocorrelation structure even this approach may be ineffective. Does this seem reasonable to you?

I am not sure how to address the approximation error. For distributions with finite support perhaps the size of the transfer entropy between two uniform distributions with the same support is an upper bound to the approximation error? I am just spitballing here.

Finally I have been considering normalizing the differential transfer entropy. I found the following interesting paper.

[](https://www.ece.ualberta.ca/~tchen/papers/DuanP_TCST13n6.pdf)

On page 8 in Equation (18) they suggest the normalized differential transfer entropy formula

NTE(X -> Y) = TE(X -> Y) / [ H_0(Y) - H( Y_{k + 1} | vec{Y}_k^{(h)} ) + TE(X -> Y) ]

where H_0(Y) is the differential (continuous) entropy of Y with a uniform distribution. They state that "If Y is uniformly distributed and the information about Y explained by the history of both X and Y is completely explained by the history of X, which means H( Y_{k + 1} | vec{Y}_k^{(h)} ) = H_0(Y), then according to (18) we obtain NTE(X->Y) = 1." Unfortunately the errors discussed in my first post means that H( Y_{k + 1} | vec{Y}_k^{(h)} ) != H_0(Y) when Y is replaced with a uniform distribution, so this normalization does not really work.

While you are obviously under no obligation to resolve any of my problems, if you have any thoughts about them I would appreciate your insights.

[cbueth]

Hey James, thanks for the follow up and digging into this more. Sorry for the delayed reply. I agree, what you point at are genuinely non-trivial issues and what you point out aligns wth my understanding.

About surrogate testing and block bootstrap: you're absolutely right that plain random permutation is a poor null for autocorrelated time series. It destroys the temporal structure of the source, so the resulting null distribution corresponds to a world where the source is i.i.d.; which may be very different from your actual null hypothesis of "no transfer of information, but the source still has its own dynamics intact." Looking at what infomeasure currently offers: the permutation test shuffles the source to break causal dependencies while preserving marginals, and the bootstrap test resamples with replacement to assess estimator variability. Both are already available. What neither of these does is preserve the autocorrelation structure of the source under the null, which is exactly what block bootstrap would address.
To my understanding, block bootstrap resamples contiguous blocks of the time series with replacement, which preserves short-range dependence within blocks. This means block length would be a central tuning parameter, in the order of the autocorrelation length of the series. For processes with long-range dependence this is still tricky where the block length needed can become impractically large, but for weakly dependent processes it's a substantial improvement over i.i.d. permutation. Compared to iterated amplitude-adjusted Fourier transform (iAAFT) surrogates, block bootstrap has the advantage of making fewer assumptions about the structure of the process (iAAFT assumes stationarity and a specific spectral structure), while being straightforward to implement.
Adding block bootstrap as a third statistical testing method to the package would be a relatively natural extension given the existing infrastructure. The core logic would mirror the existing bootstrap test, but instead of sampling individual points with replacement, it would sample blocks of a user-specified length. If you're interested in contributing this, it would be a very welcome addition and a genuinely useful one. Feel free to open a PR and I'm happy to help with review and integration.

I'd actually encourage you to think of block bootstrap as primarily addressing point 1 (variance/finite-sample noise in the significance assessment) rather than point 2 (estimator bias). The bias from high-dimensional sparse contingency tables or from KSG in high dimensions is a property of the estimator itself and won't be reduced by changing the surrogate method. It needs to be addressed through bias-corrected estimators, reduced embedding dimensions, or more data. Your intuition about using TE between two independent uniform processes as an empirical noise floor for the bias is pragmatic and not unreasonable as a sanity check, though as you noted it's a heuristic rather than a principled correction.

Regarding the Duan et al. (2013) normalisation: the formula in Eq. (18) is a nice framework for interpretability, but it implicitly assumes a low-bias estimator, as they use a kernel-based approach for the PDF estimates, which behaves better for continuous bounded-support data like yours. The breakdown you observed (where $H(Y_{k+1} | Y_k) ≠ H₀(Y)$ even for uniform i.i.d. data) is essentially the same finite-sample/high-dimensional bias problem discussed earlier, now surfacing in the denominator of the normalization. So the normalisation addresses interpretation (getting a [0,1] scale with a meaningful upper bound) but not the underlying estimation problem. You'd need to resolve point 2 first before Eq. (18) becomes reliable. Your idea of using TE between two independent uniform processes as a proxy for the approximation error floor is a reasonable heuristic in that spirit, though it conflates bias and variance since both contribute to that value.