Inconsistent Estimator?

[tux25]

Hi, I am using your package frequently to estimate WTP model and saw this post https://stats.stackexchange.com/questions/624305/mlogit-logitr-packages-fail-to-recover-true-estimates-of-mixed-logit-random-co today.
Can you please comment on that?

[jhelvy]

Thanks for bringing this to my attention, I'll look into it.

[jhelvy]

I replied with this comment. I believe the issue is simply not using enough draws for simulating the normal distributions with Halton draws. As I wrote in the comment, the default of 50 may just not be enough. I used 200 and the bias disappeared. I'm not sure what the "best" threshold is. Models with more parameters probably need more draws, so there may be some rule of thumb that scales with the number of parameters. I could easily increase the default number to perhaps 200 or so just to make sure this issue isn't there for models with smaller numbers of parameters. But the package already posts a warning when there are more random parameters and suggests using Sobol draws.

[jhelvy]

More updates on the stackexchange post. Even with higher numbers of draws there is still some small bias. I suspect the Halton draws may be the culprit. Still digging into the issue.

For sure though I am likely going to modify how the draws are done by default. I like how Stata’s cmmixlogit does it by using a flexible number of draws based on the model formulation, which makes sense (models with more random parameters really should use more draws).

[JediKnight5]

Hi @jhelvy, as you have preferred, I am continuing the discussion here instead of following up in the Stackexchange thread.

Regarding your last comments in the Stackexchange thread:

• All packages offer different ways of how draws are simulated. While your logitr package offers Halton and Sobol draws, mlogit's default are pseudo random number draws but users of this package can switch to Halton draws. cmmixlogit allows using the Hammersley sequence (the default), the Halton sequence or pseudo random numbers. Since I set the random coefficient to be normal distributed there should be no issue during the simulation with transforming this distribution to other distributions such as the log-normal, the truncated normal etc.
• I also checked the optimizer of the different packages and realized that your logitr package makes use of NLopt. According to my understanding, the most recent version of mlogit is using a hand-written optimizer instead of using the maxLik package. Stata’s cmmixlogit is using Stata’s default maximum likelihood routine ml, see https://www.stata.com/manuals17/rml.pdf. Moreover, according to my understanding, the default optimization algorithms used are Broyden-Fletcher-Goldfarb-Shanno (BFGS) for logitr and mlogit and a modified version of the Newton-Raphson (NR) algorithm for cmmixlogit. However, changing the algorithm to be used in logitr and mlogit didn’t change the outcome in my simulations.

I have attached my R script and the resulting dataset of comparing the performance of both logitr estimators (Halton and Sobol) to both mlogit estimators (pseudo random numbers and Halton) in a setting with $NumDraws = 100$ and $N=1000$. Don’t get irritated by lines 28-90 which can be easily simplified, I just wanted to allow for different constants across the alternatives in my estimation function. Moreover, I have attached my Stata code – which reads in the data generated by the R code – and the corresponding results dataset as csv file.
material.zip

If you need any additional information, please let me know.

[jhelvy]

@JediKnight5 I appreciate you looking into this and adding your code here. I think this will be a lot easier to manage in this issue, and once we figure this out we can update the SO post and link back to this conversation.

Based on your Stata code, it looks like you exported the simulated data from the R code to csv files and then read those in to Stata for each model estimation? I think that's fine, I just wanted to make sure I'm understanding how the Stata simulations are being done.

I'm actually thinking a slightly different way of organizing this is to write a separate R script to generate the data and then save all of the data as one large dataframe, keeping track of things like the seed and iteration. If you save it as a .parquet file (arrow::write_parquet()) it will be quite small in size. Then that never needs to be updated - it's generated only once, and we can then use it to stress test all sorts of things in all of these packages.

I'm still thinking the consistent bias we're seeing is due to 1) the draws, 2) the optimizer, or 3) a combination of both. That you're getting the same results with a different algorithm doesn't necessarily rule it out because it could also be due to differences in convergence criteria. Basically, regardless of the algorithm used, it may still be stopping prematurely, and we may need to modify convergence criteria. I believe all of those controls can be passed in a call to logitr().

As for the draws, another idea is to try to export the draws being used themselves and then use those same draws in the other packages. I'm not sure if this can be done in the other packages, but in logitr you can pass a set of "standard draws" using the standardDraws argument. I haven't tested that argument too thoroughly, but I'd like to see if we could generate draws in different ways and then see what happens with different approaches. This would allow the ability to test strategies other than Halton and Sobol in logitr. The "standard draws" are supposed to be draws from a standard normal distribution, which are then scaled / shifted based on the model parameters. You can see how this is done in the code. The getStandardDraws function obtains the standard draws (using either halton or sobol), then the makeBetaDraws function shifts and scales those draws according to the parameters. I believe this is the way the other packages also work, but I'm not sure. As far as I can tell, I don't see what might lead to a slightly positive bias from using this approach. Note also that I'm using a separate package for making the halton or sobol draws (randtoolbox::sobol and randtoolbox::halton). I figured that package must be using a well-vetted approach for making these draws, but perhaps I'm wrong.

[JediKnight5]

@jhelvy:
Regarding your Stata question: Yes, your assumption is correct. I exported the data set I generated within my R code to a csv file (per iteration of the Monte-Carlo simulation).

Unfortunately, Stata cannot read in parquet files but I collected the data I generated for testing the different estimators in the following parquet file: LINK IS EXPIRED

I like your idea of keeping the draws of the standard normal distribution as a fixed input and compare the estimators’ performance accordingly. I think mlogit allows for this but, unfortunately, Stata’s cmmixlogit does not. I won’t have time in the next couple of days since this will probably require me to dive deeper into the code of logitr and mlogit but I will have a look at it by the end of next week.

The following simple piece of analysis comparing halton draws from different packages to pseudo random number draws doesn’t suggest that there is an issue with the randtoolbox package used in logitr.
halton_vs_pesudo_random_draws.zip

[jhelvy]

All great progress on this! I think doing as much as we can to hold inputs fixed (simulated data, draws, etc.) and just vary the model settings will lead us to the path that gets down to the bottom of these differences.

One other thought I had was to also keep track of the log-likelihood achieved at the solution for each simulated dataset. Ideally these should all be converging to the same solution, but if they're not we should be able to see that by comparing the log-likelihoods. That is, if we find that the log-likelihood is generally better with Stata’s cmmixlogit than logitr, that means logitr is just stopping pre-maturely. That may be able to be addressed by changing the optimizer convergence settings. If not, then I could consider using a different optimization routine. I chose NLopt because it is so well-established and well-tested in the open source world. If on the other hand the log-likelihood values are looking about the same, then we can narrow in on the draws (or perhaps something else even).

[JediKnight5]

I had a look at the codes of logitr and mlogit to better understand how to specify the respective standardDraws and the random.nb options.

My understanding is that logitr is expecting a matrix with dimensions NumDraws * number of explanatory variables in the model as input for the standardDraws option. @jhelvy: Can you confirm this? In this case, I have two related questions:

1. What is the purpose of drawing standard normal for fixed coefficient variables? Practicality reasons? I saw that you zero them out in line 27 of https://github.com/jhelvy/logitr/blob/master/R/draws.R
2. My understanding is that we would need in total $K$ * $N$ * $R$ standard normal draws where $K$ is the number of random coefficient variables, $N$ the number of individuals of a balanced panel dataset and $R$ the number of repetitions / draws. In other words, we would need a matrix of $R$ * $K$ standard normals for every individual time combination.

Regarding mlogit’s random.nb option, I think this option has erroneously no impact at all. It is not called in any line of https://github.com/cran/mlogit/blob/0ebddceda198eb567975e8f7da586d29cd88fda3/R/mlogit.R and it seems that random.nb is always generated by the make.random.nb function of https://github.com/cran/mlogit/blob/0ebddceda198eb567975e8f7da586d29cd88fda3/R/lnl.rlogit.R. Moreover, passing a meaningless argument type to this option, such as a string, doesn't lead to a code break or at least a warning.

Disclaimer: I have spend a considerably amount of time to look at both codes, but please apologize if I have overlooked something given the number of lines and code nestings.

[jhelvy]

I really appreciate the time you've spent on this. I know this is super tedious and time-consuming, but it's really helpful. I'm quite swamped at the moment with a lot of other responsibilities, so this would take me a lot longer to get to without your help.

Yes, you are correctly understanding how I have set up the creation of the standardDraws matrix. In reviewing my code, it does seem redundant to be taking standard draws for the fixed (non-random) coefficients. They get zeroed-out in line 39 of the draws.R file. We still need them because computing the utilities and log-likelihoods for a model with any random parameters is a different function, and it's much faster to do everything in matrixes. So eventually those 0s get replaced with the fixed coefficient repeated.

But taking draws for them may be an issue. It may be better / more appropriate to only take normal draws for the random coefficients and just bind on a matrix of zeros for the fixed parameters. In fact, this could very well be part of the problem. Just changing how this is done in the getStandardDraws function may fix things.

As for the dimensions, I am only taking $R$ * $K$ standard normals in total, not for every individual - time combination. This has to do with how I set up the calculations for computing the utilities and ultimately the log-likelihoods and gradients. I am fairly sure this is an appropriate approach.

[JediKnight5]

I actually cannot see how your described procedure of drawing standard normal draws for fixed coefficients and then replacing them with repeated means should adversely affect the estimation.

After having another look into your code, I realized that I struggle to understand the purpose of the makePartials function and why it takes as argument a matrix of $R * K$ standard normals. You seem to be very convinced regarding this approach, so might I ask you to share a reference or some additional explanation? My approach is the one briefly described on slide 24 of this slide deck but they might exist better / more efficient ways.

Given the ineffective random.nb option of the mlogit function, I have reached out to the mlogit author. Unless a change to the code is made either by him or by us creating a modified version, our idea of comparing mlogit and logitr estimates using the same set of draws is impeded. While this seems to be just a minor modification, I would prefer to rely on the author and his knowledge on his code instead of rewriting it by myself.

[jhelvy]

Okay, I think we may be getting down to something here. First, the makePartials function is essentially storing a pre-computed part of the gradients that doesn't change across iterations of the search algorithm. You can see on page 9 of the JSS article on logitr what I mean by this:

That page only shows the general idea for simple logit, but the idea should hopefully be clear. For mixed logit, part of the partial derivatives involves multiplying by the standard normal draws. So effectively you end up with a big matrix of constants that you then use when computing the gradients inside the optimization search. This is one of the big ways to speed up computations.

However, this may be where there is an issue. I'll have to go back to how I computed the gradients, but I believe I set this up where I don't have different draws across individuals. This may not matter in a case without a panel structure, but in your case where you do have the panel structure (denoted with the panelID input) this may cause issues.

One quick check on this would be to simulate a dataset without the panel structure to see if the bias goes away. If it does, then that very well may be where the issue lies.

[jhelvy]

Btw, I built this all based on Train's book. You can see from section 6.6 of chapter 6 where he describes the estimation process. In 6.7 on panel data he writes:

The simplest specification treats the coefficients that enter utility as varying over people but being constant over choice situations for each person.

And under equation 6.3 he writes:

The only difference between a mixed logit with repeated choices and one with only one choice per decision maker is that the integrand involves a product of logit formulas, one for each time period, rather than just one logit formula. The probability is simulated similarly to the probability with one choice period. A draw of β is taken from its distribution. The
logit formula is calculated for each period, and the product of these logits is taken. This process is repeated for many draws, and the results are averaged.

[jhelvy]

Alright I just ran another simulation where I removed the panel structure from the data. It was 10000 people, each with only one choice observation. The same results show - slight bias on the SD term. Still rather puzzled about why this outcome remains so consistent, but I don't think it's the panel structure.

[JediKnight5]

Okay, I think we may be getting down to something here. First, the makePartials function is essentially storing a pre-computed part of the gradients that doesn't change across iterations of the search algorithm. You can see on page 9 of the JSS article on logitr what I mean by this: ....

Thanks for this reference, this was very helpful and made your estimation strategy clear for the simple case. Moreover, I got a better understanding of your code.

I also run a simulation with $N = 20000$ and $T=1$ to reflect the total sample size $N*T$ which I had before in my simulation in order not to alter the ratio of $\frac{\sqrt{N*T}}{R}$. I got the following results in terms of absolute bias:

	mlogit pseudo random numbers draws	mlogit halton draws	logitr halton draws	logitr sobol draws
$\beta_{x1}$	1%	0%	0%	0%
$\mu_{\beta_{x2}}$	2%	0%	1%	0%
$\sigma_{\beta_{x2}}$	4%	0%	2%	4%

I will double check on that by running another simulation but I think there two are takeaways:

• mlogit halton draws estimates are unbiased in a cross-sectional setting. Given their superior performance in comparison to the mlogit with pseudo random numbers estimates it looks like that some of the bias might be attributable to the performance difference in spanning up a normal distribution between pseudo random and Halton draws. However, I am quite skeptical due to the magnitude of the difference. Therefore, I will double check this.
• logitr estimates (both halton and sobol draw types) still seem to be affected by another source of bias in a cross-sectional setting.

[JediKnight5]

Btw, I built this all based on Train's book. You can see from section 6.6 of chapter 6 where he describes the estimation process. In 6.7 on panel data he writes:

The simplest specification treats the coefficients that enter utility as varying over people but being constant over choice situations for each person.

And under equation 6.3 he writes:

The only difference between a mixed logit with repeated choices and one with only one choice per decision maker is that the integrand involves a product of logit formulas, one for each time period, rather than just one logit formula. The probability is simulated similarly to the probability with one choice period. A draw of β is taken from its distribution. The
logit formula is calculated for each period, and the product of these logits is taken. This process is repeated for many draws, and the results are averaged.

Thanks for this further explanation and reference. My simulation actually consists of this simplest specification where the random coefficient $\beta_{x2_n}$ only varies across people.