We need more maxent problems with known solutions

[ericpashman]

As I was writing a general moment-constraint constructor, I realized that (I think) there's a bug in my definition of (.=.) in Moment.hs. I think I'm mapping the function f in f .=. c over the probabilities rather than over the values, which would make constraints using variance wrong. ...

To test this sort of thing we're going to need dig up some more basic examples that yield known maxent distributions, particularly ones that arise from higher-order moment constraints. Or maybe we can work some out with pen and paper; most of the maxent distributions I can think of offhand have infinite support, unfortunately.

I'll look around for some examples.

[jfischoff]

Maybe we need a reference implementation to compare against. Or several and take the vote.

This seems pretty high priority to figure out.

[ericpashman]

Yes, and we should think about how we can test the different implementations we already have against each other. We should be able to compare the results of using linear and of using maxent for problems with linear constraints, for instance.

We can also probably just throw in some more sanity checks: verify that the returned probabilities sum to one, verify that constraints are satisfied, etc. Some of these can be QuickCheck properties, e.g., for all lists of integers xs and for any mu between the max and the min of xs, it should be that maxent tolerance xs [average mu] gives a result ps whose dot product with xs is close to mu. And so on.

But we also just need some problems with known solutions to test against. ...

[jfischoff]

On Thu, Oct 9, 2014 at 5:23 PM, Eric Pashman [email protected]
wrote:

Yes, and we should think about how we can test the different
implementations we already have against each other. We should be able to
compare the results of using linear and of using maxent for problems with
linear constraints, for instance.

Good idea.

We can also probably just throw in some more sanity checks: verify that
the returned probabilities sum to one, verify that constraints are
satisfied, etc.

I assumed there were tests like that, but if not we should add them.

Some of these can be QuickCheck properties, e.g., for all lists of
integers xs and for any mu between the max and the min of xs, it should
be that maxent tolerance xs [average mu] gives a result ps whose dot
product with xs is close to mu. And so on.

Right I'm still struggling to understand how to close, close should be.

But we also just need some problems with known solutions to tests against.
...

Sure. There are few examples in E.T. Jaynes works.

I think generating them is probably best.

I'll try to think of some ways.

—
Reply to this email directly or view it on GitHub
#9 (comment).