Missing Features

[JamesYang007]

• Data does not support matrix shape
• ParamView does not support matrix shape
• Param does not support matrix shape
• Constant does not support vector or matrix shape
• Dot only supports between Matrix and vector
• Binop does not support matrix
• No unary operation support
• Bernoulli should be able to work with NUTS so long as the variable that's distributed bernoulli is data
• Even more optimizations for when variable expressions are of fixed size not needed based on new design
• MH should have a config too like NUTS. Can we generalize?
• multiple chains/cores
• Expression Checker
  • Generative Model Checking:
    • no variable gets assigned a distribution more than once
    • no variable gets assigned a distribution that references the same variable
    • no distribution references variables below it (ones that are assigned a distribution after current variable)
• NUTS covariance adaptiveness
• Add more distributions
• Add more MCMC Kernels
  • adaptive metropolis
  • gibbs sampling
  • RWM
  • MALA
  • SM-MALA
  • HMC + adaptation
• Add ability to trace a function of the MCMC state (i.e. logpi/Hamiltonian)
• subsetting vector with operator[], operator()
• BoundedAbove

[ludkinm]

Hi, I like this project!
It's cool to see a project like STAN written all in C++ with the ability to add more samplers/inference methods.

I'd like to add some bullet points to this list:

• Add more distributions

• Add more MCMC Kernels

• Add ability to trace a function of the MCMC state (i.e. logpi/Hamiltonian)

I'm looking at implementing some MCMC kernels like RWM, MALA, SM-MALA, HMC and some adaptive schemes for these
(just HMC, not NUTS https://arxiv.org/pdf/1206.1901.pdf)
and some others kernels which I have been working on.

I have looked at pymc3 and pyro in python but find python is very slow for reverse mode autodiff.

While I am here, is this project aiming to replicate the features of say STAN or pymc3, or focused on just MCMC say?

[JamesYang007]

Hey, thanks for checking it out (and also contributing)! I'll append the list for sure - those are great points. A big goal of mine is to add a lot more kernels, some of them you already mentioned above. Feel free to put up an issue about any of them as an enhancement and take a crack at it!

We have also looked at pymc3 and indeed it was very very slow. I think it was like 40x slower than autoppl and STAN with the gaussian model benchmark example in the README. But a bigger issue was that it actually crashed on some examples (linear regression benchmark example), so it's just not as stable. I kept the pymc3 code for both examples as souvenir here - you can try running it too.

At the moment, I have no particular direction I'm thinking of with this project. But I will say that the end-goals I had in mind were 1) to allow researchers to compare accuracy and performance among different MCMC methods and 2) allow researchers to extend the library to implement their own MCMC method. While it does sound like I am focused purely on MCMC, I definitely think supporting other features in STAN and pymc3 would be attractive, so I am open to that as well. I just had the impression that posterior sampling was the main objective for those two. Did you have something in mind?

[ludkinm]

That is exactly what I had in mind! A test bed library for adding new samplers is what I would love (STAN seems like a closed system...)

Right now I am looking at adding HMC with adaptive covariance to my fork of the library.

[JamesYang007]

Awesome, sounds great! Yeah, I had the same impression of STAN.

I wrote an adaptive method for NUTS which may be useful for your HMC method. It should be under mcmc/hmc. Looking forward to seeing your HMC implementation!

[JamesYang007]

#46 adds support for matrix for existing operations