Combiner accepting DQ arrays (bitwise value combinations)

[SaOgaz]

Hi,

i was wondering if there are any plans to have Combiner take in a DQ style array for masking, (i.e. scaling to zero). I've been getting a lot of requests for a mscombine replacement, and I think people use this feature a lot. If there are no plans for this, I'm thinking about adding a wrapper around Combine to enable this, which would possibly end up in the one of the STScI packages... thoughts?

[MSeifert04]

@SaraOgaz What exactly doesn't work for "DQ style mask arrays"?

ccdproc currently automatically casts the mask to a boolean array. Meaning that any "mask" value other than zero (so if it has any DQ flag) is treated as "masked" and ignored during combination:

>>> import ccdproc
>>> import numpy as np

>>> # image with "some" quality flags (totally flagged)
>>> ccd = ccdproc.CCDData(np.ones((3,3)), mask=np.ones((3,3), dtype=int), unit='adu')
>>> ccd.mask[1,1] = False  # let's unmask one pixel
>>> # image with no quality flag
>>> ccd2 = ccdproc.CCDData(np.ones((3,3)) * 2, mask=np.zeros((3,3), dtype=int), unit='adu')

>>> ccdcomb = ccdproc.Combiner([ccd, ccd, ccd2])
>>> ccdcomb.average_combine()
CCDData([[ 2.        ,  2.        ,  2.        ],
         [ 2.        ,  1.33333333,  2.        ],
         [ 2.        ,  2.        ,  2.        ]])

But I'm not sure if I understood the question correctly. So please correct me if I'm wrong. Or is it about the combine function rather than the Combiner class?

[mwcraig]

There is no objection in principle to supporting that; a little more detail and/or an example would be helpful.

Also, you may want to take a quick look at the Astropy Proposal for Enhancement for NDData, which add supporting for DQ planes at a more fundamental level. Even a simple "+1" or "-1" for that feature would be useful :)

[SaOgaz]

@MSeifert04, I'm thinking of the situation where, in ccdproc language (and hopefully I'm understanding this package correctly), I have a flag array in my CCDData object, and I want to use a set of flag values in that flag array (in HST data's case, the flags are bitflags, so easy to test for inclusion of a certain flag in each pixel) to be used for masking when running Combiner. This is the equivalent to the 'dqbits' parametere in mscombine: http://stsdas.stsci.edu/cgi-bin/gethelp.cgi?mscombine.

I'm not sure how many telescopes outside of HST use this kind of DQ/flag arrays, so I'm not sure if this would only really be useful for HST data.

@mwcraig, I'll have to read through the discussion link you put up, thanks!

[crawfordsm]

As mentioned, there is no automatic way of doing this but there is no reason that this function could not be added. If you are interested in adding it, we are happy to help, otherwise, we can add it when we have time.

[MSeifert04]

@SaraOgaz Am I right that flags is some kind of unsigned integer array where the bit representation indicates which flags are set?

For example suppose it has 2 flags, i.e. bad pixel, hot pixel. Then a bad pixel is represented by 10, a good pixel by 00 and a bad&hot pixel by 11 (bit-representation).

And the dqbits are some sort of selection criterium like "apply flag 1", "apply no flag" and "apply all flags"? The dqbits would then be a parameter for functions or should they be stored alongside the flags?

[SaOgaz]

@MSeifert04, yeah, that's they way they work for HST data DQ (data quality) arrays. The dqbits would be a parameter, where the total was the bit total for flags you wanted to use for masking, i.e. in ints, I want to flag bit 8 and bit 32, so my dqbits value will be 40.

[mwcraig]

@SaraOgaz -- could you please provide a sample image that has a DQ array? We want to try to implement this but a concrete example would be helpful. If you are interested in working on a PR that would be great! 😀

It might be a better fit in in the combine function rather than the Combiner class...but in any event a concrete example seems like the place to start.

[pllim]

@mwcraig, if you download any data from HST Archive, you would see such a DQ array. If you have DropBox or something, I can drop one for you.