Issue filtering default boolean

I have the exact same issue as #1090 by @moneydance which was closed without an answer.

I have a Django project with a Post model and some documents already in db. Then added the field train = BooleanField(default=False). All of my post documents have now a train field with value False. But Post.objects(train=False) throws zero documents, and Post.objects(train=None) throws all documents, which is not correct.

If I change train values to True and then to False again, then Post.objects(train=False) throws all documents, which is correct (I did this from reading issue #1090), but I shouldn't have to do this in order to make the filtering work correctly.

Django 1.11.7
MongoDB 3.4.10
mongoengine 0.15.0
python 3.5.2

Any help will be appreciated.

[whenchyiv]

I'm seeing this issue as well, also with BooleanField(default=False) fields.

[amir20]

I also have this issue. Funny if I do objects(field__ne=True) then it works. This must be a bug.

[bekab95]

I also have this issue, I have to not use default=True to avoid filter error, will this be fixed ?

[erdenezul]

@fjcaros could please provide sample code to reproduce this error?

[hoang89]

i have same problem, when i set default value, sometime it be saved to null in database, so after that when reload data from database, it make mongoengine return wrong value for this field, not only boolean field but also other field with default value

@erdenezul I'm sorry, but I'm no longer using mongoengine and I don't have the code I was using at the time :(

[bekab95]

class Item (Document):
is_enabled = BooleanField(default=True)

this doesn't even put data in database if you dont tick and untick and save document again

[whenchyiv]

I'm seeing this while trying to filter for a StringField default value as well. I can see that the document objects return the correct value, but filtering for it (e.g. Model.objects(value=default_value)) does not return any documents. Changing the value and saving it works as expected, and new documents work fine.

It seems like maybe there is an issue with filtering the default values of newly added fields on existing documents? Or is this expected behavior, and the default value should be manually added to all existing documents?

[whenchyiv]

Any idea on this? I'm still seeing the behavior above with StringField default values (in addition to booelans). Existing mongo entries return None, rather than the default value as expected, even after pulling the most recent version from PyPi.

Is this the expected behavior? Previously Mongoengine would return the default value for existing documents that did not have the newly added field.

[bagerard]

We are encountering this as well in our application when upgrading to newer version and migrating the models. (Note that this is linked to #1756).

@erdenezul, here is a snippet that demonstrates the problem:

class Person(me.Document):
    name = me.StringField()

person = Person(name='John').save()
print(Person.objects().as_pymongo())    # [{'_id': ObjectId('5b71dcb7f3ede856e183a25c'), 'name': 'John'}]

# This simulates a change in the Document schema
# that is likely to occur during new application releases
class Person(me.Document):
    name = me.StringField()
    is_human = me.BooleanField(default=True)

person = Person.objects().first()
assert person.is_human is True    # Looks good

assert Person.objects(is_human=True).count() == 0    # Confusing with the previous line
person.save()
assert Person.objects(is_human=True).count() == 0    # Since its not tracked in _changed_fields, nothing gets saved
assert Person.objects(is_human=None).count() == 1
print(Person.objects().as_pymongo())    # [{'_id': ObjectId('5b71dcb7f3ede856e183a25c'), 'name': 'John'}]

# This is also confusing
person.is_human = True
person.save()    # Since it was already True, it wasn't marked as changed and nothing gets saved here
assert Person.objects(is_human=True).count() == 0

# Trick to force the default value to the database
person.is_human = False
person.is_human = True
person.save()
assert Person.objects(is_human=True).count() == 1    # There it is

As mentioned by @whenchyiv, the "issue" is when filtering the default values of newly added fields on existing documents. I believe the right way to deal with this is to apply some database migration script manually when you modify the models (i.e: If you have existing documents in the collection and add a field with a default value in a newer version of the code, you should update all documents manually using pymongo/mongoengine).

[bagerard]

To be able to close this ticket, I think it could be worth to write a page about model migration and document things like:

• how to approach it
• best practices
• potential issues if you skip migration (like this one)

It could also be useful to start tagging tickets related to migration

[gordol]

(i.e: If you have existing documents in the collection and add a field with a default value in a newer version of the code, you should update all documents manually using pymongo/mongoengine).

you can't do it with mongoengine, necessarily, because of the aforementioned issue with dirty checking. you can write an update() script and it will actually write the changes, because that circumvents mongoengine's dirty checking, but using model property getter/setters will not actually save the change into the database unless you set a non-default value first, save(), then change back to the default and save again.