Skip to content

Simple Schema is nesting an object instead of honoring the mapping #582

@anthonyhi11

Description

@anthonyhi11

PGSync version: 3.2.1

Postgres version: 16.4

Elasticsearch/OpenSearch version: 2.17.1

Redis version: 7

Python version: 3.11

Problem Description:
I have enjoyed working with pgsync so far, but I'm trying to do something very simple and I just can't get it to work how I'd expect it to. I'm hoping it's something very obvious... I've googled and looked at docs, but they all seem to assume this very simple example should just work.

Very simple example mapping:

{
    "mappings": {
        "properties": {
            "id": {
                "type": "text"
            },
            "child_id": {
                "type": "text"
            },
            "first_name": {
                "type": "text"
            },
            "last_name": {
                "type": "text"
            },
            "email": {
                "type": "keyword"
            }
        }
    }
}

The schema.json:

    {
        "database": "example",
        "index": "example_index",
        "nodes": {
            "table": "parent_table",
            "columns": [
                "id"
            ],
            "children": [
                {
                    "table": "child_table",
                    "columns": [
                        "id", 
                        "first_name",
                        "last_name",
                        "email"
                    ],
                    "relationship": {
                        "variant": "object",
                        "type": "one_to_one"
                    },
                    "transform": {
                        "rename": {
                            "id": "child_id"
                        },
                        "mapping": {
                            "first_name": {
                                "type": "text"
                            },
                            "last_name": {
                                "type": "text"
                            },
                            "email": {
                                "type": "keyword"
                            }
                        }
                    
                    }
                }
            ]
        }
    }

The mapping gets overwritten and the child data comes in like this to form this type of mapping:

{
    "mappings": {
        "properties": {
            "first_name": {
                "type": "text"
            },
            "last_name": {
                "type": "text"
            },
            "email": {
                "type": "keyword"
            },
            "child_table": {
                "type": "object",
                "properties": {
                    "first_name": {
                        "type": "text"
                    },
                    "last_name": {
                        "type": "text"
                    },
                    "email": {
                        "type": "text"
                    }
                }
            }
        }
    }
}

I thought the mapping within the transform would make sure it knows to place it at the top level, but it isn't seeming to do that and is just creating an object with the fields nested within. All the data streaming from postgres is being saved to opensearch in the incorrect mapping. This wouldn't really be an issue since I could just use this new nested object, but I need to specify the mapping to be of type "keyword" and it doesn't handle that either. Any insight would be awesome!

Error Message (if any):



Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions