- 
          
 - 
                Notifications
    
You must be signed in to change notification settings  - Fork 203
 
Description
PGSync version: 3.2.1
Postgres version: 16.4
Elasticsearch/OpenSearch version: 2.17.1
Redis version: 7
Python version: 3.11
Problem Description:
I have enjoyed working with pgsync so far, but I'm trying to do something very simple and I just can't get it to work how I'd expect it to. I'm hoping it's something very obvious... I've googled and looked at docs, but they all seem to assume this very simple example should just work.
Very simple example mapping:
{
    "mappings": {
        "properties": {
            "id": {
                "type": "text"
            },
            "child_id": {
                "type": "text"
            },
            "first_name": {
                "type": "text"
            },
            "last_name": {
                "type": "text"
            },
            "email": {
                "type": "keyword"
            }
        }
    }
}
The schema.json:
    {
        "database": "example",
        "index": "example_index",
        "nodes": {
            "table": "parent_table",
            "columns": [
                "id"
            ],
            "children": [
                {
                    "table": "child_table",
                    "columns": [
                        "id", 
                        "first_name",
                        "last_name",
                        "email"
                    ],
                    "relationship": {
                        "variant": "object",
                        "type": "one_to_one"
                    },
                    "transform": {
                        "rename": {
                            "id": "child_id"
                        },
                        "mapping": {
                            "first_name": {
                                "type": "text"
                            },
                            "last_name": {
                                "type": "text"
                            },
                            "email": {
                                "type": "keyword"
                            }
                        }
                    
                    }
                }
            ]
        }
    }
The mapping gets overwritten and the child data comes in like this to form this type of mapping:
{
    "mappings": {
        "properties": {
            "first_name": {
                "type": "text"
            },
            "last_name": {
                "type": "text"
            },
            "email": {
                "type": "keyword"
            },
            "child_table": {
                "type": "object",
                "properties": {
                    "first_name": {
                        "type": "text"
                    },
                    "last_name": {
                        "type": "text"
                    },
                    "email": {
                        "type": "text"
                    }
                }
            }
        }
    }
}
I thought the mapping within the transform would make sure it knows to place it at the top level, but it isn't seeming to do that and is just creating an object with the fields nested within. All the data streaming from postgres is being saved to opensearch in the incorrect mapping. This wouldn't really be an issue since I could just use this new nested object, but I need to specify the mapping to be of type "keyword" and it doesn't handle that either. Any insight would be awesome!
Error Message (if any):