Add Migration Assistant Type Mapping documentation #9164

AndreKurait · 2025-02-07T01:49:47Z

Description

Add Migration Assistant Type Mapping documentation

Issues Resolved

MIGRATIONS-2385

Version

all (Migration Assistant 2.1.5+)

Frontend features

n/a

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Andre Kurait <[email protected]>

github-actions · 2025-02-07T01:49:59Z

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.

peternied · 2025-02-07T18:25:14Z

...tion-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md

+8. When running metadata migration, run with the additional parameter `console metadata migrate --transformer-config-file /shared-logs-output/transformation.json`.[^1]
+9. If the transformation configuration is updated, backfill/replayer will need to be stopped and restarted to apply the changes.
+
+[^1]: The `/shared-logs-output` mount will soon be relocated with a new mountpoint on the container for the EFS volume.


Can we remove, what action would a reader of the document take with this note?

peternied

Thanks for writing this up.

Lets consider reframing this documentation for a user they doesn't know if they use this deprecated feature, this kind of jumps into 'what to do' without enough context for a user to know if they are impacted and why they should make different choices.

peternied · 2025-02-07T18:30:47Z

...tion-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md

+### Using the TypeMappingsSanitizationTransformer
+
+1. Navigate to the bootstrap box and open the `cdk.context.json` with vim.
+2. Add/Update the key `reindexFromSnapshotExtraArgs` to include `--doc-transformer-config-file /shared-logs-output/transformation.json`. [^1]


Can we supply an empty file at this location and adjust our defaults to pull from this location - that would shorten what a user need to do? We are adding a lot of steps that look very similar and are easy to do incorrectly.

peternied · 2025-02-07T18:33:03Z

...tion-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md

+          "post": "new_posts"
+        }
+      },
+      "sourceProperties": {


Can we remove the version here? Maybe I am missing how it is valuable for a user to pick a different version

peternied · 2025-02-07T18:34:17Z

...tion-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md

+          "user": "combined_activity",
+          "post": "combined_activity"


Can we simplify this name to activity?

peternied · 2025-02-07T18:35:32Z

...tion-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md

+[
+  {
+    "TypeMappingsSanitizationTransformerProvider": {
+      "staticMappings": {


Can we drop this node, so its just a list of indices and the types to be mapped?

peternied · 2025-02-07T18:40:33Z

...tion-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md

+    "TypeMappingsSanitizationTransformerProvider": {
+      "staticMappings": {
+        "activity": {
+          "user": "users_only",


So any types that are [not] included will be dropped? That seems like an easy thing to get wrong or mistype, do we have a way to preflight this and warn users what they are doing if its by accident or is there a way we can make this more intention?

{ "{index-name}": { "mappedTypes": { "users": "users_only" }, "droppedTypes": [ "post" ] } }

@peternied are you asking "any types that aren't included"? That's what I got reading this, but I'd love if the documentation said that explicitly. (and/or peter's proposal here)

Thanks for catching my typo, fixed it!

peternied · 2025-02-07T18:41:33Z

...tion-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md

+        [
+          "(.*)",
+          ".*",
+          "$1"
+        ]


Please add names property names to signal what these mean

+1, these are super confusing and not explained at all. I think we had previously discussed these being dicts, e.g. {"index_regex": "(.*)", "type_regex": ".*", "new_index": "$1"} or whatever names, but either way we definitely need more explanation.

peternied · 2025-02-07T18:44:45Z

...tion-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md

+- Within the replayer, create index split is not supported.
+  - Note: This is only impactful on ES 5.x since multi type index creation is not supported on ES 6.x.


Lets frame this like the other operations so its easier to see.

If it is a known gap, we should proactively create a github issue that folks could comment / vote on and we'd leave a link here in the docs.

peternied · 2025-02-07T18:46:41Z

...tion-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md

+  - PUT/POST /{index}/{type}/{id} 
+  - PUT/POST /{index}/{type}/
+  - GET /{index}/{type}/{id}
+  - PUT/POST /_bulk
+  - PUT/POST /{index}


Lets turn this into a table with a leading column for the operation name such as Index, etc Bulk Index/Update/Delete.

peternied · 2025-02-07T18:47:50Z

...tion-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md

+- Within the replayer, create index split is not supported.
+  - Note: This is only impactful on ES 5.x since multi type index creation is not supported on ES 6.x.
+
+### Important Notes


these important notes are at the bottom, lets see if we can better integrate them into the workflow.

mikaylathompson · 2025-02-10T22:38:47Z

...tion-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md

+
+# Handling type mapping deprecation
+
+During a migration from Elasticsearch versions 6.x and before, you may have used the type mapping feature. This page provides a cookbook of different scenarios and templates that can be used to handle these deprecations.


"handle these deprecations when migrating to OpenSearch."

mikaylathompson · 2025-02-10T22:46:53Z

...tion-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md

+6. Create a file with `vim /shared-logs-output/transformation.json`. [^1]
+7. Add your transformation configuration to the file (see examples below).
+8. When running metadata migration, run with the additional parameter `console metadata migrate --transformer-config-file /shared-logs-output/transformation.json`.[^1]
+9. If the transformation configuration is updated, backfill/replayer will need to be stopped and restarted to apply the changes.


There should also probably be a note here that already-transferred data and metadata should be cleared to avoid inconsistent and incompatible states.

mikaylathompson · 2025-02-10T22:55:21Z

...tion-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md

+    "TypeMappingsSanitizationTransformerProvider": {
+      "staticMappings": {
+        "activity": {
+          "user": "users_only",


@peternied are you asking "any types that aren't included"? That's what I got reading this, but I'd love if the documentation said that explicitly. (and/or peter's proposal here)

mikaylathompson · 2025-02-10T23:04:16Z

...tion-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md

+        [
+          "(.*)",
+          ".*",
+          "$1"
+        ]


+1, these are super confusing and not explained at all. I think we had previously discussed these being dicts, e.g. {"index_regex": "(.*)", "type_regex": ".*", "new_index": "$1"} or whatever names, but either way we definitely need more explanation.

mikaylathompson · 2025-02-10T23:04:58Z

...tion-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md

+
+### Defaults
+
+When no regex mappings are specified, the transformer will default to the following behavior:


Is this when nothing at all is specified? Or regexMappings is specified but it's empty?

mikaylathompson · 2025-02-10T23:07:22Z

...tion-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md

+```
+
+This has the effect of retaining the index name for es 6+ created indices while comibining the type and index name for es 5.x created indices. If you wish to retain the index name for es 5.x created indices, please use the `staticMappings` option or override the type mappings using the `regexMappings` option similar to the `Keep original structure`
+


Can you give an example/some more info on how these can be combined for multiple indices? Is it possible to combine regexMappings for one index with static mappings for another? Can you show an example with multiple static mappings?

mikaylathompson · 2025-02-10T23:11:54Z

...tion-assistant/migration-phases/planning-your-migration/handling-type-mapping-deprecation.md

+### Important Notes
+
+- After ES 6, documents either have no type or are forced to use the type `_doc`
+- When migrating from ES 6, some type context might not be available (e.g., during RFS operations)


I don't think I understand what this point means

Naarcha-AWS · 2025-02-14T16:28:43Z

@AndreKurait: Let me know once you've implemented @peternied and @mikaylathompson's feedback and I'll begin the writer review process.

Add Migration Assistant Type Mapping documentation

929c53e

Signed-off-by: Andre Kurait <[email protected]>

AndreKurait requested review from kolchfa-aws, Naarcha-AWS, AMoo-Miki, natebower, dlvenable and epugh as code owners February 7, 2025 01:49

github-actions bot assigned kolchfa-aws Feb 7, 2025

kolchfa-aws assigned Naarcha-AWS and unassigned kolchfa-aws Feb 7, 2025

AndreKurait mentioned this pull request Feb 7, 2025

Add support for metadata transformations in TypeMappingsSanitizationTransformer opensearch-project/opensearch-migrations#1275

Open

2 tasks

peternied reviewed Feb 7, 2025

View reviewed changes

mikaylathompson reviewed Feb 10, 2025

View reviewed changes

Naarcha-AWS added migration 3 - Tech review PR: Tech review in progress labels Feb 11, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Migration Assistant Type Mapping documentation #9164

Add Migration Assistant Type Mapping documentation #9164

AndreKurait commented Feb 7, 2025

github-actions bot commented Feb 7, 2025

peternied Feb 7, 2025

peternied left a comment

peternied Feb 7, 2025

peternied Feb 7, 2025

peternied Feb 7, 2025

peternied Feb 7, 2025

peternied Feb 7, 2025 •

edited

Loading

mikaylathompson Feb 10, 2025

peternied Feb 11, 2025

peternied Feb 7, 2025

mikaylathompson Feb 10, 2025

peternied Feb 7, 2025

peternied Feb 7, 2025

peternied Feb 7, 2025

mikaylathompson Feb 10, 2025

mikaylathompson Feb 10, 2025

mikaylathompson Feb 10, 2025

mikaylathompson Feb 10, 2025

mikaylathompson Feb 10, 2025

mikaylathompson Feb 10, 2025

mikaylathompson Feb 10, 2025

Naarcha-AWS commented Feb 14, 2025

		- Within the replayer, create index split is not supported.
		- Note: This is only impactful on ES 5.x since multi type index creation is not supported on ES 6.x.


		# Handling type mapping deprecation

		During a migration from Elasticsearch versions 6.x and before, you may have used the type mapping feature. This page provides a cookbook of different scenarios and templates that can be used to handle these deprecations.


		### Defaults

		When no regex mappings are specified, the transformer will default to the following behavior:

		```

		This has the effect of retaining the index name for es 6+ created indices while comibining the type and index name for es 5.x created indices. If you wish to retain the index name for es 5.x created indices, please use the `staticMappings` option or override the type mappings using the `regexMappings` option similar to the `Keep original structure`

Add Migration Assistant Type Mapping documentation #9164

Are you sure you want to change the base?

Add Migration Assistant Type Mapping documentation #9164

Conversation

AndreKurait commented Feb 7, 2025

Description

Issues Resolved

Version

Frontend features

Checklist

github-actions bot commented Feb 7, 2025

Choose a reason for hiding this comment

peternied left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peternied Feb 7, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Naarcha-AWS commented Feb 14, 2025

peternied Feb 7, 2025 •

edited

Loading