Amazon OpenSearch Serverless is an on-demand auto scaling configuration for Amazon OpenSearch Service. Since its release, the interest for OpenSearch Serverless had been steadily growing. Customers prefer to let the service manage its capacity automatically rather than having to manually provision capacity. Until now, customers have had to rely on using custom code or third-party solutions to move the data between provisioned OpenSearch Service domains and OpenSearch Serverless.
We recently introduced a feature with Amazon OpenSearch Ingestion (OSI) to make this migration even more effortless. OSI is a fully managed, serverless data collector that delivers real-time log, metric, and trace data to OpenSearch Service domains and OpenSearch Serverless collections.
In this post, we outline the steps to make migrate the data between provisioned OpenSearch Service domains and OpenSearch Serverless. Migration of metadata such as security roles and dashboard objects will be covered in another subsequent post.
Solution overview
The following diagram shows the necessary components for moving data between OpenSearch Service provisioned domains and OpenSearch Serverless using OSI. You will use OSI with OpenSearch Service as source and an OpenSearch Serverless collection as sink.
Prerequisites
Before getting started, complete the following steps to create the necessary resources:
- Create an AWS Identity and Access Management (IAM) role that the OpenSearch Ingestion pipeline will assume to write to the OpenSearch Serverless collection. This role needs to be specified in the
sts_role_arn
parameter of the pipeline configuration. - Attach a permissions policy to the role to allow it to read data from the OpenSearch Service domain. The following is a sample policy with least privileges:
- Attach a permissions policy to the role to allow it to send data to the collection. The following is a sample policy with least privileges:
- Configure the role to assume the trust relationship, as follows:
- It’s recommended to add the
aws:SourceAccount
andaws:SourceArn
condition keys to the policy for protection against the confused deputy problem: - Map the OpenSearch Ingestion domain role ARN as a backend user (as an
all_access
user) to the domain user. We show a simplified example to use theall_access
role. For production scenarios, make sure to use a role with just enough permissions to read and write. - Create an OpenSearch Serverless collection, which is where data will be ingested.
- Associate a data policy, as shown in the following code, to grant the OpenSearch Ingestion role permissions on the collection:
- If the collection is defined as a VPC collection, you need to create a network policy and configure it in the ingestion pipeline.
Now you’re ready to move data from your provisioned domain to OpenSearch Serverless.
- SEO Powered Content & PR Distribution. Get Amplified Today.
- PlatoData.Network Vertical Generative Ai. Empower Yourself. Access Here.
- PlatoAiStream. Web3 Intelligence. Knowledge Amplified. Access Here.
- PlatoESG. Carbon, CleanTech, Energy, Environment, Solar, Waste Management. Access Here.
- PlatoHealth. Biotech and Clinical Trials Intelligence. Access Here.
- Source: https://aws.amazon.com/blogs/big-data/use-amazon-opensearch-ingestion-to-migrate-to-amazon-opensearch-serverless/