Automate EC2 based staging environment for StrataSTOR

[raamsri]

Issue

Objective

Provision a staging environment on AWS EC2 to test StrataSTOR, ensuring compatibility with ZFS and related dependencies. The setup should be automated and validated through Go tests for ZFS operations.

Clearly, there may be other IaC choices including AWS Cloud Formation templates but not limited to Terraform. To prevent the analysis-paralysis, let's choose Terraform favoring faster initial iteration and scalable future development.

Assumptions

1. AWS Resources

   • Existing VPC, subnets, NAT, and Internet Gateways (if private subnet is used).
   • Existing EC2 keypair for SSH access.
   • The user has necessary AWS permissions to create and manage required resources.

2. Terraform State Management

   • A pre-configured S3 bucket and DynamoDB table are available for state storage and locking.

3. Validation Criteria

   • The environment is valid if the following command executes successfully:

     cd rodent/pkg/zfs/dataset/ && sudo go test -v -run TestDatasetOperations

Tasks

1. Provisioning

• Use Terraform to:
  • Create an EC2 instance with a base Ubuntu 24 LTS image.
  • Configure instance details (e.g., instance type, keypair, disk size) via variables.
  • Accommodate ZFS version and build type(source or Canonical/Distribution package) as variables as well.
  • Attach a security group allowing:
    • SSH (port 22) for administrative access.
  • Install and configure the following using user-data shell scripts:
    • ZFS 2.3.0-rc5 (built from source). Consider the variable config value.
    • Go 1.23.4.
    • Docker with Docker Compose
    • Latest version of NFS and Samba
    • Required utilities: ssh, git, setfacl, jq and other development utils as necessary

2. Validation

• Run the following command to validate the setup:

  cd rodent/pkg/zfs/dataset/ && sudo go test -v -run TestDatasetOperations

  This test suite may be an over kill. A particular Test function within this would be sufficient. For instance, TestDatasetOperations/Filesystems/Create should be okay. It creates necessary loopback devices, creates a pool with those devices, creates a file system and then destroys the pool and attached devices. Spares us from additional manual work.
• Output validation logs for debugging.

3. Cleanup

• Ensure Terraform provides a clean destroy operation to remove all resources created during provisioning.
• Confirm idempotency for apply/destroy cycles.

Deliverables

• Terraform configuration for automating the EC2-based staging environment.
• User-data scripts for software installation and configuration.
• Documentation on deploying, validating, and tearing down the environment.

Additional Considerations

• Use modular patterns in Terraform for reusability.

Parallel issue on a similar requirement for Rodent CI: stratastor/rodent#44

[yendelevium]

Hey @raamsri , I would like to take up this issue
I plan to create a virtual staging environment in AWS, which will be an EC2, and I will automate it by creating wrappers for the AWS SDK for easy use(this way we can also add other environments if necessary)

[raamsri]

Hi @yendelevium ,

Thank you for volunteering to take on this issue! Your proposal to use the AWS SDK with custom wrappers sounds interesting, but I’d like to share a few points to ensure we align with the objectives and constraints of this task:

1. Focus on Time Efficiency:
   While custom Go wrappers for the AWS SDK could provide flexibility for future extensibility, this approach might be time-intensive given the short time budget for this task. A faster approach would be to use established tools like Terraform or CloudFormation, which already handle AWS resource orchestration robustly.

2. Reusability and Maintainability:
   The goal is to create a staging environment that’s easy to deploy, destroy, and maintain. Terraform or similar IaC tools have built-in mechanisms for handling state, idempotency, and modularity, making them a great fit for this task without requiring additional custom development.

3. Room for Extensions Later:
   If the need arises to expand beyond EC2, we can revisit the idea of custom tooling after the initial setup is complete. For now, focusing on solving the immediate problem with the simplest and most reliable method ensures we meet our deadlines.

Proposed Path Forward:

• Automate the environment using Terraform (or another IaC tool of your choice).
• Include modular variables to allow for future extensibility and multi-environment support.
• Focus on robust user-data scripts to configure the EC2 instance.
• Ensure thorough documentation for deployment and teardown processes.

Your contributions and insights are valuable, and I’d love to hear your thoughts on this direction! Let’s prioritize delivering a functional setup quickly and explore custom tooling in subsequent iterations? I'll update the Issue's description to align with the goals.

[yendelevium]

Understood @raamsri. I have one concern, shouldn't rodent also be installed in the staging environment? Only then can we run the test commands. Also, i plan on using sensitive variables to store terraform credentials. The documentation will cover how to use them. Is this fine, or should we try other options like the AWS-CLI , or the AWS- vault?

[yendelevium]

Understood @raamsri. I have one concern, shouldn't rodent also be installed in the staging environment? Only then can we run the test commands. Also, i plan on using sensitive variables to store terraform credentials. The documentation will cover how to use them. Is this fine, or should we try other options like the AWS-CLI , or the AWS- vault?

Actually, scratch this, I will be storing the info in terraform.tfvars, and then git-ignoring the file

[yendelevium]

@raamsri Does this mean the environment is fine

[raamsri]

Yes, that checks out ZFS 👍