AWS Support Case:

I intended to upgrade one of my EC2 instances. Came to know that an EC2 instance can be increased only by stopping it. So I stopped the instance i-0b4edxxx on expectation that after upgrading, I would restart the instance. However, things did not go that way. After stopping the instance, it appears that the instance terminated. I could see a fresh instance i-0d4axxx being activated. On trying to access the files from the activated one, which is under Plesk hosting, I found that WordPress websites were all gone. Help appreciated. Instance ID(s): i-0b4exxx

Reply:

Hello Greetings for the day! Thank you for contacting AWS Premium Support. I’m Manav from the EC2 linux team and I will be assisting you with the case today.

From the case description, I understand that in order to upgrade one of your instances, you stopped the instance i-0b4edd040cd31256c, but it got terminated and a new instance was launched i-0d4a0e7ebb8fc67d1. Now the new instance does not have WordPress website data on it. Therefore, you need my assistance in understanding why the instance got deleted. Please correct me if I have misunderstood your concern.

Starting with my investigation, I checked the CloudTrail logs to identify the instance and reason behind its termination. Upon reviewing the logs, I could notice that the instance i-0b4edd040cd31256c was terminated on 2021-03-10 at around 07:53 UTC, as part of the scaling action performed by your autoscaling group “AutoScalingEC2”.

I reviewed the activity history of “AutoScalingEC2” using my internal tools and could confirm that the instance was terminated as it was marked unhealthy when you stopped it.

At 2021-03-10T07:53:34Z an instance was taken out of service in response to an EC2 health check indicating it has been terminated or stopped.

The same phenomenon is mentioned in this AWS documentation : https://docs.amazonaws.cn/en_us/AWSEC2/latest/UserGuide/Stop_Start.html#instance_stop

“If your instance is in an Auto Scaling group, the Amazon EC2 Auto Scaling service marks the stopped instance as unhealthy, and may terminate it and launch a replacement instance.”

Having said this, the new instance i-0d4a0e7ebb8fc67d1 was launched on 2021-03-10 at 07:53 UTC as its replacement to meet the specified desired capacity of 1.

As you might already be aware, Amazon EC2 Auto Scaling checks that all instances within the Auto Scaling group are running and in good shape by periodically checking the health state of the instances. When it determines that an instance is unhealthy, it terminates that instance and launches a new one. This helps in maintaining the number of running instances at the minimum number (or desired number, if specified) that you defined. For more information on Health checks for Auto Scaling instances, you may refer[1].

Having said the above, I would also like to inform you that the newly launched instance will not hold any data of the previous instance. Only the settings specified in the Autoscaling Launch Configuration will be applied to all instances that are launched as part of the Autoscaling group. The custom modifications or the data stored on individual instances will be independent of the other instances and will not be copied when its replacement is launched.

Unless you have created a snapshot/AMI of the previous instance, It would not be possible to recover the data of the terminated instance. I understand that this might have caused severe inconvenience to you. Kindly note that due to strict Data Privacy and Shared Responsibility Model [+], we at AWS do not store any backup for the customer resources.

I further checked your account for any available backups (snapshots or AMI) but I was unable to find any. Unfortunately, kindly understand that, in such cases, it is not possible to recover the data.

However, I can guide you on how to prevent this from happening in future. To prevent Amazon EC2 Auto Scaling from terminating unhealthy instances, suspend the ReplaceUnhealthy process. For more information, see >>> https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-suspend-resume-processes.html

Additionally, you may find the AWS documentation[2] helpful to prevent termination of Amazon EC2 instances under different scenarios.

Lastly, I strongly recommend you to take regular backups of your instances by creating a snapshot/AMI [3][4] so that you will be in a position to recover the data under situations like these.

I hope this helps. Please reach out to me if you have any additional questions or need further assistance and I will be happy to help you. Stay safe and have a great day ahead!

References:- =========

[1]Health checks for Auto Scaling instances – https://docs.aws.amazon.com/autoscaling/ec2/userguide/healthcheck.html#replace-unhealthy-instance

[2]Instance termination – https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/terminating-instances.html#Using_ChangingDisableAPITermination

[3]Creating a snapshot – https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-creating-snapshot.html

[4]Creating a Linux AMI from an instance – https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/creating-an-ami-ebs.html [+] Data Privacy FAQ https://aws.amazon.com/compliance/data-privacy-faq/

Shared Responsibility Model https://aws.amazon.com/compliance/shared-responsibility-model/

Best regards, Manav G.

Amazon Web Services

Here is the explanation by one more AWS customer support specialists:

Hello Rajeev

Thank you for your time over call today

I reached out to you over call as I saw that you had some doubts regarding Autoscaling Group and Instances in terminate and stop state.

While on call we discussed that about the nature of instance when in the Autoscaling group as well as when not in Autoscaling group.

=> While the instance is not present in an Autoscaling group, after performing a stop operation on the instance there won’t be any data loss (unless instance store volume is attached). Therefore on starting the instance back all the data would be already present in the instance’s EBS volumes.

=> While the instance is associated with an Autoscaling group, stopping an instance makes the Autoscaling group mark it as ‘unhealthy’, and in order to maintain desired number of instances, the Autoscaling group terminates the ‘unhealthy’ instance and launches a new ‘healthy’ instance as per the launch configuration.
This newly launched instance  will not hold any data of the previous instance. Only the settings specified in the AutoScaling Launch Configuration will be applied to all instances that are launched as part of the Autoscaling group.

Thanks again for your time.
I hope this helps. In case you still have any queries, please feel free to revert back and we will be happy to help.

Have a nice day ahead!

References:-

[1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Stop_Start.html