Monday, August 23, 2021

The table '/tmp/mysql/#sql_xxxxx' is full in MYSQL


 The table '/tmp/mysql/#sql_xxxxx' is full

 

The above-mentioned error may occur few different scenario.

We faced this issue suddenly when we updated our database from 8.0.21 to 8.0.23, After that when run a large query sometimes we started facing issue this.

 

Following discussion thread helped us to resolve this issue

https://bugs.mysql.com/bug.php?id=99100

 

Look like it is bug,

Based on this discussion in the forum,  we found a solution changing following parameter in MYSQL config helped us.

internal_tmp_mem_storage_engine=Memory;

 

 

 

 

 


Tuesday, May 18, 2021

S3 Bucket SNS Event Configuration: Unable to validate the following destination configurations.

 

Before Amazon S3 publish messages to a destination, you must grant the Amazon S3 principal the necessary permissions to call the relevant API to publish messages to an SNS topic

While creating SNS topics, we should have to provide a publish permission to S3 buckets

Step 1 : Create SNS topic ( Select standard, currently standard is supported by s3 notification)



Step 2: Access policy option select advanced option and update the JSON as like below.



In the JSON Editor change JSON below

{

 "Version": "2012-10-17",

 "Id": "example-ID",

 "Statement": [

  {

   "Sid": "example-statement-ID",

   "Effect": "Allow",

   "Principal": {

    "Service": "s3.amazonaws.com" 

   },

   "Action": [

    "SNS:Publish"

   ],

   "Resource": "SNS-topic-ARN",

   "Condition": {

      "ArnLike": { "aws:SourceArn": "arn:aws:s3:*:*:bucket-name" },

      "StringEquals": { "aws:SourceAccount": "bucket-owner-account-id" }

   }

  }

 ]

}

 

Step3: Create the topic and subscribe the topics.

Step 4: Go the S3 bucket and create event notification.

Step5: Select SNS option, select created SNS topic  and save changes


Friday, December 4, 2020

Restrict IAM user to access specific s3 bucket or folder

 

Sometime organization will have a separate s3 folder for each developer to manage their document. In that case system admin has to restrict the developer to specific bucket. There two way we can achieve this  

 

1. Create a separate policy for each user

2. Create bucket name same as IAM user name and write a one policy to restrict all

 

Create a separate policy for each user

Step1: Go the IAM and select user

Step2:  Click Add User and provide user name, Access type and necessary details then click Next permission

Step3: Select attach policy from permission screen and click Create Policy button



Step4: In the create policy screen select JSON and copy paste following JSON and change the bucket name







{

"Version": "2012-10-17",

"Statement": [

{

"Sid": "VisualEditor0",

"Effect": "Allow",

"Action": [

"s3:ListBucketMultipartUploads",

"s3:ListBucketVersions",

"s3:ListBucket",

"s3:ListMultipartUploadParts"

],

"Resource": "arn:aws:s3:::<<Bucket Name>>"

},

{

"Sid": "VisualEditor1",

"Effect": "Allow",

"Action": [

"s3:ListAllMyBuckets",

"s3:ListJobs"

],

"Resource": "*"

},

{

"Sid": "VisualEditor2",

"Effect": "Allow",

"Action": [

"s3:PutObject",

"s3:GetObject"

],

"Resource": "arn:aws:s3:::<<Bucket Name>>/*"

}

]

}

Step5: Click Review policy, enter policy name, description and click on create policy

Step6:  Click Refresh button, search and select created policy

Step 7: Click next and complete the action

 

Create folder name same as IAM user name and write a one policy to restrict all

Step1: In the IAM create a new policy with following policy

 

{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Sid": "VisualEditor0",

            "Effect": "Allow",

            "Action": [

                "s3:ListBucketMultipartUploads",

                "s3:ListBucketVersions",

                "s3:ListBucket",

                "s3:ListMultipartUploadParts"

            ],

            "Resource": "arn:aws:s3:::${aws:username}"

        },

        {

            "Sid": "VisualEditor1",

            "Effect": "Allow",

            "Action": [

                "s3:ListAllMyBuckets",

                "s3:ListJobs"

            ],

            "Resource": "*"

        },

        {

            "Sid": "VisualEditor2",

            "Effect": "Allow",

            "Action": [

                "s3:PutObject",

                "s3:GetObject"

            ],

            "Resource": "arn:aws:s3:::${aws:username}/*"

        }

    ]

}

Step2: Attach this created policy against each user.

Tuesday, December 1, 2020

Launching EC2 instance from EBS snapshot in AWS

 

As part of backup process, we might have scheduled a EBS volume snapshot. Sometimes we will get a requirement to launch an instance from snapshot for various purpose.

The following steps to launch the EC2 instance from EBS snapshot.

Step1 : Select the snapshot to launch an EC2 instance

Step2: Create an AMI From selected snapshot

Step3: Launch instance from AMI.

Details steps as follows

1. Go to the EBS snapshot section

2. Select the snapshot

3. In the auction dropdown button select create image.



4. In the create image screen keep the default selection as it is. Enter the image name and description click create image to create an image.



 

5. Once image is created, Go to the AMIs section. Select the AMI from the Action drop down button click launch


 

6. In the launch instance screen select instance type, configure instance and configure security group to launch the new EC2 instance

Encrypt existing unencrypted MYSQL RDS instance in AWS



Currently AWS does not support to modify existing unencrypted Amazon RDS DB instance to encrypt the instance. Also, it does not support to create an encrypted read replica from an unencrypted instance.

To Modify existing unencrypted Amazon RDS DB instance to encrypt instance we have follow the following steps.

 

Step1 : Take the snapshot of existing unencrypted RDS instance

Step2: Convert unencrypted snapshot to encrypted snapshot

Step3: Restore the new RDS MYSQL instance from encrypted snapshot

Step4: Switch your application connections to new database

 

Before doing this, we must plan a proper downtime if we do a live database. We need to make sure as part of this process no transaction needs to be performed in the existing RDS instance otherwise we will have data loss.

 

If we have to minimize the downtime, we need to create a read replica to perform this step.

 

Step1 : Take the snapshot of existing unencrypted RDS instance

 1. In the database base choose the database

 2. Select the Action dropdown button and click Take Snapshot


3. Enter the snapshot name and click Take Snapshot button

 


Step2: Convert unencrypted snapshot to encrypted snapshot

1.       Go the snapshot and select the recent snapshot

2.       In the action drop down button select Copy Snapshot



3.       Enter the new snapshot name, region, select encrypt key and Copy snapshot


 

Step3: Restore the new RDS MYSQL instance from encrypted snapshot

1.      1. Select the encrypted snapshot

2.       2.  From the action drop down button select Restore Snapshot

3.       3. In the restore snapshot screen, enter new database name, VPC, Security group and Restore DB instance

Step4: Switch your application connections to new database







Monday, June 29, 2020

Upgrading PHP7 in AWS EC2 Instance

We followed following steps to upgrade our PHP version from 5.6 to 7.3

Step 1   

First need to search which PHP 7 version is available as part of AWS package by using following command

sudo yum search  php7

 

This will list all available PHP7 packages. Choose one and Install

 

Step 2

 To Install PHP use following command. The following command will install PHP7.3

sudo yum install php73

      Step 3

 Install necessary extensions as follows

sudo yum install php73-opcache php73-mysqlnd php73-gd php73-bcmath php73-mbstring php73-pdo php73-soap

 

Step 4

If you are installing fresh PHP no need to do this step, if are using already previous version of PHP and trying to install, then this step is required.  Run the following command to select Proper php version

    alternatives --config php

Saturday, June 27, 2020

Changing AWS instance type from M4 to M5 (Ena)

We cannot directly convert our instance from M4 to M5 as new instances are provided with enhanced networking capabilities through ENA (Elastic network Adapter).

If we have to convert, first we need to enable ENA support in M4 and then it will allow us to change M5 instance type.

To enable ENA we need to follow steps.

 

1.       Needs to stop the existing M4 instance.

Note: If instance is under Auto scaling group (ASG) and if we try to stop the instance automatically will get terminated. To Avoid, take the AMI of the instance and launch the instance using AMI outside ASG and follow the following step. Once ENA is enabled take another AMI and create launch configuration using M5 instance type and update the ASG with launch configuration

 

2.       Get the Instance Id of stopped instance.

 

3.       Run the following command to enable ENA. 

Note: To run this command you can use another active instance and also you should have proper role access to do.

 

aws ec2 modify-instance-attribute --instance-id <<instance Id>> --ena-support --region <<Region>>

 

4.       To Check you can use following command

aws ec2 describe-instances --instance-ids << instance Id>> --query "Reservations[].Instances[].EnaSupport" --region <<Region>>

 

5.       Change the instance type from Action->Instance Settings option

 

6.       Start the instance

You can refer following document for more details

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-resize.html

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking-ena.html