Friday, December 4, 2020

Restrict IAM user to access specific s3 bucket or folder

 

Sometime organization will have a separate s3 folder for each developer to manage their document. In that case system admin has to restrict the developer to specific bucket. There two way we can achieve this  

 

1. Create a separate policy for each user

2. Create bucket name same as IAM user name and write a one policy to restrict all

 

Create a separate policy for each user

Step1: Go the IAM and select user

Step2:  Click Add User and provide user name, Access type and necessary details then click Next permission

Step3: Select attach policy from permission screen and click Create Policy button



Step4: In the create policy screen select JSON and copy paste following JSON and change the bucket name







{

"Version": "2012-10-17",

"Statement": [

{

"Sid": "VisualEditor0",

"Effect": "Allow",

"Action": [

"s3:ListBucketMultipartUploads",

"s3:ListBucketVersions",

"s3:ListBucket",

"s3:ListMultipartUploadParts"

],

"Resource": "arn:aws:s3:::<<Bucket Name>>"

},

{

"Sid": "VisualEditor1",

"Effect": "Allow",

"Action": [

"s3:ListAllMyBuckets",

"s3:ListJobs"

],

"Resource": "*"

},

{

"Sid": "VisualEditor2",

"Effect": "Allow",

"Action": [

"s3:PutObject",

"s3:GetObject"

],

"Resource": "arn:aws:s3:::<<Bucket Name>>/*"

}

]

}

Step5: Click Review policy, enter policy name, description and click on create policy

Step6:  Click Refresh button, search and select created policy

Step 7: Click next and complete the action

 

Create folder name same as IAM user name and write a one policy to restrict all

Step1: In the IAM create a new policy with following policy

 

{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Sid": "VisualEditor0",

            "Effect": "Allow",

            "Action": [

                "s3:ListBucketMultipartUploads",

                "s3:ListBucketVersions",

                "s3:ListBucket",

                "s3:ListMultipartUploadParts"

            ],

            "Resource": "arn:aws:s3:::${aws:username}"

        },

        {

            "Sid": "VisualEditor1",

            "Effect": "Allow",

            "Action": [

                "s3:ListAllMyBuckets",

                "s3:ListJobs"

            ],

            "Resource": "*"

        },

        {

            "Sid": "VisualEditor2",

            "Effect": "Allow",

            "Action": [

                "s3:PutObject",

                "s3:GetObject"

            ],

            "Resource": "arn:aws:s3:::${aws:username}/*"

        }

    ]

}

Step2: Attach this created policy against each user.

Tuesday, December 1, 2020

Launching EC2 instance from EBS snapshot in AWS

 

As part of backup process, we might have scheduled a EBS volume snapshot. Sometimes we will get a requirement to launch an instance from snapshot for various purpose.

The following steps to launch the EC2 instance from EBS snapshot.

Step1 : Select the snapshot to launch an EC2 instance

Step2: Create an AMI From selected snapshot

Step3: Launch instance from AMI.

Details steps as follows

1. Go to the EBS snapshot section

2. Select the snapshot

3. In the auction dropdown button select create image.



4. In the create image screen keep the default selection as it is. Enter the image name and description click create image to create an image.



 

5. Once image is created, Go to the AMIs section. Select the AMI from the Action drop down button click launch


 

6. In the launch instance screen select instance type, configure instance and configure security group to launch the new EC2 instance

Encrypt existing unencrypted MYSQL RDS instance in AWS



Currently AWS does not support to modify existing unencrypted Amazon RDS DB instance to encrypt the instance. Also, it does not support to create an encrypted read replica from an unencrypted instance.

To Modify existing unencrypted Amazon RDS DB instance to encrypt instance we have follow the following steps.

 

Step1 : Take the snapshot of existing unencrypted RDS instance

Step2: Convert unencrypted snapshot to encrypted snapshot

Step3: Restore the new RDS MYSQL instance from encrypted snapshot

Step4: Switch your application connections to new database

 

Before doing this, we must plan a proper downtime if we do a live database. We need to make sure as part of this process no transaction needs to be performed in the existing RDS instance otherwise we will have data loss.

 

If we have to minimize the downtime, we need to create a read replica to perform this step.

 

Step1 : Take the snapshot of existing unencrypted RDS instance

 1. In the database base choose the database

 2. Select the Action dropdown button and click Take Snapshot


3. Enter the snapshot name and click Take Snapshot button

 


Step2: Convert unencrypted snapshot to encrypted snapshot

1.       Go the snapshot and select the recent snapshot

2.       In the action drop down button select Copy Snapshot



3.       Enter the new snapshot name, region, select encrypt key and Copy snapshot


 

Step3: Restore the new RDS MYSQL instance from encrypted snapshot

1.      1. Select the encrypted snapshot

2.       2.  From the action drop down button select Restore Snapshot

3.       3. In the restore snapshot screen, enter new database name, VPC, Security group and Restore DB instance

Step4: Switch your application connections to new database







Monday, June 29, 2020

Upgrading PHP7 in AWS EC2 Instance

We followed following steps to upgrade our PHP version from 5.6 to 7.3

Step 1   

First need to search which PHP 7 version is available as part of AWS package by using following command

sudo yum search  php7

 

This will list all available PHP7 packages. Choose one and Install

 

Step 2

 To Install PHP use following command. The following command will install PHP7.3

sudo yum install php73

      Step 3

 Install necessary extensions as follows

sudo yum install php73-opcache php73-mysqlnd php73-gd php73-bcmath php73-mbstring php73-pdo php73-soap

 

Step 4

If you are installing fresh PHP no need to do this step, if are using already previous version of PHP and trying to install, then this step is required.  Run the following command to select Proper php version

    alternatives --config php

Saturday, June 27, 2020

Changing AWS instance type from M4 to M5 (Ena)

We cannot directly convert our instance from M4 to M5 as new instances are provided with enhanced networking capabilities through ENA (Elastic network Adapter).

If we have to convert, first we need to enable ENA support in M4 and then it will allow us to change M5 instance type.

To enable ENA we need to follow steps.

 

1.       Needs to stop the existing M4 instance.

Note: If instance is under Auto scaling group (ASG) and if we try to stop the instance automatically will get terminated. To Avoid, take the AMI of the instance and launch the instance using AMI outside ASG and follow the following step. Once ENA is enabled take another AMI and create launch configuration using M5 instance type and update the ASG with launch configuration

 

2.       Get the Instance Id of stopped instance.

 

3.       Run the following command to enable ENA. 

Note: To run this command you can use another active instance and also you should have proper role access to do.

 

aws ec2 modify-instance-attribute --instance-id <<instance Id>> --ena-support --region <<Region>>

 

4.       To Check you can use following command

aws ec2 describe-instances --instance-ids << instance Id>> --query "Reservations[].Instances[].EnaSupport" --region <<Region>>

 

5.       Change the instance type from Action->Instance Settings option

 

6.       Start the instance

You can refer following document for more details

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-resize.html

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking-ena.html


Wkhtmltopdf Letter spacing issue (Bad Kerning)

When we are using smaller font in the document and trying convert to PDF the letter spacing is completely broken.

It will too visible when we use Arial font. We tried lot of way, finally we found a solution by

By creating a 10-wkhtmltopdf.conf inside /etc/fonts/conf.d/ with following content

<?xml version='1.0'?>

<!DOCTYPE fontconfig SYSTEM 'fonts.dtd'>

<fontconfig>

 <match target="font">

  <edit mode="assign" name="rgba">

   <const>rgb</const>

  </edit>

 </match>

 <match target="font">

  <edit mode="assign" name="hinting">

   <bool>true</bool>

  </edit>

 </match>

 <match target="font">

  <edit mode="assign" name="hintstyle">

   <const>hintslight</const>

  </edit>

 </match>

 <match target="font">

  <edit mode="assign" name="antialias">

   <bool>true</bool>

  </edit>

 </match>

  <match target="font">

    <edit mode="assign" name="lcdfilter">

      <const>lcddefault</const>

    </edit>

  </match>

</fontconfig>

Following link helped us to resolve this issue

https://html.developreference.com/article/17386238/letter-spacing+is+too+large+with+wkhtmltopdf

https://github.com/kisenka/docker-kotlin-website/blob/master/scripts/10-wkhtmltopdf.conf

Wednesday, January 22, 2020

ERROR 1418 (HY000): This function has none of DETERMINISTIC, NO SQL, or READS SQL DATA in its declaration and binary logging is enabled


When you create a stored function, you must declare either that it is deterministic or that it does not modify data. Otherwise, it may be unsafe for data recovery or replication.

For example, the following function is not safe

CREATE FUNCTION generatenumber(fid INT(10))
RETURNS INT
BEGIN
 UPDATE xnumber
   SET count = (@cur_value := count+ 1)
   WHERE id=fid;
   return @cur_value;
 
END;

When you attempt to execute a stored function, if binlog_format=STATEMENT is set, the DETERMINISTIC keyword must be specified in the function definition. If this is not the case, an error is generated and the function does not run, unless log_bin_trust_function_creators=1 is specified to override this check

To relax the preceding conditions on function creation (that you must have the SUPER privilege and that a function must be declared deterministic or to not modify data), set the global log_bin_trust_function_creators system variable to 1. By default, this variable has a value of 0, but you can change it like this:

SET GLOBAL log_bin_trust_function_creators = 1;


Tuesday, January 21, 2020

#1292 - Incorrect datetime value: '0000-00-00 00:00:00' for column '' at row 1


We encounter following  error when we moved our application from existing environment to new environment

#1292 - Incorrect datetime value: '0000-00-00 00:00:00' for column ' at row 1

This error is due to our application NO_ZERO_IN_DATE mode and NO_ZERO_DATE is enabled

This mode affects whether the server permits dates in which the year part is nonzero, but the month or day part is 0. (This mode affects dates such as '2010-00-01' or '2010-01-00', but not '0000-00-00'. To control whether the server permits '0000-00-00', use the NO_ZERO_DATE mode.) This also depends on whether strict SQL mode is enabled.
  •      If this mode is not enabled, dates with zero parts are permitted and inserts produce no warning.
  •          If this mode is enabled, dates with zero parts are inserted as '0000-00-00' and produce a warning.
  •         If this mode and strict mode are enabled, dates with zero parts are not permitted and inserts produce an error, unless IGNORE is given as well. For INSERT IGNORE and UPDATE IGNORE, dates with zero parts are inserted as '0000-00-00' and produce a warning.


This mode affects whether the server permits '0000-00-00' as a valid date. Its effect also depends on whether strict SQL mode is enabled.
  • If this mode is not enabled, '0000-00-00' is permitted and inserts produce no warning.
  • If this mode is enabled, '0000-00-00' is permitted and inserts produce a warning.
  • If this mode and strict mode are enabled, '0000-00-00' is not permitted and inserts produce an error, unless IGNORE is given as well. For INSERT IGNORE and UPDATE IGNORE, '0000-00-00' is permitted and inserts produce a warning.

Please refer details of this options in https://dev.mysql.com/doc/refman/5.7/en/sql-mode.html

Solution
Change the sql_mode to allow zero dates, by removing NO_ZERO_DATE and NO_ZERO_IN_DATE. The change can be applied in the my.cnf file, so after a restart of MySQL Server, sql_mode variable will be initialized to the setting in my.cnf.
1.       Change Via PhpMyAdmin
Login to PhpMyAdmin and go to Variables and Sql_mode. Copy the existing value remove NO_ZERO_DATE and NO_ZERO_IN_DATE and apply

2.       Via Command

  • Get sql_mode : SHOW VARIABLES LIKE 'sql_mode';
  •  Result may show as follows
      ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION 

  •   Remove on the result : NO_ZERO_IN_DATE,NO_ZERO_DATE
  •  Set new configuration : SET GLOBAL sql_mode =’ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION ‘


Monday, January 20, 2020

Getting disk usage and folder usage Linux


Disk Usage
df (disk filesystem)  command will help us to know the disk usage in linux.  df command will display result in the following format





Filesystem           - List filesystem
1K-blocks             -  Total size of the file system
Used                      - used space
Available              - available space
Use%                     - percentage used.
Mounted on       - mounded as

df  command syntax is as follows
df  [options] [file]
The available options for df
Options
Description
-a, --all
include dummy file systems
-B, --block-size=SIZE

         
scale sizes by SIZE before printing them; e.g., '-BM' prints sizes in units of 1,048,576 bytes; see SIZE format below

--direct

show statistics for a file instead of mount point
produce a grand total
--total

-h, --human-readable
print sizes in human readable format (e.g., 1K 234M 2G)
-H, --si
likewise, but use powers of 1000 not 1024
-i, --inodes         
list inode information instead of block usage
-k                    
like --block-size=1K
-l, --local
--no-sync
      --output[=FIELD_LIST]

limit listing to local file systems
do not invoke sync before getting usage info (default)

use the output format defined by FIELD_LIST or print all fields if FIELD_LIST is omitted.

  -P, --portability
--sync
use the POSIX output format
invoke sync before getting usage info

  -t, --type=TYPE
limit listing to file systems of type TYPE
-T, --print-type
print file system type
  -x, --exclude-type=TYPE
   limit listing to file systems not of type TYPE

  --version                  
version information
--help    
      display this help and exit


Folder usage

du Command display the amount of disk used by the specific files or directory.
This command helps us to know the disk usage for just directories/All files, showing grand total.
For example following command
$ du -sh /var
This will display  as the option given -s( display only total) h(human  readable format) the result is as follows
17G     /var

Options are follows
Options
Description
-0, --null
end each output line with 0 byte rather than newline
-a, --all
--apparent-size
write counts for all files, not just directories
print apparent sizes, rather than disk usage; although the apparent size is usually smaller, it may be larger due to holes in ('sparse') files, internal fragmentation, indirect blocks, and the like

-B, --block-size=SIZE 
scale sizes by SIZE before printing them; e.g.,'-BM' prints sizes in units of 1,048,576 bytes;
SIZE is an integer and optional unit (example: 10M is 10*1024*1024).  Units are K, M, G, T, P, E, Z, Y (powers of 1024) or KB, MB, ... (powers of 1000)

  -b, --bytes
 equivalent to '--apparent-size --block-size=1'
-c, --total
  produce a grand total
-D, --dereference-args
dereference only symlinks that are listed on the command line
-d, --max-depth=N


--files0-from=F
print the total for a directory (or file, with --all)  only if it is N or fewer levels below the command line argument;  --max-depth=0 is the same as –summarize
summarize disk usage of the NUL-terminated file names specified in file F; if F is -, then read names from standard input
-H
equivalent to --dereference-args (-D)
-h, --human-readable
--inodes
print sizes in human readable format (e.g., 1K 234M 2G)

list inode usage information instead of block
-k
like --block-size=1K
-L, --dereference
dereference all symbolic links
-l, --count-links
count sizes many times if hard linked
-m
like --block-size=1M
-P, --no-dereference
don't follow any symbolic links (this is the default)
-S, --separate-dirs
--si
for directories do not include size of subdirectories
like -h, but use powers of 1000 not 1024
-s, --summarize
display only a total for each argument
-t, --threshold=SIZE

--time

--time=WORD

--time-style=STYLE

exclude entries smaller than SIZE if positive, or entries greater than SIZE if negative
show time of the last modification of any file in the directory, or any of its subdirectories
show time as WORD instead of modification time: atime, access, use, ctime or status
show times using STYLE, which can be: full-iso, long-iso, iso, or +FORMAT; FORMAT is interpreted like in 'date'
-X, --exclude-from=FILE
--exclude=PATTERN
exclude files that match any pattern in FILE

exclude files that match PATTERN
-x, --one-file-system
skip directories on different file systems
--help
display this help and exit
--version
output version information and exit