Monday, June 3, 2019

MYSQL 5.7 to 8..0 Migration issues I faced


When we migrated our MYSQL 5.7.22 database to 8.0.13 in AWS we have faced following issues

Issue 1: The following objects use the utf8mb3 character set. It is recommended to convert them to use utf8mb4 instead, for improved Unicode support.

Solution 1:
We used following commands to convert all the character type column’s character set to utf8mb4

      a. Following query will generate an alter script for All varchar columns belongs to utf8_unicode_ci collation. Change the database name and run. Take the output and execute into the database to convert

SELECT concat("ALTER TABLE `",table_schema,"`.`",table_name, "` CHANGE `",column_name,"` `",column_name,"` ",data_type,"(",character_maximum_length,") CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;") as _sql  FROM `columns` where table_schema like <> and data_type in ('varchar') and collation_name='utf8_unicode_ci';

     b.Following query will generate an alter script for All text, tinytext, mediumtext, longtext columns belongs to utf8_unicode_ci collation. Change the database name and run. Take the output and execute into the database to convert

SELECT concat("ALTER TABLE `",table_schema,"`.`",table_name, "` CHANGE `",column_name,"` `",column_name,"` ",data_type," CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;") as _sql FROM `COLUMNS` where table_schema like "<>" and data_type in ('text','tinytext','mediumtext','longtext') and collation_name='utf8_unicode_ci';

      c. Following query will generate an alter script for All varchar columns belongs to utf8_general_ci collation. Change the database name and run. Take the output and execute into the database to convert

SELECT concat("ALTER TABLE `",table_schema,"`.`",table_name, "` CHANGE `",column_name,"` `",column_name,"` ",data_type,"(",character_maximum_length,") CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;") as _sql  FROM `columns` where table_schema like <> and data_type in ('varchar') and collation_name= utf8_general_ci ';

      d. Following query will generate an alter script for All text, tinytext, mediumtext, longtext columns belongs to utf8_general_ci collation. Change the database name and run. Take the output and execute into the database to convert

SELECT concat("ALTER TABLE `",table_schema,"`.`",table_name, "` CHANGE `",column_name,"` `",column_name,"` ",data_type," CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;") as _sql FROM `COLUMNS` where table_schema like "<>" and data_type in ('text','tinytext','mediumtext','longtext') and collation_name= utf8_general_ci';

       e.If you are using views, views character columns also will give a similar collation issue.  To address this issue, we can create by using following command will help

SET character_set_client = latin1;
SET character_set_results = latin1;
SET character_set_connection = latin1;
DROP VIEW your_view;
CREATE VIEW your_view as (
    here_goes_your_view_query);


reference link



Issue2: 
Schema inconsistencies resulting from file removal or corruption
                Following tables show signs that either table datadir directory or frm file was removed/corrupted. Please check server logs, examine datadir to detect the issue and fix it before upgrade

Solution 2:
MySQL Shell's upgrade checker utility checkForServerUpgrade() could incorrectly report a schema inconsistency error for a table whose name included a special character such as a hyphen.
Remove “-“ hypen from table name will solve this issue

Reference link

Issue 3:  Issues reported by 'check table x for upgrade' command
Solution 3:

Within the view if you are using database reference will get some of the error like No database is selected either we can remove database reference from view/ remove before migrate those view and recreate again after migrate

Wednesday, February 20, 2019

AngularJS Upgrade (1.4 to1.7)


Step 1. In the command prompt, go to the project folder and run the following command
a. bower init ( it will create a new bower.json file, if already have bower.json file in the same folder remove and run this command)
                b.  bower update ( in the same command prompt run this command. It will update the angular and dependent library)

Step2: uppercase and lowercase methods are removed. Reference link
Due that we will get he error message as follows
Uncaught TypeError: b.lowercase is not a function
For workaround we can add following code in app.js file
angular.lowercase=function(string){
  return (typeof string === 'string') ? string.toLowerCase() : string;
 };
angular.uppercase=function(string){
 return (typeof string === 'string') ? string.toUpperCase() : string;
 };
Step3:  After upgrade we will get this error Error: transition superseded 
To resolve this error upgrade angular-ui-router
a.find and update the version of angular-ui-router in bower.json file
 "angular-ui-router": "~0.4.2",
b.  in the command prompt run the following command in the project folder
bower update angular-ui-router

Step 4 .success and .error are removed from angular 1.7, use .then and .catch as follows
find the following each text
.success(function(data, status, headers, config) {
.success(function(data, status, headers, config){
Replace with following text
.then(function(response) {
       var data = response.data;
        var status = response.status;
        var headers = response.headers;
        var config = response.config;
same as the above repeat for following. Find following text one by one
.error(function(data, status, headers, config) {
.error(function(data,status, headers, config){
Replace with following text
.catch(function(response) {
                var data = response.data;
        var status = response.status;
        var headers = response.headers;
        var config = response.config;
Step 5  Error: orderBy:notarray Value is not array-like
[orderBy:notarray] Expected array but received: {"signed":"Completed","sent":"In Progress","draft":"Draft","cancelled":"Cancelled"}
https://code.angularjs.org/1.7.7/docs/error/orderBy/notarray?p0=%7B%22signed%22:%22Completed%22,%22sent%22:%22In%20Progress%22,%22draft%22:%22Draft%22,%22cancelled%22:%22Cancelled%22%7D

in the view file use toArray:false

Monday, January 28, 2019

Merge multiple CSV files into one


Copy all the csv files which you want to merge into one folder. Navigate to the copied folder via command prompt and execute the following command to merge all csv files into one.



Copy *.csv  <>
E.g.: copy *.csv merged.csv

Tuesday, July 24, 2018

mysql_connect(): server sent charset unknown to the client


Once after migrate to MYSQL 8.0 , When I tried to access the MYSQL server from PHP . I got the following warning message and also I didn't get any response from MYSQL.

mysql_connect(): server sent charset unknown to the client

I found the solution from this link  https://stackoverflow.com/questions/43437490/pdo-construct-server-sent-charset-255-unknown-to-the-client-please-rep


https://stackoverflow.com/questions/43437490/pdo-construct-server-sent-charset-255-unknown-to-the-client-please-rep
Following settings helped me to resolve this issue. I have add the following setting in MYSQL configuration file “my.cnf” file and restart a MYSQL .
 [client]
default-character-set=utf8

[mysql]
default-character-set=utf8

[mysqld]
collation-server = utf8_unicode_ci
character-set-server = utf8

Friday, June 1, 2018

Access multiple database server from phpMyAdmin


Once phpMyAdmin is installed by default it is configured to access the local database. If we want to access the remote database we have to edit the phpMyAdmin configuration.  In linux environment configuration file is mostly available in  /etc/phpMyAdmin/config.inc.php

To connect the remote database we have to change the host name in the configuration file.

$cfg['Servers'][$i]['host']          = <<Server Name>>

To configure multiple database server
Add the following entries at the end of the file.

$i++;
 $cfg['Servers'][$i]['verbose'] =  <<Name which you want appear in the drop down>>
 $cfg['Servers'][$i]['host'] = <<Database server name or Ip Address>>
 $cfg['Servers'][$i]['port'] = '';
 $cfg['Servers'][$i]['socket'] = '';
 $cfg['Servers'][$i]['connect_type'] = 'tcp';
 $cfg['Servers'][$i]['extension'] = 'mysqli';
 $cfg['Servers'][$i]['auth_type'] = 'cookie';
 $cfg['Servers'][$i]['AllowNoPassword'] = false;

Once you have added this line,  Now we can see a Server dropdown in the phpMyAdmin login page.  Select the server name and appropriate password to login.

Thursday, May 31, 2018

Bash script to run multiple AWS EC2 instance


Step 1: open vi <<filename>>.sh
Step2:  Enter the code as follows
a.  Add this code in the first line
#!/bin/bash
b.  Connect and run the comment with in the instance
ssh -i <<keyfile >> ec2-user@<<instance Ip>> << EOF
c.  Change user into root
sudo su -
d.  Write the comment you can to execute Eg
aws s3 sync <<S3 path>>  <<local path>>
e.  Exit from root
exit
f.  exit from instance
exit
EOF
Step3: Save and exit from editor
Setp4: provide a execute permission to created .sh file
      chmod +x <<filename.sh>>
Step 5: Run the file from terminal
   ./<<filename.sh>>

Installing ClamAV Antivirus in AWS EC2 instance


Following steps will help you to setup ClamAV virus scanner in AWS Ec2 instance
Step1 : Install ClamAV
Yum install clamav clamd
Step2 : Setup up log file folder
a.       Create a new folder clamd under /var/log
b.       Provide a read write access to clamscan user
chown -R clamscan:clamscan /var/log/clamd/
Step3: If you are trying to use socket.
            a.       Create a  clamd.scan folder under /var/run
            b.       Provide a read and write access to clamscan user
chown -R clamscan:clamscan /var/run/clamd.scan/
Step4: Setup Configuration
a.       Edit the configuration file available in
vi /etc/clamd.d/scan.conf
b.       To enable logging un comment LogFile and change the path
LogFile /var/log/clamd/clamd.log
c.       If you are using Socket un comment LocalSocket and change the path
                                LocalSocket /var/run/clamd.scan/clamd.sock
d.       If you are tying to TCP port uncomment TCPSocket
Step5:  run sed -i '/^Example$/d' /etc/clamd.d/scan.conf
Step 6  Start service
                service clamd.scan start