Getting disk usage and folder usage Linux

Disk Usage

df (disk filesystem)  command will help us to know the disk usage in linux.  df command will display result in the following format

Filesystem           - List filesystem

1K-blocks             -  Total size of the file system

Used                      - used space

Available              - available space

Use%                     - percentage used.

Mounted on       - mounded as

df  command syntax is as follows

df  [options] [file]

The available options for df

Options	Description
-a, --all	include dummy file systems
-B, --block-size=SIZE	scale sizes by SIZE before printing them; e.g., '-BM' prints sizes in units of 1,048,576 bytes; see SIZE format below
--direct	show statistics for a file instead of mount point produce a grand total
--total
-h, --human-readable	print sizes in human readable format (e.g., 1K 234M 2G)
-H, --si	likewise, but use powers of 1000 not 1024
-i, --inodes	list inode information instead of block usage
-k	like --block-size=1K
-l, --local --no-sync       --output[=FIELD_LIST]	limit listing to local file systems do not invoke sync before getting usage info (default) use the output format defined by FIELD_LIST or print all fields if FIELD_LIST is omitted.
-P, --portability --sync	use the POSIX output format invoke sync before getting usage info
-t, --type=TYPE	limit listing to file systems of type TYPE
-T, --print-type	print file system type
-x, --exclude-type=TYPE	limit listing to file systems not of type TYPE
--version	version information
--help	display this help and exit

Folder usage

du Command display the amount of disk used by the specific files or directory.

This command helps us to know the disk usage for just directories/All files, showing grand total.

For example following command

$ du -sh /var

This will display  as the option given -s( display only total) h(human  readable format) the result is as follows

17G     /var

Options are follows

Options	Description
-0, --null	end each output line with 0 byte rather than newline
-a, --all --apparent-size	write counts for all files, not just directories print apparent sizes, rather than disk usage; although the apparent size is usually smaller, it may be larger due to holes in ('sparse') files, internal fragmentation, indirect blocks, and the like
-B, --block-size=SIZE	scale sizes by SIZE before printing them; e.g.,'-BM' prints sizes in units of 1,048,576 bytes; SIZE is an integer and optional unit (example: 10M is 10*1024*1024).  Units are K, M, G, T, P, E, Z, Y (powers of 1024) or KB, MB, ... (powers of 1000)
-b, --bytes	equivalent to '--apparent-size --block-size=1'
-c, --total	produce a grand total
-D, --dereference-args	dereference only symlinks that are listed on the command line
-d, --max-depth=N --files0-from=F	print the total for a directory (or file, with --all)  only if it is N or fewer levels below the command line argument;  --max-depth=0 is the same as –summarize summarize disk usage of the NUL-terminated file names specified in file F; if F is -, then read names from standard input
-H	equivalent to --dereference-args (-D)
-h, --human-readable --inodes	print sizes in human readable format (e.g., 1K 234M 2G) list inode usage information instead of block
-k	like --block-size=1K
-L, --dereference	dereference all symbolic links
-l, --count-links	count sizes many times if hard linked
-m	like --block-size=1M
-P, --no-dereference	don't follow any symbolic links (this is the default)
-S, --separate-dirs --si	for directories do not include size of subdirectories like -h, but use powers of 1000 not 1024
-s, --summarize	display only a total for each argument
-t, --threshold=SIZE --time --time=WORD --time-style=STYLE	exclude entries smaller than SIZE if positive, or entries greater than SIZE if negative show time of the last modification of any file in the directory, or any of its subdirectories show time as WORD instead of modification time: atime, access, use, ctime or status show times using STYLE, which can be: full-iso, long-iso, iso, or +FORMAT; FORMAT is interpreted like in 'date'
-X, --exclude-from=FILE --exclude=PATTERN	exclude files that match any pattern in FILE exclude files that match PATTERN
-x, --one-file-system	skip directories on different file systems
--help	display this help and exit
--version	output version information and exit

Delete all root mail / inbox on Linux / AWS EC2

In AWS EC2 due to local email notification log file size local disk space will get full

to clear down email log

cat /dev/null > /var/spool/mail/root

MYSQL 5.7 to 8..0 Migration issues I faced

When we migrated our MYSQL 5.7.22 database to 8.0.13 in AWS we have faced following issues

Issue 1: The following objects use the utf8mb3 character set. It is recommended to convert them to use utf8mb4 instead, for improved Unicode support.

Solution 1:

We used following commands to convert all the character type column’s character set to utf8mb4

      a. Following query will generate an alter script for All varchar columns belongs to utf8_unicode_ci collation. Change the database name and run. Take the output and execute into the database to convert

SELECT concat("ALTER TABLE `",table_schema,"`.`",table_name, "` CHANGE `",column_name,"` `",column_name,"` ",data_type,"(",character_maximum_length,") CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;") as _sql  FROM `columns` where table_schema like <> and data_type in ('varchar') and collation_name='utf8_unicode_ci';

     b.Following query will generate an alter script for All text, tinytext, mediumtext, longtext columns belongs to utf8_unicode_ci collation. Change the database name and run. Take the output and execute into the database to convert

SELECT concat("ALTER TABLE `",table_schema,"`.`",table_name, "` CHANGE `",column_name,"` `",column_name,"` ",data_type," CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;") as _sql FROM `COLUMNS` where table_schema like "<>" and data_type in ('text','tinytext','mediumtext','longtext') and collation_name='utf8_unicode_ci';

      c. Following query will generate an alter script for All varchar columns belongs to utf8_general_ci collation. Change the database name and run. Take the output and execute into the database to convert

SELECT concat("ALTER TABLE `",table_schema,"`.`",table_name, "` CHANGE `",column_name,"` `",column_name,"` ",data_type,"(",character_maximum_length,") CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;") as _sql  FROM `columns` where table_schema like <> and data_type in ('varchar') and collation_name= utf8_general_ci ';

      d. Following query will generate an alter script for All text, tinytext, mediumtext, longtext columns belongs to utf8_general_ci collation. Change the database name and run. Take the output and execute into the database to convert

SELECT concat("ALTER TABLE `",table_schema,"`.`",table_name, "` CHANGE `",column_name,"` `",column_name,"` ",data_type," CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;") as _sql FROM `COLUMNS` where table_schema like "<>" and data_type in ('text','tinytext','mediumtext','longtext') and collation_name= utf8_general_ci';

       e.If you are using views, views character columns also will give a similar collation issue.  To address this issue, we can create by using following command will help

SET character_set_client = latin1;

SET character_set_results = latin1;

SET character_set_connection = latin1;

DROP VIEW your_view;

CREATE VIEW your_view as (

    here_goes_your_view_query);

reference link

https://dba.stackexchange.com/questions/8239/how-to-easily-convert-utf8-tables-to-utf8mb4-in-mysql-5-5

https://stackoverflow.com/questions/9422189/why-is-my-view-utf8-and-how-can-i-change-it-to-latin1

Issue2: 

Schema inconsistencies resulting from file removal or corruption

                Following tables show signs that either table datadir directory or frm file was removed/corrupted. Please check server logs, examine datadir to detect the issue and fix it before upgrade

Solution 2:

MySQL Shell's upgrade checker utility checkForServerUpgrade() could incorrectly report a schema inconsistency error for a table whose name included a special character such as a hyphen.

Remove “-“ hypen from table name will solve this issue

Reference link

https://bugs.mysql.com/bug.php?id=94303

Issue 3:  Issues reported by 'check table x for upgrade' command

Solution 3:

Within the view if you are using database reference will get some of the error like No database is selected either we can remove database reference from view/ remove before migrate those view and recreate again after migrate

AngularJS Upgrade (1.4 to1.7)

Step 1. In the command prompt, go to the project folder and run the following command

a. bower init ( it will create a new bower.json file, if already have bower.json file in the same folder remove and run this command)

                b.  bower update ( in the same command prompt run this command. It will update the angular and dependent library)

Step2: uppercase and lowercase methods are removed. Reference link

https://github.com/angular/angular.js/commit/1daa4f2231a89ee88345689f001805ffffa9e7de#diff-1d54c5f722aebc473dbe96f836ddf974

Due that we will get he error message as follows

Uncaught TypeError: b.lowercase is not a function

For workaround we can add following code in app.js file

angular.lowercase=function(string){

  return (typeof string === 'string') ? string.toLowerCase() : string;

 };

angular.uppercase=function(string){

 return (typeof string === 'string') ? string.toUpperCase() : string;

 };

Step3:  After upgrade we will get this error Error: transition superseded 

To resolve this error upgrade angular-ui-router

a.find and update the version of angular-ui-router in bower.json file

 "angular-ui-router": "~0.4.2",

b.  in the command prompt run the following command in the project folder

bower update angular-ui-router

Step 4 .success and .error are removed from angular 1.7, use .then and .catch as follows

folder(https://code.angularjs.org/snapshot/docs/guide/migration#migrate1.5to1.6-ng-services-$http)

find the following each text

.success(function(data, status, headers, config) {

.success(function(data, status, headers, config){

Replace with following text

.then(function(response) {

       var data = response.data;

        var status = response.status;

        var headers = response.headers;

        var config = response.config;

same as the above repeat for following. Find following text one by one

.error(function(data, status, headers, config) {

.error(function(data,status, headers, config){

Replace with following text

.catch(function(response) {

                var data = response.data;

        var status = response.status;

        var headers = response.headers;

        var config = response.config;

Step 5  Error: orderBy:notarray Value is not array-like

[orderBy:notarray] Expected array but received: {"signed":"Completed","sent":"In Progress","draft":"Draft","cancelled":"Cancelled"}

https://code.angularjs.org/1.7.7/docs/error/orderBy/notarray?p0=%7B%22signed%22:%22Completed%22,%22sent%22:%22In%20Progress%22,%22draft%22:%22Draft%22,%22cancelled%22:%22Cancelled%22%7D

in the view file use toArray:false

Merge multiple CSV files into one

Copy all the csv files which you want to merge into one folder. Navigate to the copied folder via command prompt and execute the following command to merge all csv files into one.

Copy *.csv  <>

E.g.: copy *.csv merged.csv

mysql_connect(): server sent charset unknown to the client

Once after migrate to MYSQL 8.0 , When I tried to access the MYSQL server from PHP . I got the following warning message and also I didn't get any response from MYSQL.

mysql_connect(): server sent charset unknown to the client

I found the solution from this link  https://stackoverflow.com/questions/43437490/pdo-construct-server-sent-charset-255-unknown-to-the-client-please-rep

https://stackoverflow.com/questions/43437490/pdo-construct-server-sent-charset-255-unknown-to-the-client-please-rep

Following settings helped me to resolve this issue. I have add the following setting in MYSQL configuration file “my.cnf” file and restart a MYSQL .

 [client]

default-character-set=utf8

[mysql]

default-character-set=utf8

[mysqld]

collation-server = utf8_unicode_ci

character-set-server = utf8