Using docker and zfs to create limited size mysql database

Idea: Docker is run mysql server. Mysql data is stored on zfs volume and set the quota by zfs utility. Assume zpool name is zpool-mysql and mount point is /mysqldata

  1. Create volume for mysql data (test_quota)

    sudo zfs create -o quota=1.4gb -o mountpoint=/mysqldata/test_quota zpool-mysql/test_quota
    
  2. Create mysql instance. Port forwarding is from host 8306 to guest 3306. User can access database using host port 8306.

    docker run --name test_quota-db -e MYSQL_ROOT_PASSWORD=rootpassword -e MYSQL_DATABASE=my_test_quota -e MYSQL_USER=dbusername -e MYSQL_PASSWORD=dbuserpassword -v /mysqldata/test_quota:/var/lib/mysql -d  -p 8306:3306 mysql:latest
    
  3. Create systemd service for the container (test_quota-db.service)

    [Unit]
    Description=Mysql with quota setting
    After=docker.service
    Requires=docker.service
    
    [Service]
    User=
    Group=docker
    Restart=always
    ExecStart=/usr/bin/docker start -a test_quota-db
    ExecStop=/usr/bin/docker stop -t 2 test_quota-db
    
    
    [Install]
    WantedBy=multi-user.target
    Alias=test_quota-db.service
    

     

    sudo systemctl enable test_quota-db.service
    
  4. Using phpMyAdmin to manage database

    docker run --name test_quota-phpmyadmin --link test_quota-db:db -e MYSQL_ROOT_PASSWORD=rootpassword -d -p 8080:80 phpmyadmin/phpmyadmin:latest
    

     

    [Unit]
    Description=phpMyAdmin for test_quota-db service
    After=test_quota-db.service
    Requires=test_quota-db.service
    
    [Service]
    User=dockeradm
    Group=docker
    Restart=always
    ExecStart=/usr/bin/docker start -a test_quota-phpmyadmin
    ExecStop=/usr/bin/docker stop -t 2 test_quota-phpmyadmin
    
    
    [Install]
    WantedBy=multi-user.target
    Alias=test_quota-phpmyadmin.service
    

     

    sudo systemctl enable test_quota-phpmyadmin.service
    

Mysql quota for database using zfs on CentOS 7

Assume we have done Move mysql data folder to ZFS on CentOS 7.
Follow steps to create mysql database (run as root). (database files locate at /mysqldata)

// add fcontext for database storage folder. Only needs running once
semanage fcontext -a -t mysqld_db_t "/mysqldata(/.*)?"

// create folder for database
zfs create -o mountpoint=/mysqldata/test_quota -o quota=1gb mysqldata/test_quota
chown -R mysql:mysql /mysqldata/test_quota
chmod -R 700 /mysqldata/test_quota

cd /var/lib/mysql
mysql -e "create database test_quota;"
mv test_quota/db.opt /mysqldata/test_quota
mv test_quota /tmp
rm -rf /tmp/test_quota
ln -s /mysqldata/test_quota /var/lib/mysql
restorecon -R -v /mysqldata

Now the database space is only 1GB.
To delete mysql database

cd /var/lib/mysql
TABLES=$(mysql test_quota -e 'show tables' | awk '{ print $1}' | grep -v '^Tables' )

for t in $TABLES
do
	msg "Deleting $t table from ${dbname} database..."
	mysql ${dbname} -e "drop table $t"
done
echo "All tables have been dropped."
rm test_quota
mkdir test_quota
chown mysql:mysql test_quota
mysql -e "drop database test_quota"
zfs destroy mysqldata/test_quota
rm -rf /mysqldata/test_quota

Install apache2, php, mysql client, drush on ubuntu 16.04

Install mysql-client

sudo apt install mysql-client

Install php7

sudo apt install php php-xml php-gd php-curl php-mcrypt php-mbstring php7.0-mbstring php-gettext php-mysql

Install apache2

sudo apt install apache2 libapache2-mod-php mod_dbd
sudo a2enmod rewrite

Install drush

$ php -r "readfile('https://s3.amazonaws.com/files.drush.org/drush.phar');" > drush
$ php drush core-status
$ chmod +x drush
$ sudo mv drush /usr/local/bin
$ drush init

MariaDB/MySql promote slave server to master server

Scenario: Master server had a problem and was gone. Promote slave 1 as a master server.

Before: (Images are from dev.mysql.com)

redundancy-before

After:

redundancy-after

One all slave servers,  see message “Slave has read all relay log”.

$ mysql -u root -p
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 732
Server version: 10.1.19-MariaDB MariaDB Server

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]>STOP SLAVE IO_THREAD;
Query OK, 0 rows affected (0.01 sec)

MariaDB [(none)]> SHOW PROCESSLIST;
+----+-------------+-----------+------+---------+------+-----------------------------------------------------------------------------+------------------+----------+
| Id | User        | Host      | db   | Command | Time | State                                                                       | Info             | Progress |
+----+-------------+-----------+------+---------+------+-----------------------------------------------------------------------------+------------------+----------+
|  4 | system user |           | NULL | Connect |  308 | Slave has read all relay log; waiting for the slave I/O thread to update it | NULL             |    0.000 |
|  8 | root        | localhost | NULL | Query   |    0 | init                                                                        | SHOW PROCESSLIST |    0.000 |
+----+-------------+-----------+------+---------+------+-----------------------------------------------------------------------------+------------------+----------+
2 rows in set (0.00 sec)

On all slave servers

MariaDB [(none)]> STOP SLAVE;

On Slave 1

MariaDB [(none)]> RESET MASTER;

On other slave servers

MariaDB [(none)]> change master to 
    -> master_host='10.0.0.31',     # IP of Slave 1
    -> master_user='replica',     # replication ID
    -> master_password='password';     # replication ID's password

MariaDB [(none)]> start slave;

MariaDB [(none)]> show slave status\G

Reference: https://dev.mysql.com/doc/refman/5.7/en/replication-solutions-switch.html

MariaDB (Mysql) replication on CentOS 7

Master node: db01
Slave node: db02
1. On master node:

sudo vi /etc/my.cnf.d/server.cnf


[server]
# add follows in [server] section : get binary logs
log-bin=mysql-bin
# define uniq server ID
server-id=101


sudo systemctl restart mariadb

2. create user on master node:

$> mysql -u root -p 
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 4
Server version: 10.1.14-MariaDB MariaDB Server

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

# create user (set any password for 'password' section)
MariaDB [(none)]> grant replication slave on *.* to replica@'%' identified by 'password'; 
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> flush privileges; 
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> exit
Bye

3. Change setting on slave node:

sudo  vi /etc/my.cnf.d/server.cnf


[server]
# add follows in [server] section : get binary logs
log-bin=mysql-bin
# define server ID (different one from Master Host)
server-id=102
# read only
read_only=1
# define own hostname
report-host=db02

sudo systemctl restart mariadb

4. Dump databases on master node:

$> mysql -u root -p 
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 4
Server version: 10.1.14-MariaDB MariaDB Server

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

# lock all tables
MariaDB [(none)]> flush tables with read lock; 
Query OK, 0 rows affected (0.00 sec)
# show status (remember File, Position value)
MariaDB [(none)]> show master status; 
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000002 |      327 |              |                  |
+------------------+----------+--------------+------------------+
1 row in set (0.00 sec)

# remain the window above and open the another window and execute dump
[root@www ~]# mysqldump -u root -p --all-databases --lock-all-tables --events > mysql_dump.sql 
Enter password:
# back to the remained window and unlock
MariaDB [(none)]> unlock tables; 
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> exit
Bye
# transfer the dump to Slave Host
$> scp mysql_dump.sql node01.srv.world:/tmp/ 

5. Restore database on salve node

mysql -u root -p < /tmp/mysql_dump.sql

6. Configure replica information on slave node:

 mysql -u root -p 
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 5
Server version: 10.0.19-MariaDB-log MariaDB Server

Copyright (c) 2000, 2015, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> change master to 
    -> master_host='10.0.0.31',     # Master Hosts's IP
    -> master_user='replica',     # replication ID
    -> master_password='password',     # replication ID's password
    -> master_log_file='mysql-bin.000002',     # File value confirmed on Master
    -> master_log_pos=327;     # Position value confirmed on Master
Query OK, 0 rows affected (0.24 sec)
# start replication
MariaDB [(none)]> start slave;
Query OK, 0 rows affected (0.00 sec)
# show status
MariaDB [(none)]> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 10.0.0.31
                  Master_User: replica
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000002
          Read_Master_Log_Pos: 545
               Relay_Log_File: mariadb-relay-bin.000002
                Relay_Log_Pos: 755
        Relay_Master_Log_File: mysql-bin.000002
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 545
              Relay_Log_Space: 1055
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 101
               Master_SSL_Crl:
           Master_SSL_Crlpath:
                   Using_Gtid: No
                  Gtid_IO_Pos:
      Replicate_Do_Domain_Ids:
  Replicate_Ignore_Domain_Ids:
                Parallel_Mode: conservative
1 row in set (0.00 sec)

7. Test your result on master node

$> mysql -u root -p
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 402
Server version: 10.1.18-MariaDB MariaDB Server

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> create database testreplica;
Query OK, 1 row affected (0.01 sec)

MariaDB [(none)]> use testreplica;
Database changed
MariaDB [testreplica]> create table test_table( id int, name varchar(30) );
Query OK, 0 rows affected (0.02 sec)

MariaDB [testreplica]> insert into test_table values (1, 'test name');
Query OK, 1 row affected (0.00 sec)

MariaDB [testreplica]> select * from test_table;
+------+-----------+
| id   | name      |
+------+-----------+
| 1    | test name |
+------+-----------+
1 row in set (0.00 sec)

8. Test on salve node

$> mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 6
Server version: 10.1.18-MariaDB MariaDB Server

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> use testreplica;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [testreplica]> select * from test_table;
+------+-----------+
| id   | name      |
+------+-----------+
|    1 | test name |
+------+-----------+
1 row in set (0.00 sec)

Reference: https://www.server-world.info/en/note?os=CentOS_7&p=mariadb101&f=3

Move mysql data folder to ZFS on CentOS 7

1. change origin data folder as mount point

sudo systemctl stop mariadb.service
cd /var/lib
sudo mv mysql mysql.bak
sudo mkdir mysql
sudo chown mysql:mysql mysql

2. create zfs file system

sudo zfs create -o mountpoint=/var/lib/mysql mysqldata/mysql

3. mount zfs file system

sudo zfs mount -a

4. change mount point owner as mysql and cp all data file to /var/lib/mysql

sudo chown mysql:mysql mysql
cd mysql.bak
sudo cp -p -r * ../mysql
cd ..

5. install package policycoreutils-python

sudo yum install policycoreutils-python

6. run semanage fcontext and restorecon

sudo semanage fcontext -a -t mysqld_db_t "/mysql(/.*)?"
sudo grep -i mysql /etc/selinux/targeted/contexts/files/file_contexts.local
sudo restorecon -R -v /var/lib/mysql

7. start mysql

sudo systemctl start mariadb.service

Reference: MariaDB Changing Database Location

Check Mysql user’s privileges

Show root user’s privileges:

mysql> SHOW GRANTS FOR 'root'@'localhost';
+---------------------------------------------------------------------+
| Grants for root@localhost                                           |
+---------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' WITH GRANT OPTION |
+---------------------------------------------------------------------+

Show current user’s privileges:

mysql> show grants;
+---------------------------------------------------------------------+
| Grants for root@localhost                                           |
+---------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' WITH GRANT OPTION |
| GRANT PROXY ON ''@'' TO 'root'@'localhost' WITH GRANT OPTION        |
+---------------------------------------------------------------------+
2 rows in set (0.00 sec)

mysql> show grants for current_user;
+---------------------------------------------------------------------+
| Grants for root@localhost                                           |
+---------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' WITH GRANT OPTION |
| GRANT PROXY ON ''@'' TO 'root'@'localhost' WITH GRANT OPTION        |
+---------------------------------------------------------------------+
2 rows in set (0.00 sec)

mysql> show grants for current_user();
+---------------------------------------------------------------------+
| Grants for root@localhost                                           |
+---------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' WITH GRANT OPTION |
| GRANT PROXY ON ''@'' TO 'root'@'localhost' WITH GRANT OPTION        |
+---------------------------------------------------------------------+
2 rows in set (0.00 sec)