Maybe the title isn’t the best fit to describe the problem, but is the closest to reality.
I have an CentOS 7 server ( let’s call it SSD1), running Drupal apps on top of Docker.
3 weeks after install, I have noticed that drupal is slower, at postgresql level ( via drupal devel query log). Even importing the same image Docker to a SATA server ( let’s call it SATA2), that is suppose to be slower, perform better that the initial SSD server.
So, the problem isn’t anymore about Docker. In order to further isolate the problem, I have installed vanilla drupal directly on 3 bare metal servers. ( SSD1, SATA2, SSD3)
Initially, I have tested with Vanilla Drupal 7 Devel query log, but, in order to avoid false positive, I have installed also Vanilla Drupal 8 which comes with WebProfiler from Symfony.
The benchmark sites are installed on Postgresql and MariaDB also, so the performance issue isn’t specific to a certain database engine.
All the results from 6 different Drupal 8 instances are the same ( SSD1 query logs are twice slower comparing to SATA2 and SSD3)
Drupal connects to database via PDO. In order to avoid false positive due to this layer, I have investigated also the slow query logs for Postgresql, that confirms SSD1 being twice slower.
BTW, I have the initial logs after SSD1 install, that confirms that SSD1 used to be fast. On the other hand, the disks are in good health, they are monitored with Smartmontools, all tests ( short and extended looking good).
Now comes the strange part. I ran pgbench, benchmark test on PostgreSQL, at database level. SSD1, that is supposed to be slower, has 3315 transactions per second, comparing to average 2000 tps for SSD3 and SATA2. This was so confusing in the first place, but a possible explanation is that SSD1 2X http://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E5-2640+v3+%40+2.60GHz , comparing to SSD3 single http://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+D-1540+%40+2.00GHz
So, the task isn’t about a production database with performance degradation over time, but is about Vanilla drupal installs showing server slower performance, comparing to initial setup.
Important: This task isn’t about tuning Mysql/Postgres, because the same my.cnf/postgres.conf files are used on different servers. Rather is about SysAdmin/Performance Engineer, at OS level.
Warning: This is an advanced task, please do not apply unless you are familiar with described issues above.
Working workflow via TeamViewer, can’t provide direct access to production servers. Communication via Skype with voice. Being a complicated task, communication is essential, chat only is excluded.
Further details bellow:
-Software testing versions:
-Chrome devel loading time has no difference for Vanilla installations of all servers, but for complex Drupal app the difference is visible.
-Trying reverse engineering ( what changed since initial install)
Disabling Selinux, iptables, reverting network to normal Ethernet ( was change to bridge) didn’t help.
-Testing bonnie++, fortunately I have the initial results and the actual results.
Block IO seems to be better comparing to initial , but latency seq read and delete seems to be worst (more that double now).
-Write cache is off according with recommendation
-Fstrim the ssd daily on cron.
Comparing https://wordpress.org/plugins/query-monitor/ on SSD 1 and SATA2, I get better results on SSD1.
This means that the issue is a Drupal 7/8 application level. Maybe at the PDO level, since Wordpress connects directly to database, without PDO layer.
Testing with php56u, php55u, rhphp56 still shows slow results on SSD1. Only with stock php 5.3 SSD1 works fast with Drupal, but is not an acceptable solution to downgrade.